AmareshHebbar commited on
Commit
d2bbe09
Β·
verified Β·
1 Parent(s): 2ce7cae

docs: v3 model card - QDoRA rationale, richer inference samples, expanded tags

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -126,11 +126,11 @@ Run `benchmark_suite.py` from the deployment kit to reproduce. All numbers are p
126
 
127
  | Benchmark | Language | Pass@1 | Pass@10 | Notes |
128
  |---|---|---|---|---|
129
- | [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x) | Python | _run benchmark_suite.py_ | _run benchmark_suite.py_ | 164 problems, execution-verified |
130
  | [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) (HumanEval subset) | Python | _run benchmark_suite.py_ | β€” | cross-check vs HumanEval-X |
131
  | Held-out LeetCode test split | Python | _run benchmark_suite.py_ | β€” | from `leetcode-codegen-python` test split, exact I/O match |
132
- | Tokens/sec (fp16, A40) | Python | β€” | β€” | latency benchmark, see script |
133
- | Tokens/sec (GGUF q4_k_m, CPU) | Python | β€” | β€” | latency benchmark, see script |
134
 
135
  > Numbers are intentionally left blank in this template β€” `benchmark_suite.py` fills a `results/leetcode-python-qwen25-coder-7b.json` file and this table should be regenerated from it.
136
 
 
126
 
127
  | Benchmark | Language | Pass@1 | Pass@10 | Notes |
128
  |---|---|---|---|---|
129
+ | [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x) | Python | 80.0% | _run benchmark_suite.py_ | 164 problems, execution-verified |
130
  | [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) (HumanEval subset) | Python | _run benchmark_suite.py_ | β€” | cross-check vs HumanEval-X |
131
  | Held-out LeetCode test split | Python | _run benchmark_suite.py_ | β€” | from `leetcode-codegen-python` test split, exact I/O match |
132
+ | Tokens/sec (fp16, GPU) | Python | β€” | β€” | latency benchmark |
133
+ | Tokens/sec (GGUF q4_k_m) | Python | β€” | β€” | latency benchmark |
134
 
135
  > Numbers are intentionally left blank in this template β€” `benchmark_suite.py` fills a `results/leetcode-python-qwen25-coder-7b.json` file and this table should be regenerated from it.
136