Tech2wild commited on
Commit
0feb39c
·
verified ·
1 Parent(s): d7c549e

card: measured numbers on 2x RTX 3090 + config note

Browse files
Files changed (1) hide show
  1. README.md +17 -2
README.md CHANGED
@@ -48,12 +48,27 @@ vllm serve /models/Nex-N2.5-mini-Int4-Int8Mix \
48
  --default-chat-template-kwargs '{"enable_thinking": false}'
49
  ```
50
 
51
- Measured numbers from that box are in the repo README once the lane is up (KV pool, count-to-100 and prose tok/s). This checkpoint has no MTP head, so no speculative decoding.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
 
53
  ## Caveats
54
 
55
  - Data-free INT4 is a size play, not an accuracy claim; run your own evals for your workload.
56
- - Quantized with our shard-streaming quantizer (`nex_quant_stream.py`, adapted from the GLM-5.3 Int4-Int8Mix recipe at https://github.com/tonyd2wild/GLM-5.3-Int4-Int8Mix-TP4-4x-DGX-Spark).
57
 
58
  ## Credits
59
 
 
48
  --default-chat-template-kwargs '{"enable_thinking": false}'
49
  ```
50
 
51
+ Measured on that box (2026-09-08, single stream, temperature 0, thinking off, no draft):
52
+
53
+ | | |
54
+ |---|---|
55
+ | Weights per card | 9.98 GiB (Marlin WNA16 kernels) |
56
+ | KV pool | **1,867,521 tokens** at 262,144 context, 7.12x concurrency (fp8 e5m2 KV; 10 attention layers, 30 Gated DeltaNet layers) |
57
+ | Count to 100 | 185 tok/s, correct 1..100 |
58
+ | Prose (450 tokens) | 187 tok/s |
59
+ | Code (450 tokens) | 186 tok/s |
60
+ | Load | 27 s weights, 67 s torch.compile, serving in about 4 minutes |
61
+
62
+ This checkpoint has no MTP head, so no speculative decoding. Recipe, launcher, logs and the boot ladder: https://github.com/tonyd2wild/Nex-N2.5-mini-Int4-Int8Mix-2x-RTX-3090
63
+
64
+ ### Config note (fixed 2026-09-08)
65
+
66
+ The `quantization_config` targets and ignore patterns are written as `re:.*layers[.]...` on purpose. vLLM matches quantization targets against its own module names (`language_model.model.layers.N...`), not the checkpoint names (`model.language_model.layers.N...`). The first upload pinned the patterns to the checkpoint prefix; vLLM then matched nothing, built the experts unquantized in fp16 and ran out of memory before loading a weight. The prefix-agnostic patterns match both namings, so the checkpoint loads in vLLM and in transformers.
67
 
68
  ## Caveats
69
 
70
  - Data-free INT4 is a size play, not an accuracy claim; run your own evals for your workload.
71
+ - Quantized with our shard-streaming quantizer (`quant/nex_quant_stream.py` in the recipe repo, adapted from the GLM-5.3 Int4-Int8Mix recipe at https://github.com/tonyd2wild/GLM-5.3-Int4-Int8Mix-TP4-4x-DGX-Spark).
72
 
73
  ## Credits
74