card: measured numbers on 2x RTX 3090 + config note
Browse files
README.md
CHANGED
|
@@ -48,12 +48,27 @@ vllm serve /models/Nex-N2.5-mini-Int4-Int8Mix \
|
|
| 48 |
--default-chat-template-kwargs '{"enable_thinking": false}'
|
| 49 |
```
|
| 50 |
|
| 51 |
-
Measured
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
## Caveats
|
| 54 |
|
| 55 |
- Data-free INT4 is a size play, not an accuracy claim; run your own evals for your workload.
|
| 56 |
-
- Quantized with our shard-streaming quantizer (`nex_quant_stream.py`, adapted from the GLM-5.3 Int4-Int8Mix recipe at https://github.com/tonyd2wild/GLM-5.3-Int4-Int8Mix-TP4-4x-DGX-Spark).
|
| 57 |
|
| 58 |
## Credits
|
| 59 |
|
|
|
|
| 48 |
--default-chat-template-kwargs '{"enable_thinking": false}'
|
| 49 |
```
|
| 50 |
|
| 51 |
+
Measured on that box (2026-09-08, single stream, temperature 0, thinking off, no draft):
|
| 52 |
+
|
| 53 |
+
| | |
|
| 54 |
+
|---|---|
|
| 55 |
+
| Weights per card | 9.98 GiB (Marlin WNA16 kernels) |
|
| 56 |
+
| KV pool | **1,867,521 tokens** at 262,144 context, 7.12x concurrency (fp8 e5m2 KV; 10 attention layers, 30 Gated DeltaNet layers) |
|
| 57 |
+
| Count to 100 | 185 tok/s, correct 1..100 |
|
| 58 |
+
| Prose (450 tokens) | 187 tok/s |
|
| 59 |
+
| Code (450 tokens) | 186 tok/s |
|
| 60 |
+
| Load | 27 s weights, 67 s torch.compile, serving in about 4 minutes |
|
| 61 |
+
|
| 62 |
+
This checkpoint has no MTP head, so no speculative decoding. Recipe, launcher, logs and the boot ladder: https://github.com/tonyd2wild/Nex-N2.5-mini-Int4-Int8Mix-2x-RTX-3090
|
| 63 |
+
|
| 64 |
+
### Config note (fixed 2026-09-08)
|
| 65 |
+
|
| 66 |
+
The `quantization_config` targets and ignore patterns are written as `re:.*layers[.]...` on purpose. vLLM matches quantization targets against its own module names (`language_model.model.layers.N...`), not the checkpoint names (`model.language_model.layers.N...`). The first upload pinned the patterns to the checkpoint prefix; vLLM then matched nothing, built the experts unquantized in fp16 and ran out of memory before loading a weight. The prefix-agnostic patterns match both namings, so the checkpoint loads in vLLM and in transformers.
|
| 67 |
|
| 68 |
## Caveats
|
| 69 |
|
| 70 |
- Data-free INT4 is a size play, not an accuracy claim; run your own evals for your workload.
|
| 71 |
+
- Quantized with our shard-streaming quantizer (`quant/nex_quant_stream.py` in the recipe repo, adapted from the GLM-5.3 Int4-Int8Mix recipe at https://github.com/tonyd2wild/GLM-5.3-Int4-Int8Mix-TP4-4x-DGX-Spark).
|
| 72 |
|
| 73 |
## Credits
|
| 74 |
|