Tech2wild/Nex-N2.5-mini-Int4-Int8Mix

Nex-N2.5-mini Int4-Int8Mix

Data-free weight-only quantization of nex-agi/Nex-N2.5-mini (a Qwen3.5-35B-A3B-family agentic model with a vision tower, 262K context) in the compressed-tensors pack-quantized format that stock vLLM loads directly. 66 GB of BF16 becomes 20 GB, which fits a pair of RTX 3090s at TP2 with room for a real KV pool, or one DGX Spark.

Recipe

Same layer policy as our GLM-5.3 Int4-Int8Mix quant, adapted to this architecture:

Tensors	Precision
256 routed experts per layer, all 40 layers (mlp.experts.N.gate_proj/up_proj/down_proj)	INT4, symmetric, group 128
Attention q/k/v/o_proj, Gated DeltaNet in_proj_qkv, in_proj_z, out_proj, shared expert gate/up/down_proj (layers 1 to 39)	INT8, symmetric, group 128
Layer 0 side layers, routers (mlp.gate), shared_expert_gate, DeltaNet in_proj_a, in_proj_b, conv1d, A_log, dt_bias, all norms, embeddings, lm_head, the whole vision tower	BF16, untouched

Round-to-nearest per group with min-max scales (compressed-tensors QuantizationArgs), no calibration data. The source checkpoint stores the experts fused per layer (experts.gate_up_proj [256, 1024, 2048], experts.down_proj [256, 2048, 512]); the quantizer splits them into per-expert 2-D weights and writes the standard per-expert names, which is what vLLM's Qwen3.5-MoE loader reads. Self-test round-trip error: INT4 experts about 11 to 14% mean relative per weight, INT8 layers about 0.8%.

Stats: 30,720 expert matrices INT4, 244 side-layer matrices INT8, 702 tensors passed through. 93,594 tensors, 16 shards, 19.6 GiB.

Serving

Tested with vLLM v0.27.1 on 2x RTX 3090 (NVLink pair), Ampere, no FP8 compute:

vllm serve /models/Nex-N2.5-mini-Int4-Int8Mix \
  --served-model-name nex-n2.5-mini --port 8015 \
  --quantization compressed-tensors --dtype float16 \
  --tensor-parallel-size 2 --enable-expert-parallel \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --max-num-seqs 4 --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8_e5m2 --trust-remote-code --enable-chunked-prefill \
  --limit-mm-per-prompt '{"image":2,"audio":0}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Measured on that box (2026-09-08, single stream, temperature 0, thinking off, no draft):

Weights per card	9.98 GiB (Marlin WNA16 kernels)
KV pool	1,867,521 tokens at 262,144 context, 7.12x concurrency (fp8 e5m2 KV; 10 attention layers, 30 Gated DeltaNet layers)
Count to 100	185 tok/s, correct 1..100
Prose (450 tokens)	187 tok/s
Code (450 tokens)	186 tok/s
Load	27 s weights, 67 s torch.compile, serving in about 4 minutes

This checkpoint has no MTP head, so no speculative decoding. Recipe, launcher, logs and the boot ladder: https://github.com/tonyd2wild/Nex-N2.5-mini-Int4-Int8Mix-2x-RTX-3090

Config note (fixed 2026-09-08)

The quantization_config targets and ignore patterns are written as re:.*layers[.]... on purpose. vLLM matches quantization targets against its own module names (language_model.model.layers.N...), not the checkpoint names (model.language_model.layers.N...). The first upload pinned the patterns to the checkpoint prefix; vLLM then matched nothing, built the experts unquantized in fp16 and ran out of memory before loading a weight. The prefix-agnostic patterns match both namings, so the checkpoint loads in vLLM and in transformers.

Caveats

• Data-free INT4 is a size play, not an accuracy claim; run your own evals for your workload.
• Quantized with our shard-streaming quantizer (quant/nex_quant_stream.py in the recipe repo, adapted from the GLM-5.3 Int4-Int8Mix recipe at https://github.com/tonyd2wild/GLM-5.3-Int4-Int8Mix-TP4-4x-DGX-Spark).

Credits

• nex-agi for Nex-N2.5-mini (Apache 2.0).
• Qwen team for the Qwen3.5 MoE architecture.
• QuantTrio for the Int4-Int8Mix layer policy we follow.
• vLLM and compressed-tensors (Neural Magic / Red Hat) for the format and kernels.

Quantized by Tony DeAngelo (Tech2Wild / 2Wild) with Kai, 2026-09-08.