24/24 Retrieval, Yet the Embeddings Changed: An MLX Q4–Q8 Sweep with CUDA Controls

I started this experiment for a practical reason: I wanted to know which local embedding model should handle short indexing jobs, which one deserved to stay resident, and whether a smaller quantized checkpoint would actually finish the work faster on Apple Silicon.

The first result looked reassuring. Every full-dimensional model retrieved all 24 expected documents. BF16, Q4, Q6, Q8, ordinary MLX quantization, oQ, and the importance-calibrated enhanced path all scored 24/24.

Then I compared the vectors themselves.

The Q4 models had changed substantially even though the retrieval score had not moved. Across three model sizes, every Q4 variant failed the strict vector fidelity gate. Q6 passed on the 1.5B and 8B models, but not on the 0.6B model. Q8 passed everywhere.

That produced the central lesson of this small experiment:

Retrieval equivalence is not representation equivalence.

The quality boundary used in this post

Before discussing speed, I need to make the boundary explicit:

Q4 is not an acceptable-quality result in this experiment. Every Q4, oQ4, and oQ4e checkpoint failed the preregistered representation-fidelity gate. Q4 remains in the charts only as an intentionally invalid low-quality comparator and to show what a capacity-first decision would appear to buy.

Q6 is the lowest acceptable tested tier for the 1.5B and 8B models. It is not a universal floor: all 0.6B Q6 variants failed the same gate. For the 0.6B model, Q8 was the lowest tested tier that passed. Q8 was also the only tier that passed across all three model sizes.

"Acceptable" here has a narrow experimental meaning. It means passing this study's frozen retrieval-and-alignment gate. It does not mean that Q6 is proven safe for every embedding dataset or application.

What I tested

The sweep covered three decoder embedding models:

• Qwen3-Embedding-0.6B
• GTE-Qwen2-1.5B-instruct
• Qwen3-Embedding-8B

Each family included BF16 and nine quantized variants:

That is 30 decoder checkpoints in total. Every quantization branched directly from its family's BF16 MLX checkpoint. I did not use a quantized checkpoint as the source of another quantization. This full 0.6B/1.5B/8B Q/oQ/oQe matrix is the expanded study; it supersedes the earlier partial Qwen3 and oQe-only runs.

After the MLX sweep, I added CUDA-native BF16, bitsandbytes INT8, and bitsandbytes NF4 controls on Hugging Face ZeroGPU. These controls are not equivalents of MLX Q, oQ, or oQe. They test whether the broader distinction between retrieval preservation and representation fidelity survives a second runtime and quantization stack.

The frozen smoke set contained 24 query/document pairs. Every model used the same query instruction, tokenizer, last-token pooling path, L2 normalization, and comparison procedure. The enhanced variants reused one 128-sample, 512-token importance-matrix cache per family.

The gate required:

• no top-1 or recall@5 regression;
• MRR loss no worse than 0.01;
• no more than two changed query ranks;
• minimum aligned embedding cosine of at least 0.99 against BF16.

That last condition is what the retrieval score alone could not see.

Perfect rankings can conceal damaged geometry

All 30 full-dimensional checkpoints achieved top-1 = 1.0, MRR = 1.0, and zero rank changes on the frozen set. The minimum aligned cosine told a different story. Each cell below reports the best of Q, oQ, and oQe at that nominal bit width; all three variants were tested.

* GTE scope warning: these GTE rows are MLX-internal comparisons against the retained MLX BF16 baseline. Do not mix these vectors with the later CUDA GTE results; that family failed the cross-runtime parity check.

The enhanced path was consistently closest to BF16 within each nominal bit row. Its advantage was most visible at Q4, where there was more quantization damage for calibration to steer away from. It improved Q4; it did not rescue Q4 across the 0.99 boundary.

This distinction has precedent in model-compression research. Dutta et al. showed that similar aggregate accuracy can conceal answer changes after compression and argued for distance and flip measurements in addition to task accuracy. Our test applies the analogous idea to embedding vectors: report ranking preservation and representation drift as separate outcomes (Accuracy is Not All You Need).

The low-dimensional test exposed the damage earlier

At 32 dimensions, BF16 Qwen3-Embedding-0.6B retrieved 24/24 documents. Plain Q4 retrieved 18/24, oQ4 retrieved 19/24, and oQ4e recovered to 21/24. At 128 dimensions and above, all formats returned to 24/24 on this small set.

GTE-Qwen2-1.5B was unstable below 512 dimensions even before quantization, so those lower-dimensional results should not be interpreted as a quantization- only failure. The broader point is simpler: truncation can amplify a distortion that full-dimensional top-k retrieval conceals.

When does quantization actually save time?

Quantization lowered cold-start/first-encode latency and peak memory, especially for the 8B model. It did not improve resident throughput. BF16 was the fastest format once all models were loaded and warm.

Here, one operation means encoding one text sequentially at parallelism 1. The first operation starts a fresh model process against a warm filesystem cache and includes model loading, lazy materialization, tokenization, and its first 21-token encode. The model then stays resident; later operations use the measured 48-text corpus throughput. I estimated total time as:

cold start + first encode + (remaining sequential operations / resident throughput)

Strong graph disclaimer: the shaded winner bands above describe measured time efficiency only. The orange oQ4e line is not a recommendation. Every Q4 variant failed the quality gate and is shown solely as a low-quality point of comparison. The graph uses sequential, parallelism-1 operations; the 48-text corpus run is only a batch-like throughput proxy, not a true tensor-batching benchmark.

The performance-only transitions were:

• 0.6B: oQ4e through approximately 13 operations, then BF16.
• 1.5B: oQ4e for 1–3, oQ8e for 4–8, oQ6e for 9–32, then BF16.
• 8B: oQ4e through approximately 227 operations, then BF16.

Applying the quality boundary changes the operational answer:

• 0.6B: BF16 is the time-efficient acceptable choice throughout this test. oQ8e passed quality and remains an acceptable memory-saving option, but it did not establish a time advantage.
• 1.5B: oQ8e is fastest for roughly 1–8 operations, oQ6e for 9–32, and BF16 beyond that.
• 8B: oQ6e is the acceptable quantized winner through roughly 186 operations; BF16 becomes faster after that.

The 8B cold-start/first-encode difference was substantial:

The result is not paradoxical. A smaller checkpoint reduces file movement, materialization, and memory pressure, while resident quantized computation can still pay dequantization and kernel overhead. A broader Apple Silicon profiling study similarly found that lower precision does not automatically guarantee faster inference across hardware and workloads (Benazir and Lin). A community oMLX report also observed slower oQ generation than BF16, although that issue is anecdotal and does not establish the cause (oMLX issue #388).

A separate vLLM study likewise found that the preferred quantization format changes with synchronous versus continuously batched serving. Its generative CUDA workload is related deployment context, not validation of these embedding measurements (Kurtic et al.).

CUDA controls reproduced the central result

The ZeroGPU controls used each family's CUDA BF16 output as the reference for its CUDA quantized variants. This avoids counting the normal difference between MLX and CUDA implementations as quantization damage.

For Qwen3 0.6B, that runtime floor was small: CUDA BF16 versus MLX BF16 had a mean aligned cosine of 0.999836 and a minimum of 0.999603. It was measurable, but far smaller than the failed quantized cases below.

All six bitsandbytes controls still achieved 24/24 Top-1 retrieval. Their minimum aligned cosines and resident throughput were much less uniform:

The corresponding BF16 throughputs were 32.12, 40.78, and 25.41 texts/s for the 0.6B, 1.5B, and 8B families. In other words, the 0.6B and 8B INT8 controls passed the 0.99 fidelity gate but were substantially slower than warm BF16. NF4 was faster than INT8 in all three families, but it failed the fidelity gate despite perfect retrieval. This is the same measurement warning seen in the MLX sweep, not a claim that bitsandbytes and MLX formats are interchangeable.

One plausible contributor is the much smaller representation budget, not simply that one format is "linear" and the other is not. bitsandbytes LLM.int8() separates large-magnitude outliers for higher-precision computation, while NF4 stores block-normalized weights using 16 levels chosen for a normal distribution. That four-bit codebook can save more memory while perturbing an embedding model's accumulated geometry more strongly. This experiment did not isolate codebook shape, block scaling, outlier handling, and kernel behavior, so that is a mechanism-informed hypothesis rather than a causal result (QLoRA).

The 8B memory tradeoff was real even though the resident speed tradeoff was unfavorable: CUDA peak allocation fell from 15.15 GB for BF16 to 8.38 GB for INT8 and 5.37 GB for NF4.

The 8B run also exposed an infrastructure boundary. ZeroGPU's startup packer failed while placing approximately 15.1 GB of root-level tensors. Loading the mounted shards directly inside the allocated GPU call completed successfully: BF16 encoded 48 texts at 25.41 texts/s with a 15.15 GB CUDA peak. That workaround is operational evidence about this Space configuration, not a quality advantage for either model format. Success at roughly the same peak argues against a simple total-VRAM ceiling, but the evidence does not distinguish a contiguous- allocation constraint from startup-packer or wrapper behavior.

This is also why quality-qualified compression matters in practice. Memory ceilings and model-placement machinery can force a deployment decision before the theoretical quality limit is reached. Here, 8B INT8 cut peak allocation to 8.38 GB and passed the vector gate; NF4 fit in 5.37 GB but did not. On MLX, the analogous operational choices were, depending on family, the passing oQ6e or oQ8e artifacts—not the smaller but failed oQ4e result.

One family failed the cross-runtime parity check

GTE-Qwen2 1.5B produced stable rankings but unstable vectors across loading paths. Root-packed CUDA BF16 versus directly loaded CUDA BF16 had a minimum aligned cosine of 0.66259. Direct CUDA BF16 versus local MLX BF16 fell to 0.58152, again without a rank change.

I therefore exclude GTE-Qwen2 1.5B from cross-runtime representation trends until implementation, conversion, instruction, and pooling parity are resolved. Its CUDA quantized rows remain valid only as same-path comparisons against the direct CUDA BF16 reference. This caveat does not invalidate the earlier local MLX comparisons, where every GTE quantization was measured against the same retained MLX BF16 baseline.

The first suspect is the CUDA loading or model-execution path: the root-packed and direct CUDA BF16 runs already diverged while holding the checkpoint revision, tokenizer, query instruction, and pooling recipe constant. The still larger CUDA-versus-MLX gap may then add implementation or conversion differences. That ordering narrows the next diagnostic test, but it is not yet proof of the cause.

A nominal bit label is not the whole model

Standard Q, oQ, and oQe are not interchangeable implementations of the same label. Current oQ documentation describes a sensitivity-driven mixed-precision system that measures quantization error and spends extra bits where calibration indicates they matter most (oQ quantization documentation).

Recent variable-bit research makes a related but distinct argument: precision can be allocated non-uniformly at group granularity instead of treating one global bit width as the complete description of a model. That work learns the allocation during training, whereas this experiment evaluates post-training quantization, so it is context rather than direct validation (Variable Bit-width Quantization).

The practical implication is that “4-bit versus 6-bit” is insufficient provenance. A model card should record the quantizer, group size, mixed- precision plan, calibration procedure, source checkpoint, and runtime version.

Mixed-index migration passed ranking and failed thresholds

This is a deterministic local engineering smoke test, not MTEB and not a claim of universal embedding quality. Twenty-four pairs are enough to expose a measurement problem, not enough to establish production retrieval quality.

I tested all 54 migration directions using the saved MLX vectors: BF16 queries against each quantized document index, and quantized queries against the BF16 document index. All 54 retained 24/24 Top-1 retrieval.

Fixed similarity thresholds did not survive as cleanly. I calibrated a threshold once on BF16 and did not retune it for the candidate formats. The largest break was Qwen3-Embedding-0.6B standard-Q4 queries against a BF16 index: 5 of 24 expected matches became false negatives even though their Top-1 result was unchanged.

A safe migration should therefore recalibrate rather than merely lower the threshold:

1. Preserve a labeled validation slice containing both matches and hard negatives near the old boundary.
2. Score it with the old and candidate query/index combinations.
3. Sweep the candidate threshold to recover the chosen recall and false-positive target.
4. Run the old and new paths side by side for a canary period.
5. If no candidate threshold preserves the required operating point, rebuild the index instead of treating the formats as compatible.

That adds a more operational version of the central lesson:

Rank compatibility is not threshold compatibility. Changing the query or index representation may require re-indexing or threshold recalibration, even when a small retrieval benchmark remains perfect.

Practical conclusions

For these three models and this gate:

• Q4/oQ4/oQ4e: invalid quality; capacity-first comparison only.
• Q6/oQ6/oQ6e: lowest acceptable tested tier for 1.5B and 8B.
• Q8/oQ8/oQ8e: only tier that passed across all three model sizes.
• oQe: closest to BF16 within every nominal bit row.
• BF16: best when the model remains resident and memory is available.
• Quantized 8B: useful for short-lived jobs because cold-start and memory savings remain operationally meaningful before BF16 amortizes its load cost.
• Mixed-index migration: all 54 directions preserved Top-1, but fixed thresholds still produced false negatives.
• CUDA controls: reproduced the ranking-versus-fidelity split; Qwen3 INT8 passed the vector gate, while all tested NF4 controls failed it.

The decision is therefore not “which format is fastest?” It is:

1. Which formats pass the application's quality boundary?
2. How many operations will run before the model is unloaded?
3. Is the workload sequential, truly batched, or concurrent?
4. Does the existing vector index need to remain compatible?

Scope and next test

A larger study should add multiple datasets, calibration seeds, embedding architectures, clustering, semantic textual similarity, reranking, threshold stability, and true batching. The GTE discrepancy also needs a controlled implementation-and-pooling parity study before its CUDA and MLX vector spaces can be compared directly.

Reproducibility note

The tested local stack was:

• Mac Studio, Apple M1 Max (10 CPU cores), 32 GB unified memory
• macOS 27.0 prerelease, build 26A5388g
• oMLX 0.5.3
• mlx-lm 0.31.3
• MLX 0.32.0

The CUDA control stack used PyTorch 2.11.0+cu130 on an NVIDIA RTX PRO 6000 Blackwell Server Edition ZeroGPU slice. Results were generated in a private Space so unfinished controls could not be mistaken for published benchmark claims.

Pinned input evidence:

• Qwen/Qwen3-Embedding-0.6B revision 97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3
• Qwen/Qwen3-Embedding-8B revision 1d8ad4ca9b3dd8059ad90a75d4983776a23d44af
• Alibaba-NLP/gte-Qwen2-1.5B-instruct local MLX BF16 weight SHA-256 9e9da58bd1371c47a08bc82f58bd29d33a6831094372dd717d68148f061dd11a
• Alibaba-NLP/gte-Qwen2-1.5B-instruct upstream revision a9af15a6372d7d6b25e9fb07c2ccb9e1fe645644
• frozen 24-pair dataset SHA-256 324c0b753d6802803e7b5d03997442bc205029241c19fb5fd3bc97bc94b86d72

The public MLX model matrix and its model cards are collected at MLX Embedding Quantization Matrix.

This version pin matters. Current oQ documentation has evolved to newer oQ+ terminology and different 8-bit format details. In this post, oQe means the enhanced path implemented by oMLX 0.5.3; the tested artifacts declare affine, group-size-64 quantization. Current main documentation should not be read as the exact implementation specification for these historical artifacts.

The complete frozen evidence, exact model-revision lock, checksums, and portable runner are in the public reproducibility bundle. The runner first verifies the published evidence and can regenerate the 54-direction mixed-index analysis without downloading model weights:

hf download TiGa-RCE/embedding-quant-repro-2026-07-28 \
  --repo-type dataset --local-dir embedding-quant-repro
cd embedding-quant-repro
uv sync
uv run python reproduce.py verify
uv run python reproduce.py mixed-index

To rerun the representative local gate on Apple Silicon:

uv sync --extra mlx
uv run python reproduce.py mlx --profile quick

The complete 30-checkpoint download is deliberately opt-in with --profile full --family all.

The bounded CUDA controls run through the public ZeroGPU Space:

hf auth login
uv sync --extra zerogpu
uv run python reproduce.py cuda

That default invokes only the three 0.6B BF16/INT8/NF4 controls. The complete nine-run matrix is deliberately opt-in with --family all --variant all and may exceed the visitor's available ZeroGPU quota. Both runtime paths use the same locked model revisions and frozen evaluation inputs.

References

• Abhinav Dutta et al., Accuracy is Not All You Need, NeurIPS 2024.
• Eldar Kurtic et al., “Give Me BF16 or Give Me Death?” Accuracy–Performance Trade-Offs in LLM Quantization, ACL 2025.
• Afsara Benazir and Felix Xiaozhu Lin, Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective.
• Hamish Ogilvy, Variable Bit-width Quantization: Learning Per-Group Precision for “Bigger-but-Smaller” Language Models.
• jundot/oMLX, oQ Quantization Documentation.
• jundot/oMLX, Issue #388: Why is the inference speed of the oQ-quantized model slower than the original version?.
• Hugging Face, ZeroGPU Spaces documentation.
• Hugging Face Transformers, bitsandbytes quantization documentation.
• Tim Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023.