Model card: recalibrate val bpb under upstream packing; document the token_bytes regeneration

README.md

@@ -35,7 +35,7 @@ minimum validation bpb: 0.719042
 | | value |
 |---|---|
 | parameters | 1,384,122,122 |
-| min val bpb | 0.719042 |
+| min val bpb | 0.719042 (varlen packing, 10.5M val tokens — see the caveat below) |
 | CORE | 0.2517 |
 | train time | 110.74 min on 8xH100 80GB |
 | throughput | 1,212 ms/step, 864,856 tok/sec (steps 20–99) |
@@ -49,19 +49,47 @@ DecoderStack tracks 0.2–1.4% behind through step 1000 (where the upstream base
 stopped) while running ~6% faster per step (1,212 vs 1,291 ms/step). Full head-to-head in
 [`logs/baseline_report.md`](logs/baseline_report.md).
 
-⚠️ **but val bpb is not really comparable across the two pipelines.** Upstream nanochat
-batches with best-fit cropping, discarding ~35% of tokens to pack rows exactly (its own
-dataloader docstring's figure) against DecoderStack's 11–12% `seq_len` truncation — so the
-two are not scoring the same text. And upstream measures loss with **cross-document
-attention**: its rows concatenate several documents with no isolation (`cu_seqlens` appears
-nowhere in upstream `nanochat/gpt.py`), where DecoderStack's varlen path isolates documents
-in validation as in training. The direction of that bias has not been measured. Treat the
-throughput numbers as the solid comparison and the bpb deltas as indicative.
+⚠️ **but 0.719042 is not comparable to an upstream nanochat val bpb.** Upstream batches with
+best-fit cropping — it fills each row with largest-doc-that-fits and crops a document to
+finish the row, then attends causally across the whole row with no document isolation
+(`cu_seqlens` appears nowhere in upstream `nanochat/gpt.py`). DecoderStack packs varlen and
+isolates documents in validation as in training. Different truncation points, different
+conditioning context: the two are not scoring the same predictions.
 
-`ChrisMcCormick/nanochat-varlen-d24-2026-03-22` is the closest sibling release
-(0.724772 val bpb, 0.2614 CORE, 97.4 min). It is varlen-trained, so it does not carry the
-caveat above, but it is still **not** a controlled comparison: it trained with FP8 and a
-different 32k tokenizer.
+### Measured 2026-08-01: the packing gap is larger than the model gap
+
+Re-measured this checkpoint under **upstream's own row packer**, driven with this repo's
+tokenizer over the same pinned val shard and unwound into varlen batches whose `cu_seqlens`
+sit on the row boundaries — so the model predicts exactly the tokens upstream would ask
+for, with exactly the same context.
+
+| packing | val tokens | bpb |
+|---|---|---|
+| upstream best-fit rows | 41.9M | **0.737183** |
+| upstream best-fit rows | 10.5M | 0.735274 |
+| varlen, document-isolated | 41.9M | 0.724316 |
+| varlen, document-isolated | 10.5M | 0.720717 |
+
+(all on the legacy `token_bytes` basis, for continuity with the numbers above; the
+regenerated basis reads ~0.05% higher — see [⚠️ Tokenizer](#️-tokenizer))
+
+Three things follow, and they revise the comparison:
+
+1. **Packing alone is worth ~0.0129 bpb** — same model, same shard, same budget. That is
+   *larger* than the gap between this model and its sibling release, so bpb must never be
+   cross-compared between the two packings.
+2. **`ChrisMcCormick/nanochat-varlen-d24-2026-03-22` is ahead, not behind.** Its 0.724772
+   was measured under best-fit rows; the like-for-like number here is **0.737183**, so it
+   leads by ~0.0124 bpb. The raw headline figures (0.719042 vs 0.724772) pointed the other
+   way purely because of packing. It also trained with FP8 and a different 32k tokenizer,
+   so this is still not a controlled comparison — but it is now an honest one.
+3. **The 10.5M val budget reads low.** Upstream's default is `--eval-tokens 80*524288` =
+   41.9M; this run used 10.5M, which comes in ~0.002–0.004 bpb optimistic rather than
+   merely noisier.
+
+Reproduce: `agent-ops/stacks/2026-08-01_1241pm_d24-val-bpb-recalibration/`.
+
+Throughput remains the clean comparison: 1,212 vs 1,291 ms/step, ~6% faster per step.
 
 Both numbers also deserve error bars: Karpathy's re-runs of an identical d24 config span
 **0.0153 in CORE** (5 runs, mean 0.261), so our 0.2517 and that 0.2614 are inside each
@@ -77,6 +105,23 @@ one in `ChrisMcCormick/nanochat-varlen-d24-2026-03-22` — 31,474 of 32,759 merg
 differ. Both are 32,768 tokens and both load without complaint, so nothing will warn you.
 Pairing these weights with the wrong tokenizer produces garbage, not slightly-worse text.
 
+### `token_bytes.pt` was regenerated 2026-08-01
+
+bpb divides summed nats by summed BYTES, and the bytes-per-token table was built by
+decoding each id to a Python `str` and measuring `.encode("utf-8")` — which corrupts every
+token that is not valid standalone UTF-8 (a lone continuation byte round-trips through
+U+FFFD and measures 3 bytes instead of 1). **191 of 32,768 ids** were affected here.
+Upstream fixed the calculation in
+[`2ce972a`](https://github.com/karpathy/nanochat/commit/2ce972a), but the fix lives in
+`tok_train.py` and so only lands when a tokenizer is retrained — a released `token_bytes.pt`
+keeps the stale values forever.
+
+`tokenizer/token_bytes.pt` here is now the corrected table; `tokenizer/token_bytes_legacy.pt`
+is the pre-fix one, kept so previously published numbers stay reproducible. Measured on this
+model, the correction moves bpb by **+0.05%** (0.724316 → 0.724741 varlen at 41.9M), matching
+upstream's estimate. `tokenizer.pkl` is untouched — encode/decode are unchanged, this is
+purely the byte-accounting used by the metric.
+
 ## Contents
 
 ```