TightX/code-llm-435m-sft

code-llm-435m · sft — the instruction-tuned variant

The instruction-tuned version of the 435M from-scratch code model whose pretrained weights are in ../base. Same architecture, same 32,000-token byte-level BPE vocabulary, same 1,024-token context — only the weights differ. It reads a plain-English request ("write a function that checks whether a number is prime") or continues a bare def signature, and writes Python.

This is the checkpoint the post-training log selects for release: first of seven candidates on a task set that never took part in any selection decision, and still in the top tier on the tasks that did.

Architecture

Identical to ../base — see that card for the full table. In short: 434,680,832 parameters, 22 layers, d_model 1,024 / 16 heads, SwiGLU (d_ff 4,096), RoPE (theta 500,000), RMSNorm, causal SDPA, untied embeddings, context 1,024, released file in bf16.

Post-training

Base	base/model.safetensors — the weight-soup of the pretraining run, not its last step
Data	147 M tokens of gold-standard instruction data (only examples whose reference solution passes its unit tests) interleaved with 29.4 M tokens of signature→body pairs, 5:1
Epochs	1
Learning rate	2e-5, cosine to 10%, 50-step warmup
Batch	micro 8 × 4 accumulation = 32,768 tokens per step
Precision	bf16 + TF32 + torch.compile
Seed	46
Hardware / time	one RTX 5090 (32 GB), ~90 minutes

Roughly 30 post-training runs were evaluated to reach this one. The approaches that did not work — instruction-free training, teaching the two objectives in two stages, self-distillation, weight averaging ("soups"), ratio sweeps — are documented with their numbers, including the mistakes, in the GitHub repository.

Results

Suite	Protocol	Strict	Lenient
The 11 tuning tasks	20 attempts per task per prompt style (reported on a 55-point-per-column scale: 82.2 / 110 = 74.7%)	82.2 / 110	85.8 / 110
12 held-out tasks (never used for selection)	10 attempts per task per prompt style	71.0% of 240	76.5%

Per column on the tuning suite: natural-language requests 49.8 / 55, bare signatures 32.5 / 55.

The held-out row is the point of the exercise. A model that led the tuning board at 81.6 (89.8 on the 5-attempt protocol) finished last there — 55.8 against 59–65 for the rest — because part of its advantage was a formatting habit that those 11 tasks happened to reward. Selecting on a fixed set of tasks repeatedly is how a fake champion is manufactured; the honest number is the one from tasks the selection never saw.

Limits

1. The benchmark is 23 short algorithmic tasks. It is a proxy for "writes small functions correctly" and says nothing about debugging, library use, or multi-file changes.
2. Seed variance is the largest effect measured: about ±6 points. The released checkpoint is a single seed, so that is the honest interval around its quality.
3. The held-out set was used exactly once, but it is only 12 tasks. It rules out the specific failure described above; it does not prove broad generalisation.
4. quicksort and two_sum are near-zero for every arm, including this one. That is a capability boundary of a 435M model trained on 31 B tokens, not a tuning miss.
5. The signature column is sensitive to output convention — returning a new list versus mutating in place — which is why a lenient metric is reported next to the strict one.

Files

File	What
model.safetensors	bf16 weights, 157 tensors — verified bit-identical to the training checkpoint after the fp32→bf16 cast
config.json	architecture config exactly as stored in the training checkpoint
tokenizer.json	byte-level BPE, 32,000 tokens (the same tokenizer as ../base)
SHA256SUMS.txt	artifact hash

Loading it requires the model class from the training repository (src/train.py, class CodeLLM with ModelConfig(**config.json)); a state dict built by hand will not reproduce the forward pass.

Intended use and limits

Research and education: what a few-hundred-million-parameter model can learn end to end on real data, and what makes the difference between a checkpoint that looks good and one that is good. Not for production code generation, not a coding assistant, not a substitute for a competent developer: at 1,024 tokens of context and 435 M parameters it neither plans nor reasons. Trained only on public code and instruction data; no personal data. Outputs may reproduce patterns (and licensing quirks) from the training corpus despite filtering and deduplication.

Links

DESIGN.md (decisions and why), CHALLENGES.md (every bug and contaminant), the ablation report, the post-training log with its corrections, and the evaluation harness are in the GitHub repository. MIT licensed.