TightX/code-llm-435m-base

code-llm-435m — a Python code-completion model trained from scratch on one RTX 5090

A 435M-parameter decoder-only Transformer for Python code completion. Data pipeline, tokenizer, architecture, training loop, checkpoint merging and evaluation were all written and run by one person on one consumer GPU — this repository holds the weights; the code, design record, ablation report and failure log live in the GitHub repository linked below.

This folder is the pretrained base. Its weights are the merged (weight-averaged) product of the pretraining run: under this training scheme the merged checkpoint, not the last step, is the finished pretrained model — hence the folder name. The instruction-tuned variant is in ../sft.

Architecture

Parameters	434,680,832 (bf16)
Layers	22
d_model / heads	1024 / 16 (head_dim 64)
FFN	SwiGLU, d_ff 4096
Position	RoPE, theta 500000
Norm	RMSNorm (pre-norm)
Attention	causal SDPA (no biases anywhere)
Embeddings	untied (input embed + output head)
Vocabulary	32,000 byte-level BPE, trained on the filtered corpus (≈3.5 chars/token)
Context	1024 tokens
Precision	bf16 (released file)

Training

• Data: three Python sources (codeparrot / the_stack / star_coder), six-layer quality filtering, copyright filtering, metadata stripping; concatenated into one continuous 31.0 B-token stream (no dataset boundaries, no optimizer resets — an earlier version was silently retrained on the same 8 B tokens twice by a resume bug, and the design change makes that class of bug impossible).
• Schedule: WSM — constant learning rate, then a 30,000-step cooldown, then weighted averaging of the last 10,000 checkpoints (this file is that merged model, not the last step).
• Hardware: a single RTX 5090, 32 GB, Blackwell sm_120.

What it can and cannot do

Can: complete short Python functions when given a signature and the opening indentation. On a human-graded benchmark (10 docstring-free tasks × 5 seeds, scored 0/1/2, max 100) this model scores 66/100; the earlier 353M version — same architecture, same GPU — scores 6/100. The difference is the data pipeline, not the architecture. Sample, verbatim (temperature 0.2, seed 0; this is a partial-credit example, not a showcase):

def quicksort(arr):
    if len(arr) <= 1:
        return arr
    else:
        pivot = arr[0]
        left = [x for x in arr if x < pivot]
        right = [x for x in arr if x == pivot]      # <- wrong: should be > pivot
        return quicksort(left) + [pivot] + quicksort(right)

The recursion, the base case and the partition are there; one comparison operator is wrong, so the function drops elements. That is what "66/100" looks like at this scale — the structure is learned before the detail is, which is exactly why the benchmark is human-graded rather than pass/fail.

Cannot: follow instructions — this is a completion model, not a chat model (supervised fine-tuning is documented separately in the GitHub repo). It is blind to docstrings: at this scale a from-scratch model reads a docstring as the end of the function, so standard HumanEval docstring prompts score ≈0 and docstring-free prompts are used instead. At 18 B tokens the same pipeline still scored 0/50 on a five-algorithm suite — implementation ability appears between 18 B and 31 B tokens, which the ablation report documents rather than hides.

Files

File	What
model.safetensors	bf16 weights, 157 tensors — verified bit-identical to the training checkpoint after the fp32→bf16 cast
config.json	architecture config exactly as stored in the training checkpoint
tokenizer.json	byte-level BPE, 32,000 tokens
SHA256SUMS.txt	artifact hash

Loading it requires the model class from the training repository (src/train.py, class CodeLLM with ModelConfig(**config.json)); torch.load of a state dict built by hand will not reproduce the forward pass described above.

Intended use and limits

Research and education: understanding what a few-hundred-million-parameter model actually learns when trained end to end on real data. Not for production code generation, not for instruction following, not a substitute for a competent developer. Trained only on code; no personal data. Outputs may reproduce patterns (and licensing quirks) from the training corpus despite copyright filtering.

Links

Everything else — DESIGN.md (decisions and why), CHALLENGES.md (every bug and contaminant), the ablation report, the fine-tuning log with corrections, and every evaluated generation — is in the GitHub repository. MIT licensed.