tinyllm-cpu-char

A tiny CPU-trained character-level GPT-style language model created as an educational end-to-end LLM experiment.

This is not a general chatbot. It is a deliberately small model trained to demonstrate the full loop:

1. build a vocabulary
2. train a causal transformer
3. save a checkpoint
4. generate text from learned character patterns

Safety / provenance rule

This repo contains checkpoints created locally by us. We did not pull external model weights or external quote datasets into the training machine for the philosophy experiments. The philosophy corpus was extracted from existing local workspace philosophy book material, and a provenance manifest is included at data/philosophy_quotes_local_manifest.json.

Checkpoints

Original toy corpus checkpoint

• Checkpoint: checkpoints/tinyllm_overfit_3k.pt
• Architecture: tiny GPT-style causal transformer
• Tokenization: character-level
• Parameters: 106,688
• Training device: CPU
• Dataset: tiny included toy corpus, data/tiny_corpus.txt
• Training steps: 3,000
• Final loss: train 0.1903, val 3.7094

Local philosophy quote checkpoints

These were trained from local-only quote-style corpora derived from existing workspace philosophy files. They are intentionally tiny and educational.

Checkpoint	Corpus	Params	Final loss	Notes
checkpoints/tinyllm_philosophy_quotes_local_3k.pt	1200 lines	111,040	train 1.9311, val 1.9449	Learns broad philosophy texture; not overfit.
checkpoints/tinyllm_philosophy_quotes_tiny200_4k.pt	200 lines	110,720	train 1.5459, val 1.8426	Stronger style learning; still not memorized.
checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt	50 lines	107,840	train 0.3808, val 2.5139	Best overfit-ish quote-style checkpoint.

Quick inference

pip install torch huggingface_hub
python hf_infer.py --repo-id Vishwas1/tinyllm-cpu-char \
  --filename checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt \
  --prompt "Wisdom" --tokens 300

For local inference after cloning:

python sample.py --ckpt checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt \
  --prompt "The soul" --tokens 300

Training examples

Original toy overfit:

python train.py --steps 3000 --eval-interval 500 --eval-iters 10 \
  --batch-size 16 --n-embd 64 --block-size 64 \
  --out checkpoints/tinyllm_overfit_3k.pt

Local philosophy tiny50 overfit-ish run:

python train.py --data data/philosophy_quotes_local_tiny50.txt \
  --steps 3000 --eval-interval 500 --eval-iters 10 \
  --batch-size 16 --n-embd 64 --block-size 64 --lr 0.001 \
  --out checkpoints/tinyllm_philosophy_quotes_tiny50_3k.pt

Limitations

• Character-level only.
• Generates memorized/stylized snippets, not reliable knowledge.
• Weird spelling and malformed words are expected.
• Intended for learning and experimentation.