pocketcoder100M

A GPT-style (nanoGPT architecture) language model trained from scratch on Python code, using the deepseek-ai/deepseek-coder-6.7b-base tokenizer.

• Total tokens trained: 2,955,935,744
• Block size: 512
• Vocab size: 32022

Loading

This is a custom architecture, not a native transformers model, so AutoModel.from_pretrained won't work out of the box. To reload:

import json, torch
from safetensors.torch import load_file

with open("config.json") as f:
    cfg = json.load(f)

config = GPTConfig(**cfg)
model = GPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()