🤏 quecto mode

appvoid

45 4 173

https://ko-fi.com/appvoid

AI & ML interests

singularity through byte-level tokens and small language models

Recent Activity

repliedto their post 14 minutes ago

If you lack ideas for a cool model, here's one. Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery. You would have answered the question: Can a model learn human languages from structure alone?

posted an update about 3 hours ago

repliedto Banaxi-Tech's post about 19 hours ago

Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget. Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through. The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting. Follow us for more: https://huggingface.co/BananaMind @vovaRL @Banaxi-Tech Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP

View all activity

Organizations

Posts 38

Post

If you lack ideas for a cool model, here's one.

Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery.

You would have answered the question: Can a model learn human languages from structure alone?

Post

1379

Random corporate secret of tonight:

Try overfitting a tiny model on billions of high-quality datapoints: you can't. You can do 100 epochs and see the model still improving.

You're welcome.