Moreover, the implications are that finetuning models already pretrained on structure could outperform ones starting from scratch.
🤏 quecto mode
appvoid
appvoid
deepak-banka's profile picture
rakuten-sunilkumar-yadav's profile picture
FILALIHicham's profile picture·
AI & ML interests
singularity through byte-level tokens and small language models
Recent Activity
repliedto their post about 1 hour ago
If you lack ideas for a cool model, here's one.
Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery.
You would have answered the question: Can a model learn human languages from structure alone? posted an update about 4 hours ago
If you lack ideas for a cool model, here's one.
Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery.
You would have answered the question: Can a model learn human languages from structure alone? repliedto Banaxi-Tech's post about 20 hours ago
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
https://huggingface.co/BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXPOrganizations
replied to their post about 1 hour ago
posted an update about 4 hours ago
Post
31
If you lack ideas for a cool model, here's one.
Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery.
You would have answered the question: Can a model learn human languages from structure alone?
Train a model from scratch on wikipedia with one twist: the tokenizer changes the actual token ids used on every sample fed. If somehow still learns English, you have made an astonishing discovery.
You would have answered the question: Can a model learn human languages from structure alone?
- 1 reply
replied to Banaxi-Tech's post about 20 hours ago
I'm pretty sure you need to check the memory window of that model before training. Check falcon-h1-tiny blogpost to understand what I mean.
reacted to Banaxi-Tech's post with 🔥 about 20 hours ago
Post
1027
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
- 2 replies
posted an update 1 day ago
Post
1426
Random corporate secret of tonight:
Try overfitting a tiny model on billions of high-quality datapoints: you can't. You can do 100 epochs and see the model still improving.
You're welcome.
Try overfitting a tiny model on billions of high-quality datapoints: you can't. You can do 100 epochs and see the model still improving.
You're welcome.
- 1 reply
posted an update 3 days ago
Post
834
- GLM 5.2
- Flux 3
- New Qwen model
- New small model leaderboards
- Lots of people finetuning smol models.
- Some even under 12 year olds clauders are here (was not on my bingo card this year)
- ChatGPT's Sol became a lot faster this week
- LFM2.5 2.6b
- Kimi K3 (though only a few will run it)
- New Ling 3.0 Tiny
- New video model that is making south park videos?
- Deepseek v4 flash being more honest than bigger models
- The new model from meta
Everything Everywhere All At Once
- Flux 3
- New Qwen model
- New small model leaderboards
- Lots of people finetuning smol models.
- Some even under 12 year olds clauders are here (was not on my bingo card this year)
- ChatGPT's Sol became a lot faster this week
- LFM2.5 2.6b
- Kimi K3 (though only a few will run it)
- New Ling 3.0 Tiny
- New video model that is making south park videos?
- Deepseek v4 flash being more honest than bigger models
- The new model from meta
Everything Everywhere All At Once
- 1 reply
replied to Banaxi-Tech's post 3 days ago
Congrats on this release!
reacted to Banaxi-Tech's post with 🔥 3 days ago
Post
2791
We're excited to release BananaMind 2 Micro, our smallest model yet.
It fits a compact architecture in only 2.9M parameters achieving the highest parameter efficiency on BananaMind Base Bench against comparable models.
It achieves comparable performance to GPT S2 5M and GPT S 5M at almost half the size while beating CMA 1M Mini.
BananaMind 2 Micro achieved the #1 spot on the Open SLM Leaderboard for the sub 3M category (not added yet but it achieves #1)
For the training we used Muon + the XSA Refresh Gate with a 5e-2 lr for Muon and 4e-3 for the 1D weights.
Its score on our efficiency measure is 0.326 getting the first place with Syn 2.6M on the second place scoring 0.291 and GPT S 5M at 0.235*
Check it out at BananaMind/BananaMind-2-Micro and follow us at:
@vovaRL
@DedeProGames
@Banaxi-Tech
BananaMind
Our new releases aren't stopping 🚀 August 13-14 BananaMind 2 Pro
It fits a compact architecture in only 2.9M parameters achieving the highest parameter efficiency on BananaMind Base Bench against comparable models.
It achieves comparable performance to GPT S2 5M and GPT S 5M at almost half the size while beating CMA 1M Mini.
BananaMind 2 Micro achieved the #1 spot on the Open SLM Leaderboard for the sub 3M category (not added yet but it achieves #1)
For the training we used Muon + the XSA Refresh Gate with a 5e-2 lr for Muon and 4e-3 for the 1D weights.
Its score on our efficiency measure is 0.326 getting the first place with Syn 2.6M on the second place scoring 0.291 and GPT S 5M at 0.235*
Check it out at BananaMind/BananaMind-2-Micro and follow us at:
@vovaRL
@DedeProGames
@Banaxi-Tech
Our new releases aren't stopping 🚀 August 13-14 BananaMind 2 Pro
- 9 replies
replied to their post 4 days ago
dont forgot my models
Should submit them to the leaderboard, would help with publicity!
yep
reacted to Enderchef's post with 👍 4 days ago
Post
3247
GPT-X2.5-135M is finally released! 🚀
The new flagship from Axiomic Labs takes 3rd on the open SLM leaderboard trailing only the SmolLMs, check it out and follow us:
AxiomicLabs/GPT-X2.5-135M
The new flagship from Axiomic Labs takes 3rd on the open SLM leaderboard trailing only the SmolLMs, check it out and follow us:
AxiomicLabs/GPT-X2.5-135M
- 2 replies
replied to their post 5 days ago
I think not everyone will enjoy it... but if you code you certainly will find it appealing.
replied to their post 5 days ago
great! don't forget rose-mini!
posted an update 5 days ago
Post
2360
I don't know if it was us or one of you guys or maybe all of us at once but lately we have seen a finetuning/pretraining explosion of models below 200m params and we can't be more happy about it keep coming tinkerers all of this is possible because of you!
- 8 replies
replied to their post 5 days ago
sure!
replied to their post 5 days ago
maybe some staff from huggingface will see this and do something accordingly
replied to their post 5 days ago
Ty man! (your check is in the mail)
lol sure!
reacted to ajibawa-2023's post with 🔥 6 days ago
Post
6993
Shell-Code-Large
Dataset: ajibawa-2023/Shell-Code-Large
Shell-Code-Large is a large-scale corpus of Shell scripting source code comprising approximately 640,000 code samples stored in JSON Lines (.jsonl) format. The dataset is designed to support research in large language model (LLM) pretraining, code intelligence, DevOps automation, cloud infrastructure engineering, system administration, and software engineering automation.
By providing a high-volume, language-specific corpus focused exclusively on Shell scripting, Shell-Code-Large enables systematic experimentation in automation workflows, deployment pipelines, infrastructure management, and command-line tooling. These domains remain foundational to Linux systems, cloud-native platforms, CI/CD environments, and modern DevOps practices.
Shell-Code-Large addresses the need for a dedicated Shell-focused dataset at substantial scale, enabling targeted research into scripting patterns, command composition, workflow orchestration, infrastructure automation, and operational engineering practices
Dataset: ajibawa-2023/Shell-Code-Large
Shell-Code-Large is a large-scale corpus of Shell scripting source code comprising approximately 640,000 code samples stored in JSON Lines (.jsonl) format. The dataset is designed to support research in large language model (LLM) pretraining, code intelligence, DevOps automation, cloud infrastructure engineering, system administration, and software engineering automation.
By providing a high-volume, language-specific corpus focused exclusively on Shell scripting, Shell-Code-Large enables systematic experimentation in automation workflows, deployment pipelines, infrastructure management, and command-line tooling. These domains remain foundational to Linux systems, cloud-native platforms, CI/CD environments, and modern DevOps practices.
Shell-Code-Large addresses the need for a dedicated Shell-focused dataset at substantial scale, enabling targeted research into scripting patterns, command composition, workflow orchestration, infrastructure automation, and operational engineering practices
replied to their post 6 days ago
cool stuff, just gave a follow
replied to their post 6 days ago
And Rose-Medium
Yep
replied to their post 6 days ago
Also, @GODELEV did something cool with rose-mini.