---

What we work on

Sol Labs is built around one question:

How capable can a model become before scaling parameters stops being the easiest answer?

Instead of immediately making models larger, we experiment with:

• recurrent and weight-shared transformers
• extreme token-to-parameter ratios
• compact tokenizers
• grouped-query attention
• learned n-gram memory
• loop-conditioned recurrence
• XSA-style attention
• channel / feature mixing ideas
• optimizer design
• tiny coding models
• compact text-to-image models
• aggressive ablation and failed-experiment tracking

A Sol model does not need to be large to be interesting.

---

☀️ The Sol family

Not every model shares the same architecture.

Some are serious candidates for long training runs; others exist specifically to test an idea quickly and tell us whether it works.

---

Featured model — Sol Lite Base

Sol Lite Base is the current flagship tiny-language-model architecture.

Architecture

4,096-token digit-aware byte-level BPE
                ↓
       embedding + EngramLite
                ↓
          1 prelude block
                ↓
     4 recurrent blocks × 2
                ↓
        5 finishing blocks
                ↓
             RMSNorm
                ↓
       tied vocabulary head

Sol Lite Base combines:

• selective recurrence
• 3:1-style grouped-query attention
• XSA value subtraction
• loop conditioning
• EngramLite
• tied embeddings
• RoPE
• gated SiLU MLPs

The goal is simple: spend parameters where they matter, then reuse computation where possible.

---

EngramLite

One of Sol Labs' recurring research ideas is giving tiny transformers a cheap learned memory for local patterns.

EngramLite adds a small bank of learned bigram / trigram representations near the input of the model.

For Sol Lite Base:

EngramLite entries: 2,048

The idea is to avoid spending precious transformer capacity relearning every common short-range token pattern from scratch.

---

Recurrence

Tiny models run into a brutal tradeoff:

more layers = more capability
more layers = more parameters

Sol Labs frequently explores recurrent depth as a way around that.

A block can be reused multiple times while loop conditioning tells it which pass it is currently performing.

That lets a model gain additional effective computation without allocating a completely separate set of weights for every step.

---

Data

We care as much about what a tiny model sees as the architecture itself.

Current Sol Labs training preferences lean toward high-signal datasets such as:

• FineWeb-HQ / strong FineWeb subsets
• UltraFineWeb L1 English
• UltraFineWeb L3 English
• FineMath and higher-quality math subsets
• FinePDFs
• Cosmopedia
• recent Nemotron-CC data

We prefer gradual curriculum shifts over one fixed mixture for an entire run.

The broader goal is to move from clean general language toward increasingly dense educational, technical, mathematical, and reasoning-heavy material.

---

Overtraining tiny models

Sol Labs intentionally trains small models on far more tokens than conventional parameter-count heuristics might suggest.

For example:

Sol Lite Base
~15M parameters
18B-token target

We are interested in the regime where parameter count stays tiny while:

• data quality rises
• total tokens increase dramatically
• effective depth increases
• architectural efficiency improves

A major research question for us is whether many tiny models are simply undertrained.

---

Optimizer research

Sol Labs also develops experimental optimizers.

SimO — Simple Optimizer

The core SimO idea is intentionally minimal:

gradient information chooses the direction of the update, not its magnitude.

In the base formulation, a parameter moves by roughly the current learning-rate-sized step:

gradient says up   → +LR
gradient says down → -LR

The learning rate can then adapt based on training behavior.

Variants explored include:

• SimO
• SimOW
• Gated SimO
• Gated SimOW
• Gated Muon-SimO
• M-SimOW

We have also experimented with more aggressive optimizer families:

• Navier
• Stokes
• Flux
• BAD
• Good

These are research optimizers, not claims that every Sol model uses them.

---

What the optimizer experiments taught us

One recurring result has been especially important:

lower pretraining loss does not automatically mean a better tiny model.

In internal small-model experiments, some runs learned very quickly early on and then stalled on downstream capability even while loss continued improving.

That changed how we evaluate Sol models.

We care about:

training loss
+ held-out loss
+ downstream benchmarks
+ checkpoint behavior over time

—not just whichever run ends with the smallest loss.

---

Sol Q

Sol Q is the coding-focused branch of Sol Labs.

The long-term goal is to explore how much practical coding ability can be compressed into very small models through:

• code-heavy curricula
• architecture specialization
• strong token efficiency
• overtraining
• distillation
• optimizer research

Coding models are particularly interesting for Sol Labs because their output space is highly structured, making them a useful testbed for parameter efficiency.

---

SolPix

SolPix is Sol Labs' compact text-to-image research family.

The project explores whether modern image-generation ideas can be scaled aggressively downward rather than only upward.

Current directions include:

• compact latent representations
• highly compressed image latents
• small transformer backbones
• diffusion / flow-style objectives
• architecture efficiency at tens-of-millions-of-parameters scale

SolPix is separate from the Sol language-model stack but follows the same philosophy:

make the architecture earn every parameter.

---

Experimental models

Some Sol names are deliberately more playful because the models are experiments.

Examples include:

Sol Smoothie · Sol Milkshake · Sol Tiramisu · Sol Chocolate · Sol Frappé · Sol Lassi

These models may test:

• different recurrence patterns
• unusual width/depth ratios
• new memory systems
• attention alternatives
• tokenizer changes
• training recipes
• optimizer changes

Poor results are still useful results.

Sol Labs does not treat every trained checkpoint as a success.

---

Research philosophy

1. Small first

Before scaling parameters, ask whether the architecture, tokenizer, data, optimizer, or training duration can improve.

2. Overtrain on purpose

Tiny models are cheap enough to explore unusually large token budgets.

3. Reuse computation

Weight sharing and recurrence can buy effective depth without matching parameter growth.

4. Keep failed experiments

A bad architecture tells us what not to build next.

5. Benchmark checkpoints, not just final loss

The best model may appear before the final training step.

6. Architecture should match scale

Ideas that make sense at 70B parameters are not automatically efficient at 15M.

7. Have fun with it

Research can be serious without every model being named Transformer-Experimental-v7.

---

Current research areas

tiny language models
recurrent transformers
weight sharing
GQA
Engram / n-gram memory
XSA
loop conditioning
channel mixing
nGPT / normalized architectures
mHC-style residual ideas
optimizer design
extreme overtraining
small coding models
compact image generation

---

Model naming

The core Sol hierarchy is generally:

Nano → Lite → Flash → Pro

with experimental names used when a model branches away from the main family.

Versioned generations may appear as:

Sol Nano 2
Sol Lite 2
Sol Flash 2
Sol Pro 2
Sol Pro 3

Names do not guarantee identical architecture between generations.

---

Openness

Where practical, Sol Labs aims to publish enough information to make experiments understandable and reproducible:

• architecture details
• parameter counts
• tokenizer details
• training tokens
• dataset sources
• optimizer settings
• checkpoints
• benchmark methodology
• negative results
• corrected metrics when bugs are found

If a logging bug changes a reported result, we would rather correct it than preserve a prettier number.

---

What Sol Labs is not

Sol Labs is not trying to claim that a 15M model replaces frontier-scale AI.

The point is different.

We want to know how far careful engineering can push the very bottom of the scale — where models are:

• cheap to train
• cheap to run
• easy to iterate on
• useful for architecture research
• accessible to individual researchers

Sometimes the most interesting scaling experiment is the one where you refuse to scale the parameter count.

---