Musicroll-50M
Musicroll-50M is a small Qwen3-style causal language model over a 259-token byte-level music vocabulary. It was used as the MusicRoll negative-control model in the replication artifact for "Many Next-Token Predictors are In-Context Learners".
The checkpoint is provided as model.safetensors with a compact tokenizer:
- vocabulary size: 259
- hidden size: 512
- layers: 16
- attention heads: 4
- key/value heads: 2
- context length: 4096
- BOS/EOS/PAD token ids: 256/257/258
The bitstring replication harness encodes tasks as piano-roll-like text and loads this checkpoint through the local MLX model loader.
Intended Use
This upload is primarily for reproducing the MusicRoll negative-control runs in the ICLManyReplication artifact. General music-generation behavior has not been characterized here.
- Downloads last month
- -
Model size
50.6M params
Tensor type
F32
·
Hardware compatibility
Log In to add your hardware
Quantized