mlboydaisuke/LFM2.5-1.2B-Instruct-ExecuTorch

LFM2.5-1.2B-Instruct — ExecuTorch XNNPACK 8da4w

lfm2_5_1_2b_xnnpack_8da4w.pte (741 MB)

• Source: LiquidAI/LFM2.5-1.2B-Instruct (hybrid conv/attention)
• License: LFM Open License v1.0
• Quantization: 8da4w (8-bit dynamic activation / 4-bit weight) + 8-bit embedding (embedding_quantize: "8,0"; cuts 1143 MB → 741 MB vs the fp32-embedding v1)
• Export: executorch 1.4.0 export_llm, dynamic shape, max_seq_length 2048, XNNPACK extended_ops
• Config: llm_params/lfm2_5_1_2b_xnnpack_8da4w_e8.yaml

Verification (2026-08-13)

Mac gate (greedy via native.py, chat template): correct 2-sentence Rayleigh-scattering answer, 170.8 tok/s on M-series Mac (reference only). v1 (fp32 embedding) passed 3/3 (Paris / Japanese / haiku) with identical quant settings otherwise.

iPhone 17 Pro / iOS 27 (ETBench, XNNPACK CPU, default threads), re-measured 2026-08-14 on this 8-bit-embedding build:

metric	value
load	0.6 s
ttft (short prompt)	0.05-0.06 s
decode	65-86 tok/s (86 short answer, 65 at 128 tokens)

Outputs correct (Paris; coherent 128-token story). The earlier 1143 MB fp32-embedding build loaded in 1.6 s and decoded 55-81 tok/s, so quantizing the embedding table cut both the file and the load time without costing throughput.

Usage note — chat template is required. This is an instruct model: raw untemplated text makes it emit <|im_end|> immediately (looks like broken generation but is not). Always wrap prompts as <|startoftext|><|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n, eos ids [7].

Core ML build (Neural Engine, iOS 18+)

lfm2_5_1_2b_coreml.pte (2.35 GB, no quantization)

The same model on ExecuTorch's Core ML delegate. That path did not run until now: a buffer written a step after it is read was taken for constant data and handed to the delegate, Core ML compiled it into a state, and the runtime — told by take_over_mutable_buffer=False that it has none — failed at execute on layers_N_conv_conv_state. Fix in pytorch/executorch#21979.

The .pte needs no patched runtime. The fix is export-side; this file was verified on stock executorch 1.4.0 from pip.

Verification (Mac arm64, 2026-08-21)

Greedy next-token argmax against LiquidAI/LFM2.5-1.2B-Instruct in fp32 eager, five prompts, every position after the second counted — 27 of 27 agree:

prompt	Core ML	eager
The capital of France is	Paris	Paris
The largest planet in our solar system is	Jupiter	Jupiter
Shakespeare wrote a play called Romeo and	Juliet	Juliet

Decode 68 tok/s on the Mac, median of 16 steps after four warm-up steps.

Reset the cache between sequences. The KV cache persists across execute calls, so a second prompt started at position 0 reads the first one's keys. Load a fresh method per sequence.

Why no quantization

coreml_quantize: c4w runs but loses accuracy on this path — measured on the 350M sibling, 21/27 against 27/27 unquantised. The XNNPACK 8da4w file above stays the small build.