---
license: apache-2.0
base_model:
  - openbmb/BitCPM-CANN-8B
tags:
  - core-ai
  - coreai
  - on-device
  - iphone
  - apple-silicon
  - ternary
  - bitnet
  - 1.58-bit
  - minicpm
language:
  - en
  - zh
pipeline_tag: text-generation
library_name: coreai
---

# BitCPM-8B → Apple Core AI (1.58-bit ternary, on-device)

The zoo's **first 1.58-bit ternary LLM** and **first sub-int8 packed-GEMM Metal kernel**, running
fully on-device on **iPhone** (and Mac) through Apple **Core AI** (`.aimodel` / `.aimodelc`, iOS 27 /
macOS 27).

An 8B model whose transformer weights are just **{-1, 0, +1}** — so it generates at a **3–4B-class
footprint and speed** while keeping 8B-class quality.

- **Base:** [openbmb/BitCPM-CANN-8B](https://huggingface.co/openbmb/BitCPM-CANN-8B) (MiniCPM4-8B
  architecture, quantization-aware trained to ternary), Apache-2.0.
- **Zoo + conversion code:** https://github.com/john-rocky/coreai-model-zoo

## On-device (iPhone 17 Pro, A19 Pro — CoreAIChat pipelined GPU engine, greedy)

| bundle | decode | prefill | resident | load |
|---|---:|---:|---:|---:|
| **`gpu-pipelined/` (AOT h18p)** | **17 tok/s** | **13 tok/s** | **~2.1 GB** | 9 s cold |

Headroom ~4.3 GB, no jetsam. An int4 8B would need ~5–6 GB resident — the ternary weight stream is
the win. On **M4 Max** the same graph decodes **62.7 tok/s** and is **token-identical** to the torch
ternary reference (3/3 probe prompts, greedy).

## What's ternary here

`BitCPM-CANN-8B` ships its ternary weights as **TQ2_0** (2 bits/weight): per 256-element block along
the reduction axis, each weight is a code in {-1, 0, +1} times one fp16 scale. The **224 transformer
linears** (q/k/v/o + gate/up/down × 32 layers) run a custom Metal kernel that packs **16 ternary
codes into one uint32** and does a sign-add/subtract matvec with the per-block scale — **no codebook**.
The embedding (Q4_K) and LM head (Q6_K) stay higher-precision. Quality retained vs full precision:
**95.7–97.2%** (OpenBMB).

## Run

In the zoo's **CoreAIChat** app (Model → "BitCPM-8B 1.58bit"), or via Foundation Models:

```swift
import FoundationModels
import CoreAILanguageModels
let model = try await CoreAILanguageModel(resourcesAt: bundleURL)   // gpu-pipelined/ AOT h18p
let session = LanguageModelSession(model: model)
print(try await session.respond(to: "The capital of France is"))   // -> "Paris."
```

Decode-only static bundle: set `COREAI_CHUNK_THRESHOLD=1` (prefill runs as pipelined S=1 steps). Chat
is ChatML; eos `<|im_end|>` (73440). The `gpu-pipelined/` bundle is **AOT-compiled** for the h18p GPU
(`xcrun coreai-build compile … --preferred-compute gpu --architecture h18p`) — a custom-Metal-kernel
graph survives AOT (outputs bit-identical to the source IR). ANE is not supported (GPU-only kernel).

## License

Apache-2.0, inherited from [openbmb/BitCPM-CANN-8B](https://huggingface.co/openbmb/BitCPM-CANN-8B).
This repository redistributes a converted Core AI artifact only.