Ternary (2-bit) speech models: running a Quran ASR model in 17 MB
A speech model for a phone has to be small twice over: small to download over a mobile connection, and small enough in compute to run several times a second without draining the battery. Our Quran models store every weight in 2 bits. This post explains what that means, how it is run, and what it costs.
The size problem
The models are cached streaming FastConformers with 8 or 10 blocks. In 32-bit floats, the 8-block cores are about 197 MB and the 10-block core about 250 MB. In 8-bit form they are 53 and 70 MB. That is still a lot to ask a visitor to download before a web page can listen.
The ternary files are 16.8 MB for Naml Tajweed, 16.6 MB for Naml Letters and 24.5 MB for Naml Letters Plus, plus 5.9 MB of front end shared by all three.
Ternary weights
Each weight is one of three values, −1, 0 or +1, multiplied by a scale. There is one scale per block of 128 inputs of a layer, so the scale adapts to each part of the weight matrix while the weights themselves need only 2 bits. A ternary matrix multiply is mostly additions and subtractions; the zeros are skipped.
This is not quantisation after the fact. A float model squeezed to 2 bits loses badly. The ternary model is trained (distilled from a larger teacher) with the ternary constraint in place, so it learns weights that work at that precision.
Running it: our own kernels
General runtimes are not built for ternary weights. ONNX Runtime's 8-bit path on the same model, in WebAssembly, takes about 122 ms per 0.32 s step on a 4 GB Android phone. Our own WebAssembly module, written in Rust, does the front end and the core in one call with hand-written kernels:
| Kernel | What it does | Budget Android phone, per step |
|---|---|---|
| r8 | int8 activations × ternary weights, relaxed-SIMD 8-bit dot products | 18 ms |
| i16 | the same integer maths with plain SIMD128 16-bit dot products | 71 ms |
| ORT wasm, 8-bit | ONNX Runtime on the 8-bit model | 122 ms |
On one CPU core of our server the r8 kernel takes about 15 ms per step for Naml Tajweed and Naml Letters and about 19.5 ms for Letters Plus: real-time factors of 0.047, 0.048 and 0.061. An iPhone in Safari runs it in about 15 ms per step.
The model only does work for new audio: the attention keys and values and the convolution state of the last 70 frames (5.6 s) are cached inside the module between steps.
What it costs in accuracy
Ternary is not free. The real question is whether the smaller model still does its job, and for us the job is tracking and timing, not dictation.
- Timing: on 31 reviewed clips (about 1,240 ayahs) Naml Letters puts an ayah's start and end both within 250 ms for 51% of ayahs, ahead of the wav2vec2 aligners we compared (details).
- Agreement with the 8-bit model: the wasm core's int8 activations are not bit-identical to the 8-bit ONNX model, but in our checks the two pick the same output on 99.8% of frames, and word times differ by a median of 0 ms.
TODO: publish word error rates of the ternary models next to their float teachers on a test set whose audio we may share.
Why this matters beyond Quran apps
On-device speech recognition is usually presented as a trade-off between quality and size. Ternary weights move that curve: a domain-specific streaming model that fits in the size of a few photos and runs at a few percent of a phone CPU. For a fixed domain like the Quran, where the tracker supplies the text and the model only has to provide evidence, it is enough.
The models, the wasm core and a small JavaScript runner are free on Hugging Face under CC BY 4.0. The next post compares this CPU path with WebGPU on real phones.