WebAssembly SIMD vs WebGPU for on-device speech recognition: lessons from phones
When we moved our Quran speech models into the browser, the obvious plan was WebGPU: a GPU is built for matrix multiplies, and WebGPU finally exposes it to web pages. On desktops it worked as expected. On the phones our users actually have, it was more than ten times slower than the CPU. This post shares the numbers and the rule we ended up with.
The workload
Our models (Naml Tajweed, Letters and Letters Plus) are streaming: every 0.32 seconds they process 4 new frames of audio with a cached context of 70 frames. Each step is small, a few million multiply-adds per block, and latency-bound. That shape matters: a GPU shines on big batches, and a streaming step is not one.
The measurements
Per 0.32 s step of Naml Tajweed (8 FastConformer blocks, 2-bit ternary weights):
| Device | Path | Per step |
|---|---|---|
| Budget Android phone (4 GB, Mali Valhall GPU, Chrome) | our WebGPU core | 226-238 ms |
| same phone | our wasm core, relaxed SIMD (r8) | 18 ms |
| same phone | our wasm core, plain SIMD128 (i16) | 71 ms |
| same phone | ONNX Runtime wasm, 8-bit model | 122 ms |
| MacBook (Apple M4 Pro, Chrome) | our WebGPU core | 1.8-1.9 ms |
| same Mac | ONNX Runtime WebGPU, 8-bit model | 6.8-11.2 ms |
| same Mac | our wasm core (r8) | a few ms |
| iPhone, Safari | our wasm core (i16) | ~15 ms |
| Server, one Hetzner CPU core, Node | our wasm core (r8) | ~15 ms |
A step's budget is 320 ms. Anything under a few tens of milliseconds is comfortably real time; 230 ms on the Mali phone technically fits but leaves the phone almost no room for the tracker, the page and everything else.
Why the phone GPU loses
Several things add up on a mobile GPU for this workload:
- Dispatch overhead. A step is dozens of small kernels. On a phone GPU, the fixed cost of each dispatch and synchronisation is large compared with the arithmetic inside it.
- Small batches. Four frames per step does not fill a GPU. The CPU's SIMD units, by contrast, are fully used by a matrix-vector product of this size.
- Bandwidth and transfers. If the attention and convolution caches went back and forth between CPU and GPU each step it would be about 2.3 MB each way; we keep them on the GPU, but the final log-probabilities still have to come back every step.
- No int8 dot products. Relaxed SIMD gives WebAssembly 8-bit dot-product instructions; our r8 kernel multiplies int8 activations by ternary weights with them. That is a large part of why r8 is four times faster than the i16 kernel on the same phone.
On Apple Silicon the GPU is fast enough that all of this hardly matters, and both paths finish in a few milliseconds.
Safari and iOS
iOS Safari does not ship relaxed SIMD, so the r8 kernel is not available there; the i16 kernel runs instead, at about 15 ms per step on an iPhone. We also found that starting the WebGPU runtime crashed on iPhone and iPad in our tests, so iOS always runs the wasm path.
The rule we ship
- If the browser has relaxed SIMD (current Chrome and Edge, including Android): use the wasm core with the r8 kernel. Never WebGPU.
- Else, if it has WebGPU (except iOS): use our WebGPU core.
- Else, if it has SIMD128 (Safari on iOS, browsers without WebGPU): the wasm core with the i16 kernel.
- Else: ONNX Runtime on wasm, the slow fallback.
The rule is one function in the app, so it can change as browsers do. A later improvement would be to time one step of each path at load and keep the faster.
Lessons
- Measure on the phone your users have, not on your laptop. Our development machine said WebGPU; the phone said otherwise.
- A general runtime is a starting point. Writing our own kernels for ternary weights made the CPU path about 7 times faster than ONNX Runtime's 8-bit wasm path on the same phone.
- Streaming changes the maths. Small, frequent steps favour the CPU's SIMD units over a GPU's throughput.
The wasm core and the models are free to download under CC BY 4.0, with a minimal JavaScript runner. How we got the weights to 2 bits is in the post on ternary models.