Ayah and word timestamps for Quran audio: benchmarking forced alignment models
Highlighting the ayah while audio plays, cutting a long recording into ayah files, building a word-by-word player: all need timestamps, the time each ayah (or word) starts and ends. Doing it by hand takes hours per recording. This post describes how we benchmark automatic alignment and where our models stand.
Forced alignment in one paragraph
When the text is known, you do not need a model to transcribe the audio, only to say how well each piece of audio matches each piece of text. A CTC speech model outputs, for every 40 ms slot, the probability of each letter or phoneme. Forced alignment finds the path through the known text that best explains those probabilities (the Viterbi algorithm). The path's boundaries are the timestamps.
Recitation adds complications general alignment tools do not expect: long pauses, repeated ayahs, an imam going back two ayahs, a basmala or isti'adha before the surah, a takbir in prayer. A good aligner must handle repeats rather than assume the audio contains the text exactly once.
The benchmark
- 31 clips, about 1,240 ayahs, with ayah boundaries reviewed by hand in a listening tool.
- The clips mix clean studio recitation, phone recordings (including rain, whispering and fast recitation), distant microphones and long sessions with repeats.
- The score: for each ayah, are both its start and its end within 250 ms of the reviewed truth? Within 500 ms? Requiring both ends is stricter than averaging them.
Results
| Aligner | Both ends within 250 ms | Both within 500 ms |
|---|---|---|
| Naml Letters, layer 8, Viterbi alignment | 51% | 68% |
| wav2vec2, Viterbi alignment | 43% | 62% |
| wav2vec2, qalign (production pipeline) | 33% | 55% |
Naml Letters is a 17 MB streaming model that runs on a phone CPU; the wav2vec2 models are larger offline models. We have not isolated why the small model aligns better; TODO: an ablation (layer, decoding, training audio) before we explain it.
What the numbers do not say
- Half the ayahs are still outside 250 ms. For a highlight that follows audio, 250-500 ms is usually invisible; for cutting audio files, a boundary in the middle of a breath matters, and a pause-snapping step after alignment helps.
- Some groups are hard for every model. In earlier rounds of this benchmark female reciters, distant microphones and rainy phone recordings had the widest spread between models.
- Models lag consistently. Streaming models emit a letter slightly after it is heard. A fixed offset per model, measured once, removes most of it; our scores include that calibration.
- The truth was partly started from model output and corrected by hand, which can favour the model it started from on those clips. We note this rather than hide it.
Word timestamps
The same path gives word boundaries. We use them in our tajweed analysis, which aligns each word to its expected tajweed units, and in the live tracker. TODO: publish a word-level score on the same clips.
Running your own comparison
If you are choosing an aligner for your own library, a small benchmark is worth the afternoon it takes:
- Pick clips like yours. Studio murattal, masjid recordings and phone recordings behave differently; a model that wins on one can lose on another.
- Review the truth by ear. Mark each ayah's first and last sound, not the silence around it. A listening tool with waveform and keyboard shortcuts makes this a few minutes per clip.
- Score both ends. An aligner that gets starts right and ends wrong will cut audio files badly; averaging hides it.
- Calibrate each model once. Measure its median signed error on a few clips, subtract it, then score the rest.
- Look at the tails. The 90th percentile error tells you how often a user will see a visibly wrong highlight.
Using it
The Naml models and the wasm core are free under CC BY 4.0. The runner returns per-slot log-probabilities
(lastLogProbs) for your own Viterbi alignment. The tracker code contains our repeat-aware alignment and the node
pipeline we use to score it. If you need a recording library aligned for you, ask us.