Source-linked AI summary
FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speech
Wei Chu, Yuanzhe Dong, Ke Tan, Dong Han, Yichao Zhou, Ruchao Fan, Bingshen Mu, Jingbei Li, Vishwas Shetty, Sarthak Bisht, Ziyue Qiu, Massa Baali, Rita Singh, Bhisha Raj
TL;DR
Published forced-alignment results are difficult to compare because studies use different normalization, data splits, and boundary matching. FA-Bench fixes these choices in a shared benchmark and scores boundaries with adjacent-label checks. Its results show that clean-speech MAE rankings do not predict robustness and that word timestamps can carry systematic offsets.
Problem
Forced-alignment comparisons use different transcript normalization, data splits, and boundary matching, often on clean speech, making published numbers difficult to compare.
Source: §1 · p. 1Method
FA-Bench fixes evaluation choices and scores every boundary using adjacent-label matches, while grouping boundaries by position and match status.
Source: §1 · p. 1, §2.3 · p. 3Results
Clean-speech MAE rankings do not predict robustness, and current systems show systematic word-timing offsets that absolute error does not reveal.
Source: §3 · pp. 3–4, §4 · p. 4Takeaways & Limitations
Grouping boundaries by location and adjacent-label matches turns an overall score into information about where systems lose accuracy.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Forced alignment estimates the timestamps of each word, phone or character in speech given its transcript. Published comparisons normalize transcripts, split the data and match boundaries differently, so their numbers cannot be read together. We present FA-Bench, an open framework that fixes those choices once and releases the code, splits, phone mapping, text normalization and scoring script, with results published periodically. Track 1 gives every aligner the reference transcript and Track 2 gives it a recognizer's output, on the same audio, clean and degraded four ways, with 21 open models and 9 commercial APIs under a unified protocol. We score every boundary of an utterance and check the two labels beside it, so a word the recognizer missed or invented is charged. Using a tolerance-based F1 as our primary metric eliminates the 9% to 14% score inflation that standard MAE causes on recognition-dependent systems in conversational speech. We then group boundaries by position and how many adjacent words were recognized correctly, which shows where a system lost the score. We discovered systematic bias in how current systems time words, with Whisper about 150 ms early and several commercial ASR APIs over 50 ms late. Code and results are at https://github.com/olewave/fa-bench
1. INTRODUCTION
Published alignment results are difficult to compare because studies use different transcript normalization, data splits, and boundary matching, often on clean speech. FA-Bench provides a shared framework for evaluating forced alignment and timestamped ASR.
1. INTRODUCTION
- Existing results cannot be read against each other because studies differ in transcript normalization, data splits, and boundary matching, and most use clean speech.Some evaluations use another aligner’s boundaries as ground truth, and Buckeye studies choose their own splits.Source: §1 · p. 1
- FA-Bench evaluates word and phone forced alignment as well as timestamps from ASR systems and forced alignment using ASR transcripts.The benchmark also examines systematic bias in current systems’ word start and end times.Source: §1 · p. 1
- FA-Bench releases code, splits, phone mapping, normalization, and scoring to support reproducible comparisons and periodic benchmark results.Its harness can also run on developers’ proprietary data.Source: §1 · p. 1
- Its scoring checks both labels beside every boundary, addressing evaluations that can overlook misread units or dropped words.The paper notes that some evaluations skip utterance edges or score only matched words or word ends.Source: §1 · p. 1
- FA-Bench groups boundaries by position and adjacent-label matches to show where ASR timestamps lose score.This distinguishes recognition errors from poor timing of correctly recognized words and separates utterance edges from interior boundaries.Source: §1 · p. 1
2. THE BENCHMARK
FA-Bench publishes system metrics in a shared record and evaluates word and phone tiers under two tracks that address different questions.
2. THE BENCHMARK
- Every system’s metrics are published in records/1.Source: §2 · p. 2
2.1. Tasks
- FA-Bench evaluates word and phone tiers in Track 1, which measures timing alone, and Track 2, which measures timing plus recognition errors.The two tracks answer different questions.Source: §2.1 · p. 2
2.2. Data
FA-Bench uses hand-marked word and phone boundaries from TIMIT and Buckeye, holds evaluation speakers out, and defines transcript handling and scoring choices. Its scoring includes every reference boundary and uses adjacent labels to determine hits.
2.2. Data
- TIMIT and Buckeye provide hand-marked word and phone boundaries, with Buckeye phone tiers annotated from audio rather than expanded from dictionaries.In Buckeye, canonical and realized pronunciations differ for 58.7% of word tokens.Source: §2.2 · p. 2
- All evaluation speakers are held out; Buckeye’s custom split preserves sex×age structure, with six speakers per cell for training and two each for development and test.TIMIT uses the corpus core test set and Kaldi’s development split.Source: §2.2 · p. 2
- Buckeye event tags retain a hidden spoken word when its phone tier contains speech, and aligner outputs are mapped back to input tokens when their letters match.TIMIT .WRD transcripts are lower-cased without further modification.Source: §2.2 · p. 2
- The benchmark adds recorded MUSAN noise, music, and babble, plus simulated room impulse responses, following the Kaldi VoxCeleb recipe.Source: §2.2 · p. 2
2.3. Scoring
- Phones are mapped to TIMIT-39 for comparison without moving boundaries; the only dropped label is the glottal stop.The glottal stop accounts for 1.6% of TIMIT phones and none of Buckeye’s.Source: §2.3 · p. 2
- Track 2 MAE uses only boundaries matched after aligning recognized and reference words, so recognition errors can leave reference boundaries out of the calculation.For Qwen3-ASR on noisy Buckeye, 13.5% word errors leave MAE computed on 83% of reference boundaries.Source: §2.3 · p. 2
- Boundary F1 counts a hit only when both adjacent labels match and the time is within tolerance; unlike MAE, recall includes every reference boundary.A misread word costs the two boundaries beside it, while an empty utterance costs all its boundaries.Source: §2.3 · p. 3
- Grouping boundaries by location and adjacent-label matches supports analysis of recognition errors, utterance edges, and boundaries whose two labels are wrong.Source: §2.3 · p. 3
3. RESULTS
Results expose weaknesses in clean-speech MAE rankings and systematic timestamp offsets, while re-aligning ASR output improves 20 ms F1 over bundled alignment. Grouping boundaries reveals how recognition errors and utterance position relate to alignment accuracy.
3. RESULTS
- Third on clean TIMIT word MAE, MAPS ranks last under degradation; four of fourteen aligners shift at least three places on TIMIT and seven on Buckeye.F1 rankings barely move because gross errors inflate MAE but count as only one miss for F1.Source: §3 · p. 3, Table 1 · p. 3
- On TIMIT, degradation shifts PER by at most 0.3 points for six of eight systems, while phone MAE moves by up to 82 ms.Clean-audio PER of 26–36% measures dictionary phones against hand transcription; only 43–54% of TIMIT phone boundaries have both neighboring labels matched.Source: §3 · p. 3, Table 2 · p. 4
- Olign’s interior word MAE is 9.3 vs 14.3 ms clean and 15.0 vs 19.7 ms degraded against MFA, but noisy TIMIT MAE is 33.0 vs 30.2 ms.Averaged over clean and degraded audio, Olign overshoots by 15 ms and Google Chirp 2 by 56 ms.Source: §3 · pp. 3–4
- 40%: Google Chirp 2 followed by Olign improved 20 ms F1 on average over four test cells; Track 2 F1 is higher for M2I than M1I boundaries (0.36 vs 0.25).Recognition errors lower alignment scores, while boundaries whose adjacent words are both recognized are timed more accurately.Source: §3 · p. 4, Table 3 · p. 4
- On Buckeye test, utterance-end F1 is 0.18 below beginning-boundary F1 and 0.17 below internal-boundary F1 across Track 2 systems.Table 3 groups boundaries by utterance position and adjacent-label matches.Source: §3 · p. 4, Table 3 · p. 4
- Whisper is about 150 ms early; seven of eight commercial ASR APIs start late, with three more than 50 ms late.Whisper’s durations are nevertheless right to within 16 ms, so a duration check would pass despite the offset.Source: §3 · p. 4
- 80 ms timestamp steps leave only half of reference boundaries within 20 ms of a grid point, capping F1 near 0.5; Qwen3-FA scores 0.40.The passage attributes Qwen3-FA’s score to this coarse-grid cap.Source: Table 1 · p. 3, §3 · p. 4
4. CONCLUSIONS
FA-Bench provides a common evaluation setting for forced alignment and ASR timestamps, fixing transcript normalization, held-out splits, and phone inventory while releasing scoring code. Its results show that word error rate alone does not determine timestamp accuracy, clean-speech MAE rankings do not survive degradation, and word boundaries carry offsets that absolute error misses.
4. CONCLUSIONS
- FA-Bench fixes transcript normalization, held-out splits, and phone inventory, and releases scoring code for evaluating forced alignment and ASR timestamps.Grouping boundaries by utterance position and adjacent-label matches makes it possible to identify where systems lose score.Source: §4 · p. 4