Source-linked AI summary
mu-bench: A Multilingual Utterance Transcription Benchmark
Andrea Li, Soham Ray
TL;DR
ASR is often evaluated on read, English-centric speech with WER, which can penalize surface differences rather than meaning. mu-bench introduces a multilingual phone-call benchmark with reference-guided normalization and an LLM judge; among evaluated providers Mandarin is hardest, and UER on normalized text agrees better with human judgment than WER.
Problem
ASR evaluation often uses read, English-centric speech and WER, which penalizes surface differences even when meaning is preserved.
Source: §1 · p. 1, §2.2 · p. 1Method
mu-bench provides multilingual phone-call utterances and pairs reference-guided normalization with an LLM judge that scores whether utterances preserve meaning.
Source: §1 · p. 1, §3.2 · p. 2, §3.3 · p. 2Results
Mandarin lags all other locales for the evaluated providers, and UER on LLM-normalized text agrees best with human judgment.
Source: §5.3 · p. 4, §5.1 · p. 4Takeaways & Limitations
Provider accuracy and latency do not align, so the fastest provider is not necessarily the most accurate on this benchmark.
Source: §5.3 · p. 4The study is limited to five locales and banking; expansion requires checking normalization and UER quality against each language’s structure.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin, centered on form-field inputs such as names, email addresses, and confirmation codes. We release Utterance Error Rate (UER), an LLM judge of whether a transcript preserves meaning, calibrated against human raters, together with an LLM normalizer that makes WER comparable across providers' output formats. On 1,847 human-rated transcripts, UER agrees with annotators at $\kappa$ = 0.78, versus 0.53 for exact-match WER on normalized text. We rank six commercial providers on a public leaderboard; the best reaches 11.9% UER, and Mandarin is hardest for all six.
1. INTRODUCTION
The paper introduces mu-bench to evaluate speech recognition in multilingual phone interactions with an AI banking agent. It contributes a dataset, evaluation tools, and a public leaderboard.
1. INTRODUCTION
- mu-bench benchmarks phone conversations with an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin.The dataset focuses on caller utterances.Source: §1 · p. 1
- The benchmark releases Utterance Error Rate (UER), an LLM judge calibrated against human judgment, and an LLM normalizer for aligning text.The normalizer makes WER comparable across providers’ output formats.Source: §1 · p. 1
- A public leaderboard lets providers submit transcripts and compare models.Source: §1 · p. 1
2. RELATED WORK
Existing ASR benchmarks do not combine multilingual, conversational, and form-field-input coverage. The section also describes how surface-based WER and text normalization complicate evaluation of meaning-preserving transcripts.
2.1. Benchmarks and leaderboards
- Read-speech corpora are multilingual but not conversational, conversational corpora are English-only, and multilingual conversational datasets do not target input collection.The missing combination matters for calls collecting names, account numbers, addresses, and confirmation codes, where a misheard token is consequential.Source: §2.1 · p. 1
2.2. Metrics and text normalization
- WER gives equal weight to every word edit, so “this is the cat” and “this is a cap” both score 25% against “this is a cat,” although only the latter changes meaning.Source: §2.2 · p. 1
- The section describes the Whisper normalizer as English-specific, with only basic handling elsewhere, and introduces an open LLM normalizer for multilingual and form-field text.It also notes that semantic WER variants include embedding-based SemDist and LLM-based LASER [14].Source: §2.2 · p. 1
3. METHODS AND APPROACH
The methods construct a multilingual phone-call dataset, normalize transcripts against their references, and use an LLM judge to measure significant utterance-level errors across five locales.
3.1. Dataset
- Native speakers called an AI banking agent using their own phones and environments, and professional annotators produced and independently verified clean verbatim transcripts with timestamps.Caller turns flagged unintelligible are excluded; only caller-side audio is released.Source: §3.1 · pp. 1–2
3.2. Language-aware normalization
- Reference-guided normalization reformats a prediction into the reference’s rules to score provider outputs fairly, using the reference to resolve one-to-many spoken-form mappings.Rewriting toward the reference can hide a real error; the paper evaluates this leak and compares reference-blind normalization.Source: §3.2 · p. 2
3.3. Utterance Error Rate
- UER gives significant errors a score of 1, meaning-preserving word differences a score of 2, and semantically equivalent utterances a score of 3; it is the unweighted mean of significant-error rates across five locales.The judge receives the normalized prediction and reference because WER can penalize semantically equivalent wording.Source: §3.3 · p. 2
4. EXPERIMENTAL SETUP
The experiments evaluate commercial speech-recognition APIs under batch and streaming settings, measure end-of-speech latency, and compare model configurations and human ratings.
4.1. Evaluated providers and settings
- Six commercial APIs are tested through batch endpoints and three through streaming, using 8 kHz mono PCM16 audio and default settings apart from a language hint.Batch sends one utterance per request; streaming sends 20 ms frames.Source: §4.1 · p. 2, Table 5 · p. 3
4.2. Latency measurement
- Latency is wall-clock time from end of speech to final transcript, measured at concurrency 1 and reported at p50 and p95.For streaming, the clock starts after the last frame; time-to-first-token is excluded so modes remain comparable.Source: §4.2 · p. 2
4.3. Model
- The model comparison varies the normalizer and judge one at a time over five seeded runs.The table reports precision, recall, cost per 1,000 rows, and the share of verdicts that change between seeds.Source: Table 4 · p. 3, §4.3 · p. 3
4.1. Evaluated providers and settings
- The provider-results table reports UER and includes a separate input-collection UER, while the normalization-failure table classifies semantically equivalent outputs reconciled only by mu-bench.The former is sorted by UER; the latter counts utterance-provider pairs in each failure class.Source: Table 5 · p. 3, Table 6 · p. 3
4.4. Human annotation study
- Native speakers rated sampled transcript rows against references as no error, minor error, or significant error, with provider names hidden and row order randomized.The study yielded 1,847 rated rows.Source: §4.4 · p. 3
5. RESULTS
The results show that UER aligns better with human judgments than WER, while provider accuracy varies by locale and endpoint. Form-field inputs show a marked gap between WER and UER, and padding silence does not materially improve UER.
5.1. Which normalization and evaluation metric agree best with human judgment?
- UER on LLM-normalized text agrees best with human judgment; WER>0 has recall near 1 but precision of 0.19–0.46, flagging surface differences as errors.The normalizer’s UER gain is within noise (0.779 vs. 0.764; paired difference 0.015, 95% CI [−0.010, 0.044]), while it makes WER usable (0.53 vs. 0.10).Source: §5.1 · pp. 3–4
5.2. How much does the underlying LLM model matter?
- 0.09 κ is lost when the judge changes from gpt-6-astra to gpt-5.6-luna (0.774 to 0.682), compared with a 0.02 loss for the normalizer swap (0.775 to 0.755).Smaller judges have lower recall (0.58 and 0.60 vs. 0.76), while matching or exceeding the larger judge’s precision; pooled κ standard deviations are at most 0.011.Source: §5.2 · p. 4
5.3. How do frontier providers perform?
- English is strongest and Mandarin lags all other locales; Deepgram has the lowest latency but highest UER, while Google has the highest median latency and lowest UER.Streaming and batch transcripts differ for many utterances, and switching modes moves macro UER by 2–4 points in either direction (Google +4.4, OpenAI −3.1).Source: §5.3 · p. 4
5.3.1. Statistical validity
- 32 of 36 provider pairs are separated at p < 0.05 and 27 at p < 0.001; all three providers run in both modes show batch–stream differences (p ≤0.006).Despite transcript changes on 20% of identical ElevenLabs requests and 45% of identical OpenAI requests, UER moves by at most 0.4 pp and the leaderboard is stable under resends.Source: §5.3.1 · p. 4
5.4. How do form-field inputs affect accuracy?
- Input-collection utterances have lower normalized WER than other utterances (15.0% vs. 18.7%) yet higher UER (28.6% vs. 17.0%) across six batch providers.They comprise 1,706 of 4,270 utterances (40%) and are unevenly distributed across locales; Google leads on non-collection speech (6.4% vs. 17.7% on collected values), while ElevenLabs is best on input collection and OpenAI and Deepgram are worst at about 34%.Source: §5.4 · p. 4
5.5. What failures does LLM normalization catch?
- 49.3% of raw predictions that differ from the reference are reconciled by mu-bench normalization, versus 23.2% by Whisper normalization.On human-rated rows, mu-bench rewrites 12 of 263 significant errors (4.6%) into the reference; judging raw text changes no provider’s UER by more than 0.6 pp and leaves rankings unchanged.Source: §5.5 · p. 4
5.6. Does padding the audio with silence improve quality?
- Prepending 1 or 3 s of silence changes 18–38% of transcripts but moves UER by at most 2.1 pp for Azure and within 1 pp for other providers, while adding latency.mu-bench therefore clips short audio without padding.Source: §5.6 · p. 4
6. FUTURE WORK
Future work identifies three expansion directions: more locales, contexts beyond banking, and additional provider settings.
6. FUTURE WORK
- More locales are a planned expansion, with native speakers needed to review normalization and UER quality against each language’s structure.Source: §6 · p. 4
- Future work will extend evaluation to domain contexts beyond banking.Source: §6 · p. 4
- Planned provider-setting evaluations include dynamic keyword boosting, prompting, and noise suppression.Source: §6 · p. 4
8. COMPLIANCE WITH ETHICAL STANDARDS
The study involved compensated human participants who consented to recording and public release; callers role-played fictional customers, and data collection followed the Helsinki Declaration and amendments.
8. COMPLIANCE WITH ETHICAL STANDARDS
- Native speakers were hired and compensated to place calls, transcribe recordings, or rate transcript similarity.Source: §8 · p. 5
- Callers consented to recording and public release, and used invented customer details, leaving no real personal data in released audio or transcripts.Source: §8 · p. 5
- Sierra AI had no formal ethics review committee; the study followed the 1964 Helsinki Declaration and later amendments.Source: §8 · p. 5