ASR4REAL: An extended benchmark for speech models

1. INTRODUCTION

• Public datasets overrepresent clean read speech by native US English speakers, while spontaneous conversations, noise, and diverse accents are comparatively rare.Many publications also evaluate performance on few datasets without considering out-of-domain resilience or model-size efficiency.Source: §1 · p. 1
• The benchmark tests models under increasingly difficult real-world conditions, focusing here on accented, rehearsed, and spontaneous speech.The authors describe it as an intermediate benchmark for identifying model strengths and weaknesses and helping researchers find methods suited to production systems.Source: §1 · p. 1
• The benchmark reports a small gender accuracy gap, larger accent and socio-economic-status disparities, and a substantial performance drop on spontaneous speech.It also finds that current language models are not adapted to spontaneous speech; Librispeech is described as no longer sufficiently representative.Source: §1 · p. 1

Robustness

• Robustness work includes smart data augmentation to reduce domain-transfer gaps, multi-task training, and distillation for transfer learning.The section notes that CHiME Challenge WER is about twenty times higher than Librispeech WER for state-of-the-art models.Source: §2 · p. 1

ASR benchmark

• Benchmark research surveyed ASR use cases without testing academic models, while a real-life evaluation did not examine accent, gender, or social-status gaps.Source: §2 · p. 1

Fairness

• Fairness studies identified performance gaps between white and African American speakers and called for disability representation in training data.Source: §2 · p. 1

3.1. Our reference dataset: Librispeech

• Librispeech is a reference dataset for ASR training and testing, built from filtered and segmented LibriVox audiobooks into clean, short sequences with accurate transcripts.Its training set contains 960 hours of labelled audio split into clean and other categories, each with development and test sets.Source: §3.1 · pp. 1–2

3.2. Read speech with accent : ALLSSTAR and NISP

• ALLSSTAR includes speakers reading the same text, reducing grammar and vocabulary variation; NISP contributes English read recordings by Indian speakers from five mother-tongue groups.ALLSSTAR labels gender and mother tongue; the study uses only NISP's English section.Source: §3.2 · p. 2

3.3. Improvised speech with accent: VoxPopuli

• VoxPopuli supplies rehearsed and improvised political speech; metadata was used to select English speakers who spoke exactly one additional language, yielding 23 European accents.The study assumes that additional language is the speaker's mother tongue and that these speakers are non-native English speakers.Source: §3.3 · p. 2

3.4. Conversational Speech: Buckeye and CORAAL

• Buckeye and CORAAL both contain two-person conversations, with Buckeye speakers described as students and CORAAL speakers as African Americans from diverse social backgrounds.Source: §3.4 · p. 2

4. MODELS

• The benchmark compares open-source systems and Google's enhanced-video speech-to-text API without retraining, using each model's recommended language model and decoding parameters.Table 1 reports model parameter counts, pretraining and fine-tuning data sizes, and speaker-normalized median Librispeech WER.Source: §4 · p. 2, Table 1 · p. 2

4.1. SpeechBrain: an accessible ASR toolkit

• SpeechBrain contributes a Librispeech-trained Transformer and CRDNN, each combined with a Transformer language model trained on Common Crawl.SpeechBrain is described as an open-source toolkit covering speech recognition, speech enhancement, and other tasks.Source: §4.1 · p. 2

4.2. Wav2Vec 2.0 and pretraining

• The evaluated wav2vec 2.0 models were pretrained on audiobooks and fine-tuned on Librispeech, with a Librispeech-trained 4-gram language model used for decoding.The passage motivates pretraining by noting that it can achieve strong Librispeech results with little labelled audio and transfer across languages.Source: §4.2 · p. 2

4.3. RASR: increasing ASR’s models robustness

• The study evaluates RASR small, RASR big, and RASR small-distill, using a Common Crawl 4-gram language model and the release's decoding parameters.The distilled model compresses a larger transformer by matching its softmax loss distribution; Table 2 describes the accented-speech WER statistics considered.Source: §4.3 · pp. 2–3, Table 2 · p. 3

5.1. Metrics

• Speaker-normalized median and P90 WER were used because mean WER is sensitive to outliers and does not describe distribution spread.Source: §5.1 · p. 3

5.2. Accented speech

• Accented read speech produced higher WER than Librispeech, with significant accent disparities and sometimes a two-fold gap on NISP; gender gaps were small on ALLSSTAR and wider on NISP, favoring women.ALLSSTAR models performed best on native English speakers and struggled with non-native speakers from Asian countries.Source: §5.2 · p. 3

5.3. Rehearsed Speech

• All models had significantly worse performance on rehearsed VoxPopuli speech than on ALLSSTAR, but its accent disparities were smaller than the Asian–native English gap on ALLSSTAR.Source: §5.3 · p. 3

5.4. Conversational data

• For every tested model, CORAAL WER was higher for speakers from the lower working class; only RASR big and distilled reached acceptable error rates.Table 4 reports median WER, P90, and median WERs for upper- and lower-working-class speakers.Source: Table 4 · p. 4, §5.4 · p. 4

5.5. Impact of the language model

• On RASR big, optimal decoding reduced WER by up to 25% on accented and rehearsed speech, but by only 9% on Buckeye and 5% on CORAAL.The authors conclude that standard language models may not be adapted for spontaneous speech.Source: §5.5 · p. 4, Table 5 · p. 4

• Wav2vec2.0 and SpeechBrain transferred well to accented and rehearsed speech despite training only on clean read speech without specific accent adaptation.Source: §6 · p. 4
• RASR models were most robust to domain shift overall and handled conversational speech to some extent; Google API robustness on accented and conversational data contrasted with its weak Librispeech performance.The authors conclude that Librispeech is no longer a good reference for evaluating ASR systems.Source: §6 · p. 4
• The tested models showed no significant performance disparity along gender lines, while accent and socioeconomic-status differences remained evident.Source: §6 · p. 4