Source-linked AI summary
Whisper-RIR-Mega: A Paired Clean-Reverberant Speech Benchmark for ASR Robustness to Room Acoustics
Mandip Goswami
TL;DR
Existing ASR benchmarks provide limited paired, controlled evidence of robustness to room acoustics. Whisper-RIR-Mega pairs clean LibriSpeech utterances with real RIR-Mega reverberant versions and evaluates five Whisper models. Reverberation increases WER and CER across all models, with larger models showing smaller WER penalties.
Problem
Existing reverberant-speech benchmarks often lack paired clean references, broad real-RIR coverage, or stratification by RT60 and DRR, limiting controlled robustness evaluation.
Method
Whisper-RIR-Mega pairs LibriSpeech utterances with versions convolved using real RIR-Mega impulse responses and stratifies splits by available RT60 or DRR metadata.
Results
Reverberation increases WER and CER for every Whisper model, with WER penalties ranging from 2.31 to 15.50 percentage points and larger models more robust.
Takeaways & Limitations
Robustness to room acoustics should be evaluated explicitly using paired clean-reverberant benchmarks such as Whisper-RIR-Mega.
Takeaways & Limitations
The benchmark is limited to English LibriSpeech speech and a single room impulse response per utterance.
Abstract
from arXiv · showhide
We introduce Whisper-RIR-Mega, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics. Each sample pairs a clean LibriSpeech utterance with the same utterance convolved with a real room impulse response from the RIR-Mega corpus, with stratified splits by reverberation time (RT60) and direct-to-reverberant ratio (DRR). We evaluate five Whisper models (tiny through large-v3) on 1600 test samples and report word error rate (WER) and character error rate (CER) under clean and reverberant conditions. Reverberation consistently degrades performance across all model sizes; the reverb penalty in WER ranges from 2.31 to 15.50 percentage points depending on the model. Whisper-large-v3 shows the smallest penalty; Whisper-tiny shows the largest. We release the dataset, evaluation code, and baseline results to support reproducible research on robust ASR.
1 Introduction
Whisper-RIR-Mega addresses gaps in reverberant-speech benchmarking by pairing clean utterances with real-RIR reverberant versions and stratifying test splits by acoustic conditions. Baselines across five Whisper models show consistently worse reverberant performance, with WER penalties ranging from 15.50 percentage points for tiny to 2.31 for large-v3.
- Motivation: Room reflections and reverberation can degrade ASR accuracy, motivating controlled evaluation of robustness to acoustic conditions.Such evaluation supports progress in acoustic modeling and dereverberation.
- Benchmark contribution: Whisper-RIR-Mega pairs each clean utterance with a reverberant version convolved with a real RIR from RIR-Mega.The benchmark addresses existing limitations involving unpaired references, synthetic or limited RIR sets, and missing acoustic stratification.
- Benchmark contribution: Stratified splits by RT60 or DRR balance the test set across acoustic conditions when metadata is available.The acoustic measures are reverberation time (RT60) and direct-to-reverberant ratio (DRR).
- Baseline results: 1600 test samples across five Whisper models show higher WER and CER under reverberant speech.The evaluated models are tiny, base, small, medium, and large-v3.
- Baseline results: 15.50 percentage points is the largest reverb penalty in WER for Whisper-tiny, while 2.31 percentage points is the smallest for Whisper-large-v3.All five models exhibit a reverberant-performance penalty.
- Resources: The dataset, evaluation code, and leaderboard are released on Hugging Face to support reproducible robust-ASR benchmarks.The release is intended to encourage reproducible benchmarking and further work on robust ASR.
2 Related Work
Prior work improves reverberation-robust ASR through training, front-end, and end-to-end methods, supported by benchmarks and resources spanning simulated and real reverberant speech. Key resources include LibriSpeech, RIR-Mega and RIR-Mega-Speech, while Whisper supplies a strong open ASR baseline.
- Reverberation-robust ASR has used multi-condition training, dereverberation front-ends, and end-to-end systems.
- REVERB and CHiME are benchmark datasets providing simulated or real reverberant speech.
- RIR-Mega provides machine-friendly RT60, DRR, and C50 metadata, while RIR-Mega-Speech pairs LibriSpeech with RIR-Mega and adds per-file acoustic annotations.
- Whisper provides strong open ASR baselines, and LibriSpeech is widely used as a clean-speech source.
3 Dataset Construction
Whisper-RIR-Mega pairs LibriSpeech test-clean utterances with measured RIR-Mega room responses, using stratified acoustic-condition sampling. The dataset stores processed clean-reverberant pairs with transcripts and metadata, and uses a deterministic validation/test split for evaluation.
- Speech and RIR pairing: LibriSpeech test-clean utterances are paired with one measured RIR-Mega impulse response, with sampling stratified across RT60 and DRR quantile bins when metadata exists.The speech source is 16 kHz LibriSpeech test-clean, and the RIRs come from RIR-Mega's train split.
- Signal processing and storage: 16 kHz convolution uses normalized RIR energy and peak-normalized output without background noise, storing clean and reverberant signals as FLAC with transcripts and available RIR metadata.Each sample receives a unique sample_id derived from the LibriSpeech ID and RIR assignment.
- Dataset splits: 2000 paired samples are split deterministically into 20% validation and 80% test, with RT60-stratified assignments; evaluation uses 1600 test samples.No training split is included in the default configuration, and validation and test reflect similar room-acoustic distributions.
4 Experimental Setup
The evaluation compares five Whisper model sizes under reproducible decoding and CPU settings. Performance is measured with mean WER and CER on 1600 test samples in clean and reverberant conditions, with penalties defined as reverb minus clean.
- Models and decoding: Five Whisper models, from tiny through large-v3, are evaluated with beam size 5, best-of 5, temperature 0, and English decoding.All runs use CPU inference with FP16 disabled for reproducibility.
- Metrics: Mean WER and CER are computed with jiwer after lowercase, punctuation removal, and whitespace-collapse normalization.Each metric is averaged over the test set separately for clean and convolved reverberant utterances.
- Evaluation conditions: The test set contains 1600 samples, evaluated under clean and reverberant conditions with ΔWER and ΔCER defined as reverb minus clean.The clean condition uses original utterances, while the reverberant condition uses convolved utterances.
5 Results
Across 1600-sample clean and reverberant test sets, reverberation degrades Whisper ASR performance for every model. The WER penalty is largest for Whisper-tiny and smallest for Whisper-large-v3, while CER shows a similar degradation pattern.
- 1600 samples per condition are evaluated using mean WER and CER with lowercase and punctuation removed during standard normalization.
- 29.0%–54.9% clean WER spans Whisper-large-v3 to Whisper-tiny, respectively.
- 15.50 percentage points is Whisper-tiny’s largest WER reverb penalty, versus 2.31 percentage points for Whisper-large-v3.
- 3.80 pp is Whisper-tiny’s largest CER reverb penalty, while Whisper-medium has the smallest at 0.48 pp.
- WER-versus-RT60 and WER-versus-DRR plots are provided for reverberant conditions when RIR metadata is available.
6 Discussion
Reverberation degrades ASR across Whisper model sizes, with larger models showing lower sensitivity. The benchmark is limited to English and one room impulse response per utterance, motivating broader extensions and community evaluation.
- 6 Discussion: Reverberation hurts ASR performance across all Whisper sizes, but degradation varies by model.Whisper-tiny is more sensitive in WER delta, while Whisper-large-v3 shows the smallest penalty.
- 6 Discussion: Larger Whisper models exhibit monotonically smaller WER degradation from reverberation.Whisper-tiny is the most sensitive, and Whisper-large-v3 has the smallest reverb penalty.
- 6 Discussion: The benchmark is limited to English LibriSpeech and a single room impulse response per utterance.These constraints define the current evaluation scope.
- 6 Discussion: Future extensions could add multiple room impulse responses, other languages, and additive noise.The authors also encourage community use of the dataset and reporting results on the public leaderboard.
7 Conclusion
Whisper-RIR-Mega is a paired clean-reverberant speech benchmark for evaluating ASR robustness to room acoustics. Reverberation consistently worsens WER and CER, while larger Whisper models are more robust; the dataset, code, and leaderboard are publicly available.
- Contributions: Whisper-RIR-Mega pairs LibriSpeech speech with RIR-Mega reverberation, using stratified splits and baselines for five Whisper models.The benchmark supports evaluation of ASR robustness to room acoustics.
- Findings: Reverberation consistently increases WER and CER across the evaluated Whisper models.
- Findings: 2.31 to 15.50 percentage points is the reported WER reverb penalty, with larger models showing greater robustness.
- Availability: The dataset, evaluation code, and leaderboard are publicly available to support reproducible research on robust ASR.