Source-linked AI summary
RIR-Mega-Speech: A Reverberant Speech Corpus with Comprehensive Acoustic Metadata and Reproducible Evaluation
Mandip Goswami
TL;DR
Comparing reverberant-speech methods remains difficult because corpora often lack per-file acoustic annotations or reproducible evaluation documentation. RIR-Mega-Speech provides an annotated, rebuildable corpus and scripts, finding higher ASR error under reverberation with WER varying systematically by room acoustics. Its scope is limited by LibriSpeech’s read English audiobook speech.
Problem
Existing reverberant-speech corpora often lack per-file acoustic labels or sufficient documentation to reproduce evaluations and compare results.
Method
The paper constructs a standardized corpus with RT60, DRR, and C50 metadata and releases code to regenerate audio, compute metrics, and reproduce evaluations.
Results
WER increases monotonically with RT60 and decreases with DRR in Whisper small evaluations of reverberant speech.
Takeaways & Limitations
The corpus makes reverberant acoustic conditions transparent and reported results independently verifiable.
Takeaways & Limitations
Using LibriSpeech limits coverage to read English audiobook speech, excluding spontaneous speech, non-native accents, and other languages.
Abstract
from arXiv · showhide
Despite decades of research on reverberant speech, comparing methods remains difficult because most corpora lack per-file acoustic annotations or provide limited documentation for reproduction. We present RIR-Mega-Speech, a corpus of approximately 117.5 hours created by convolving LibriSpeech utterances with roughly 5,000 simulated room impulse responses from the RIR-Mega collection. Every file includes RT60, direct-to-reverberant ratio (DRR), and clarity index ($C_{50}$) computed from the source RIR using clearly defined, reproducible procedures. We also provide scripts to rebuild the dataset and reproduce all evaluation results. Using Whisper small on 1,500 paired utterances, we measure 5.20% WER (95% CI: 4.69--5.78) on clean speech and 7.70% (7.04--8.35) on reverberant versions, corresponding to a paired increase of 2.50 percentage points (2.06--2.98). This represents a 48% relative degradation. WER increases monotonically with RT60 and decreases with DRR, consistent with prior perceptual studies. While the core finding that reverberation harms recognition is well established, we aim to provide the community with a standardized resource where acoustic conditions are transparent and results can be verified independently. The repository includes one-command rebuild instructions for both Windows and Linux environments.
1. Introduction
Reverberation degrades speech recognition, while existing corpora make acoustic-condition analysis and reproducible comparison difficult. RIR-Mega-Speech addresses this gap with comprehensive metadata, regeneration code, and checkable evaluation procedures.
- Research gap: Existing reverberant corpora often lack per-file acoustic labels, redistributable RIRs, or sufficient reproduction documentation.These omissions hinder comparison across papers and assessment of whether improvements generalize.
- Contribution: RIR-Mega-Speech attaches RT60, DRR, and C50 values to every reverberant file using the source RIR.The corpus is designed to make acoustic conditions transparent at file level.
- Contribution: The project releases code to regenerate audio, compute metrics, and reproduce the reported evaluation results.It also uses utterance-level nonparametric bootstrap confidence intervals and paired tests where applicable.
- Scope: The paper emphasizes standardized, independently checkable evaluation rather than algorithmic novelty or new acoustic findings.The expected relationship between higher RT60 and WER is presented as established rather than novel.
2. Related Work
Prior reverberant-speech resources provide valuable recordings, challenge data, or RIRs, but commonly omit per-file acoustic metadata or require users to assemble processing pipelines. The paper positions its corpus within this landscape using standard room-acoustic measures and established reverberation–intelligibility relationships.
- Existing corpora: The REVERB Challenge provides paired clean and reverberant speech with evaluation scripts, but released annotations lack per-file acoustic metadata.This limits post-hoc analysis by acoustic condition.
- Existing corpora: CHiME, AISHELL-4, and AMI include naturally varying reverberant speech but generally do not provide RT60 or DRR per utterance.Their primary emphasis is challenge or meeting-speech tasks rather than general-purpose annotated corpora.
- RIR resources: Released RIR and noise collections support augmentation, but users must perform convolution and manage metadata instead of receiving ready-to-use transcribed pairs.This applies to the Acoustic Echo Cancellation and Deep Noise Suppression challenges.
- Existing corpora: VCTK-based reverberant datasets commonly provide clean–reverberant pairs but lack comprehensive per-file annotations or systematic coverage analysis.They are often used for speech-enhancement evaluation.
- Acoustic measures: RT60 measures 60-dB energy-decay time, DRR measures direct-versus-reflected energy balance, and C50 is a speech-related clarity metric.These measures connect corpus annotations to established room-acoustic analysis.
- Reproducibility: Reproducibility discussions increasingly encourage releasing code, while complete data and scripts for recreating figures and tables remain less common.The paper follows standardized benchmark practices from computer vision and NLP.
3. Dataset Construction
RIR-Mega-Speech combines LibriSpeech with simulated RIR-Mega responses to create a reproducible reverberant corpus with per-file acoustic metadata. Its construction uses speaker-stratified splits, standardized RIR measurements, and documented sampling choices, while leaving acoustic coverage and speech diversity limited.
- Source Materials: Approximately 5,200 LibriSpeech utterances provide clean speech spanning 1.5 to 36 seconds.The source material consists specifically of the dev-clean and test-clean subsets, selected for public availability and reliable transcriptions.
- Source Materials: Roughly 5,000 simulated RIR-Mega responses cover varied room configurations, dimensions, absorption coefficients, source-receiver distances, and microphone placements.The subset targets broad acoustic coverage across indoor environments while maintaining computational feasibility.
- Convolution and File Generation: Each clean utterance is convolved in the time domain with up to ten randomly sampled RIRs, producing 53,230 reverberant files totaling 117.5 hours.Variants are excluded when convolution causes clipping or metadata is problematic; files are stored as 16-bit PCM WAV at 16 kHz with accompanying metadata.
- Acoustic Parameter Computation: RT60, DRR, and C50 are computed for every source RIR before convolution using explicitly specified acoustic procedures.RT60 uses Schroeder backward integration and a -5 dB to -35 dB fitting range; DRR uses a 2.5 ms direct-only window, while C50 is also computed as a standard room-acoustics measure.
- Acoustic Parameter Computation: The narrow 2.5 ms DRR window isolates the true direct path but can produce very low DRR values and differs from definitions that include early reflections.The authors identify this mismatch with perceptual clarity definitions as a limitation and plan alternative DRR definitions in future releases.
- Dataset Splits and Limitations: Speaker-stratified train, development, and test splits contain 43,660, 4,620, and 4,950 files, but acoustic sampling is not stratified and coverage remains uneven.The clean speech is limited to read English from LibriSpeech, excluding spontaneous speech, non-native accents, and other languages.
4. Corpus Statistics and Coverage
The corpus uses speaker-stratified splits and provides broad but uneven acoustic coverage, concentrated in typical room conditions. Duration is not strongly correlated with RT60, reducing confounding in WER analyses.
- Acoustic statistics: Mean RT60 is 0.44 seconds, spanning 0.09–1.51 seconds; mean DRR is 3.32 dB with extreme negative outliers.The narrow direct-window definition contributes to the broad DRR range and values below -100 dB.
- Splits: 43,660 training, 4,620 development, and 4,950 test files are split by speaker, preventing speaker overlap across partitions.Acoustic distributions are similar across splits, but they are not explicitly balanced by acoustic condition.
- Coverage: RT60 ranges from about 0.3–0.4 seconds in the main mode to a 1.5-second tail, covering larger halls but not highly reverberant spaces such as cathedrals.Office and classroom-like conditions dominate the distribution.
- Acoustic statistics: Most files have DRR between -5 and +15 dB, while very negative values arise when weak direct peaks fall below early reflections.These extreme files represent challenging conditions but may be less common in typical real rooms.
- Coverage: Coverage is densest at RT60 values of 0.2–0.6 seconds and DRR values of 0–10 dB, with few samples at high RT60 and very low DRR.The uneven distribution reflects the RIR pool and nonstratified sampling.
- Coverage: Long and short utterances span RT60 bins without strong correlation, reducing the risk that duration confounds acoustic trends in WER.The corpus also shows visible energy smearing and late-reflection buildup in reverberant spectrograms.
5. Evaluation Setup
Evaluation uses Whisper small on paired clean and reverberant test utterances, with standardized decoding and bootstrap-based uncertainty estimation. The setup also analyzes WER trends across fixed RT60 and DRR bins.
- Model Selection: Whisper small is evaluated with default beam search, beam size 5, English transcription, and no hyperparameter tuning.References and hypotheses receive the same lowercase-and-punctuation normalization before WER computation.
- Evaluation Sets: 1,500 test utterances form paired clean–reverberant comparisons, with one randomly selected reverberant variant decoded per utterance.Pairing controls for between-utterance variance and avoids overlap with potential training scenarios.
- Evaluation Sets: Table 2 reports Whisper small WER for paired clean and reverberant utterances, with bootstrap 95% confidence intervals in parentheses.The table summarizes results from N=1,500 paired utterances.
- Statistical Analysis: Bootstrap 95% confidence intervals use 2,000 utterance-level resamples and percentile bounds, avoiding a normality assumption.The skewed distribution of per-utterance WER motivates the nonparametric procedure.
- Statistical Analysis: Paired WER differences are bootstrapped by utterance, providing greater power than independent-sample comparisons by controlling for utterance difficulty.The paired procedure is applied to the clean-versus-reverberant comparison.
- Acoustic Trends: WER trends are described within fixed RT60 bins from 0.2 to 1.2 seconds and DRR bins from -10 to 15 dB, each with bootstrap confidence intervals.No multiple-comparison adjustment is applied because the analysis describes trends rather than tests hypotheses.
6. Results
Reverberation raises Whisper small's recognition error rate, with errors worsening as RT60 increases and improving as DRR increases. Acoustic conditions jointly shape difficulty, while duration and loudness have smaller effects.
- Clean vs Reverberant: Paired Baseline: 2.50 percentage points: Whisper small's paired WER increase from 5.20% on clean speech to 7.70% on reverberant speech.The relative increase is 48%, with paired 95% confidence interval 2.06–2.98 percentage points.
- Clean vs Reverberant: Paired Baseline: Most per-utterance WER points lie above the diagonal, indicating more errors on reverberant speech.A few points fall below the diagonal, and the scatter reflects varying acoustic conditions and utterance difficulty.
- Trends with RT60 and DRR: WER increases from about 6% at RT60 = 0.2–0.4 seconds to about 10% at RT60 = 1.0–1.2 seconds.The trend is monotonic, and confidence intervals do not overlap for the extreme bins.
- Trends with RT60 and DRR: WER decreases as DRR increases, with the strongest effect below 0 dB and a plateau near clean-speech levels above 10 dB.The heatmap identifies high RT60 with low DRR as the worst combined condition.
- Trends with RT60 and DRR: Duration has a weaker effect than RT60 or DRR after acoustic conditions are accounted for.The duration trend is weak, and duration is not strongly correlated with RT60 in the corpus.
- Ablations: Loudness Normalization and Additive Noise: 8.00% WER after loudness normalization is close to the 7.70% reverberant baseline, whereas additive noise produces 30.95% WER.The normalization confidence intervals overlap the baseline; the noise result is treated as a sanity check rather than a reverberation-specific finding.
7. Discussion and Limitations
RIR-Mega-Speech is positioned as a reproducible evaluation resource rather than an algorithmically novel benchmark. Its main boundaries are simulated acoustics, uneven coverage, and a non-standard narrow DRR definition.
- What This Corpus Provides: The corpus combines scale, per-file acoustic annotations, and reproducibility for evaluating dereverberation and robust ASR methods.It is not intended to introduce a novel task or metric.
- What This Corpus Provides: One-command rebuild scripts make audio generation, metric computation, and figure production explicit across Windows and Linux.The authors caution that hardware or software differences may prevent identical results.
- What This Corpus Does Not Provide: The corpus uses simulated rather than measured RIRs, so it may omit complexities of real rooms such as furniture effects, scattering, and HVAC noise.Evaluation on real-world recordings such as REVERB or CHiME is recommended for validation requiring measured room responses.
- What This Corpus Does Not Provide: Acoustic coverage is uneven, with some RT60–DRR regions containing hundreds of files and others fewer than ten.The authors plan to expand coverage to more extreme conditions.
- What This Corpus Does Not Provide: The 2.5 ms direct-only DRR window is non-standard and may not align with perceptual relevance because it excludes early reflections.Alternative DRR definitions are planned for future versions.
- What This Corpus Provides: Compared with other corpora, the corpus emphasizes a universal per-file metadata CSV and one-command rebuild scripts.REVERB and CHiME provide paired data but not per-file RT60/DRR annotations, while some challenge resources lack convolved transcribed speech.
8. Reproducibility Details
The reproducibility package documents the software, hardware, repository layout, and end-to-end commands used to rebuild and evaluate the corpus.
- Environment: Python 3.10, PyTorch, torchaudio, transformers, and librosa comprise the main software dependencies.Scripts were developed on Windows 10 with PowerShell 7 and tested on Ubuntu 20.04 with bash.
- Environment: GPU is required for Whisper inference but not for RIR metric computation or convolution.This separates model evaluation requirements from corpus-generation requirements.
- Runtime: 2 to 3 hours is the reported runtime for convolving 53,230 pairs on a 16-core CPU.The build is I/O bound because it reads and writes WAV files.
- Runtime: 1 to 2 hours is the reported runtime for Whisper small evaluation on 1,500 files using one GPU with 24 GB VRAM.Bootstrap confidence-interval resampling adds under 5 minutes using cached WER values.
- Repository and Commands: The repository separates clean speech, RIRs, generated audio and metadata, figures, and build/evaluation scripts into distinct directories.The universal metadata CSV is stored with the generated outputs.
- Repository and Commands: make all.ps1 on Windows or make all.sh on Linux executes the full pipeline.The commands provide a single entry point for the documented rebuild process.
9. Planned Releases
The planned release structure separates the full corpus, a lightweight subset, evaluation materials, and metric computation tools. Licensing follows the source materials.
- Planned Artifacts: Four planned artifacts target distinct uses: the full corpus, RIR-Mega-Lite, RIR-Mega-Eval, and RIR-Mega-Metrics.RIR-Mega-Lite is planned as a 10–20 hour subset for fast benchmarking.
- Planned Artifacts: The full RIR-Mega-Speech release will include the corpus and universal metadata CSV.The separate artifacts are intended to support different use cases.
- Planned Artifacts: RIR-Mega-Eval will provide evaluation scripts and baseline result CSVs, while RIR-Mega-Metrics will compute RT60, DRR, and C50.These components separate evaluation workflows from standalone acoustic-metric computation.
- Licensing: LibriSpeech is licensed CC BY 4.0, and simulated RIRs from RIR-Mega are released under MIT.Derived convolved audio can be redistributed under the same terms as the clean source.
10. Conclusion
RIR-Mega-Speech provides a standardized, rebuildable corpus with documented acoustic conditions and reproducible evaluation. Its Whisper small results quantify expected reverberation effects, while uneven acoustic coverage and Whisper-only comparisons constrain the current release.
- Results: 48% relative WER increase under reverberation was observed with Whisper small.Error rates increased monotonically with RT60 and decreased with DRR, with confidence intervals reported.
- Contribution: RIR-Mega-Speech documents acoustic conditions and provides reproducible scripts so results can be independently verified.The corpus uses simulated RIR-Mega impulse responses with systematic acoustic coverage and verified ground-truth parameters.
- Corpus scope: The corpus includes train, development, and test splits stratified by speaker.The release documents version v1.0 of the corpus.
- Limitations: Uneven acoustic coverage and Whisper-only comparisons limit the current evaluation scope.Future versions may expand acoustic coverage and baseline diversity, and may revise acoustic metric definitions.