PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech Robust Bench: A Robustness Benchmark For Speech Recognition

Muhammad A. Shah, David Solans Noguero, Mikko A. Heikkila, Bhiksha Raj, Nicolas Kourtellis

arXiv:2403.07937v3eess.AScs.CLcs.LGcs.SD

TL;DR

ASR robustness benchmarks often cover limited scenarios or make it difficult to isolate weaknesses, despite the need to assess reliability under physical and digital corruptions. SRB combines real recordings and 114 simulated perturbations to evaluate models, finding robustness differences across models and speaker subgroups.

  • Problem

    Existing ASR benchmarks often cover only a few corruption types, use uncontrolled natural data, or make comparisons across models difficult.

    Source: §1 · p. 2, §2.3 · p. 3

  • Method

    SRB evaluates ASR models using real scenario recordings and a bank of 114 digitally simulated perturbations.

    Source: §1 · p. 2

  • Results

    Whisper was most robust on average among tested models, larger models tended to be more robust, and robustness disparities appeared across speaker subgroups.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SRB supports standardized robustness comparisons and evaluations of subgroup disparities, with transformed English and Spanish test sets released for researchers and developers.

    Source: §6 · p. 10

    SRB may not cover all real-world scenarios and has so far been tested only on English and Spanish, which may limit generalizability to other languages and dialects.

    Source: §5 · p. 10

Abstract

from arXiv · show

As Automatic Speech Recognition (ASR) models become ever more pervasive, it is important to ensure that they make reliable predictions under corruptions present in the physical and digital world. We propose Speech Robust Bench (SRB), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 114 input perturbations which simulate an heterogeneous range of corruptions that ASR models may encounter when deployed in the wild. We use SRB to evaluate the robustness of several state-of-the-art ASR models and observe that model size and certain modeling choices such as the use of discrete representations, or self-training appear to be conducive to robustness. We extend this analysis to measure the robustness of ASR models on data from various demographic subgroups, namely English and Spanish speakers, and males and females. Our results revealed noticeable disparities in the model's robustness across subgroups. We believe that SRB will significantly facilitate future research towards robust ASR models, by making it easier to conduct comprehensive and comparable robustness evaluations.

1 INTRODUCTION

Existing ASR robustness benchmarks are fragmented and often miss digital corruptions or fail to isolate specific weaknesses. SRB combines real and simulated scenarios for systematic evaluation, and its tests reveal model-level and subgroup robustness differences.

1 INTRODUCTION

  • Existing ASR benchmarks focus on one or a few scenarios, leaving developers to assemble disparate evaluations that hinder reliable model comparisons.Natural-speech benchmarks may reflect average real-world performance without identifying which uncontrolled corruptions challenge a model.Source: §1 · pp. 1–2
  • SRB combines scenario-specific real recordings with 114 digital perturbations spanning environmental, equipment, special-effect, generated-speech, and adversarial distortions.The recordings include accented speech and interpersonal conversations, selected to limit confounding corruptions.Source: §1 · p. 2
  • Whisper was the most robust on average among tested models, larger models tended to be more robust, and robustness disparities appeared across speaker subgroups.The subgroup analyses compare English and Spanish speakers and male and female speakers.Source: §1 · p. 2
  • SRB is presented as a comprehensive benchmark intended to enable direct comparisons and fine-grained robustness and subgroup analyses.The authors also report releasing transformed test sets and code to support reproducibility and extensibility.Source: §1 · p. 2

2 RELATED WORK

Prior ASR robustness methods modify models or input features, while adversarial robustness evaluates resistance to specially designed perturbations. Existing ASR benchmarks remain limited in coverage, comparability, and ability to pinpoint weaknesses.

2.1 ROBUST AUTOMATIC SPEECH RECOGNITION

  • Model-based methods adapt models, denoise audio, or train on noisy data, while feature-based methods extract speech-relevant signal components; recent systems instead learn features end to end.Model-based methods are most effective when noise sources or deployment environments are known; learned-feature proponents posit that larger training datasets improve robustness, though the paper reports exceptions.Source: §2.1 · pp. 2–3

2.2 ADVERSARIAL ROBUSTNESS

  • Adversarial attacks use perturbations designed to change model responses, with white-box methods assuming model access and black-box methods relying on queries.SRB includes white-box attacks optimized for individual inputs and attacks intended to mis-transcribe multiple inputs.Source: §2.2 · p. 3

2.3 ROBUSTNESS BENCHMARKS FOR SPEECH

  • Many ASR benchmarks cover only a few scenarios, and digital effects, generated speech, and adversarial examples are often overlooked.SRB combines previously represented scenarios with newer ones in a single benchmark.Source: §2.3 · p. 3
  • Researchers often evaluate models on disparate benchmark subsets or natural-speech datasets, making robustness comparisons across studies difficult.SRB covers many challenging scenarios in one benchmark to unify evaluations.Source: §2.3 · p. 4
  • Natural-speech benchmarks may not reveal specific weaknesses because their corruption sources are uncontrolled or unknown.SRB limits and identifies noise sources within utterances to help pinpoint scenarios or perturbations where models struggle.Source: §2.3 · p. 4

3 SPEECH ROBUST BENCH

SRB evaluates ASR robustness through scenario simulation, transcription, and metric computation. It spans real and simulated speech conditions, then reports error and difficulty-normalized measures to compare performance across scenarios.

3 SPEECH ROBUST BENCH

  • SRB simulates challenging conditions, transcribes clean and noisy recordings with the target model, and compares predicted with reference transcripts using accuracy and robustness metrics.Speech-quality scores are used to normalize metrics for differences in scenario difficulty.Source: §3 · p. 4

3.1 SCENARIO SIMULATION

  • The benchmark organizes scenarios into six families: clean speech, social gatherings, speech variations, environmental effects, digital augmentations, and adversarial attacks.Figure 2 distinguishes real-world recordings in dashed boxes from digitally simulated perturbations in solid boxes.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Clean speech datasets provide baselines and perturbation sources, while CHiME-6 and AMI contribute near- and far-field meeting and dinner-party recordings.Clean English and Spanish speech is drawn from LibriSpeech, TEDLIUM, and MLS; accented and synthetic speech add pronunciation and prosodic variation.Source: §3.1 · pp. 4–5
  • Environmental scenarios add noise at 10, 20, 30, or 40 dB SNR and simulate spatial acoustics; digital augmentations include standard processing, special effects, and white noise.Environmental noise is drawn from ESC-50, MS-SNSD, MUSAN, and WHAM!, while spatial acoustics use echo and room impulse responses.Source: §3.1 · p. 5
  • SRB includes utterance-specific attacks optimized for an individual recording and utterance-agnostic attacks optimized over a held-out set.The latter represents an attacker introducing a shared perturbation into the transcription pipeline; both attack types use SNR constraints from 10 to 40 dB.Source: §3.1 · p. 5

3.2 METRICS

  • WER measures normalized word-level edit distance, while WER Degradation (WERD) subtracts clean-set WER from scenario-set WER.For inherently noisy scenarios, the clean comparison set is LibriSpeech for English and MLS for Spanish; digitally perturbed scenarios compare each perturbation with its clean source.Source: §3.2 · p. 6
  • Normalized WER and WERD divide metrics by estimated speech-quality degradation, weighting errors on easier scenarios more than errors on harder ones.SRB uses DNSMOS and PESQ estimates; adversarial scenarios use WERD rather than NWERD because model-specific quality scores would make comparisons difficult.Source: §3.2 · p. 6

4 EVALUATION

The evaluation applies SRB to recent ASR models, analyzes robustness across non-adversarial and adversarial perturbations, and examines model correlates and subgroup disparities. Results vary by perturbation type, model characteristics, language, and speaker gender.

4.1 MODELS

  • The evaluation covers recent English and Spanish ASR models, including Whisper, Wav2Vec, HuBERT, Canary, Parakeet, and MMS; Table 1 reports a selected subset, excluding small or less accurate models.Utility is measured by clean-speech WER; robustness uses NWERD for non-adversarial perturbations and WERD for adversarial perturbations, averaged across datasets, perturbations, and severities.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 6

4.2.1 ROBUSTNESS IN NON-ADVERSARIAL SCENARIOS

  • Whisper large is most robust on average to non-adversarial perturbations, while Canary ranks fifth despite leading OAL and is especially weak on special effects and spatial acoustics.Wav2Vec and HuBERT models also show weak robustness to accented speech, social scenarios, and spatial acoustics despite pretraining on 60K hours of speech.Source: §4.2.1 · p. 7
  • For Spanish non-adversarial perturbations, Whisper large is most robust; all other evaluated models struggle with accented speech, especially Canary, while MMS is comparatively consistent across categories.Canary has an NWERD of 700% (WERD=205%) on accented speech.Source: §4.2.1 · p. 8

4.2.2 ROBUSTNESS IN ADVERSARIAL SCENARIOS

  • Adversarial robustness differs from non-adversarial rankings: Wav2Vec leads on utterance-specific attacks, MMS and smaller Whisper variants on utterance-agnostic attacks, and Spanish Wav2Vec models generally outperform Whisper and Canary.For Spanish, Wav2Vec base Spanish is the most robust against adversarial attacks; Spanish attack WERD is higher than English.Source: §4.2.2 · p. 8

4.3 CORRELATES OF ROBUSTNESS

  • RNN-T models outperform seq2seq and CTC models on non-adversarial robustness, whereas CTC models have the lowest adversarial WERD; clean-data accuracy is positively related to robustness, more weakly for adversarial attacks.The relationship between adversarial WERD and clean WER is weaker than the relationship for non-adversarial robustness.Source: §4.3 · pp. 8–9, Figure 5 · p. 9
  • Larger models correlate with better robustness, including within Whisper, Parakeet, and Wav2Vec families but not HuBERT; training-data size appears to have only a minor influence.The comparison is based on robust linear regression across models and NWERD trends within model families.Source: §4.3 · pp. 8–9, Figure 4 · p. 8, Figure 5 · p. 9

4.4.1 DISPARITY IN ROBUSTNESS ACROSS LANGUAGES IN MULTI-LINGUAL MODELS

  • Multilingual-model robustness generally favors English over Spanish; Whisper has lower WERD on English across almost all categories, while Canary and MMS are similar across some categories.The English–Spanish WERD gap on environmental noise and spatial acoustics is greater for Whisper large than for Canary.Source: §4.4 · p. 9, Figure 6 · p. 9, §4.4.1 · p. 9

4.4.2 DISPARITY IN ROBUSTNESS ACROSS GENDERS

  • Gender disparities vary by dataset: models are more robust for females on TEDLIUM, Common Voice, and AMI, but more robust for males on LibriSpeech and Spanish MLS.The largest gender disparity is reported for Spanish MLS, where Canary appears most biased against females; adversarial perturbations increase Whisper large’s WER more for females on LibriSpeech.Source: §4.4.2 · pp. 9–10, Figure 7 · p. 10

5 LIMITATIONS

SRB covers diverse datasets, distortions, and adversarial attacks, but may not represent every real-world scenario. Its evaluation is limited to English and Spanish, which may miss robustness challenges in other languages and dialects.

5 LIMITATIONS

  • SRB may not encompass all possible real-world scenarios despite its diverse datasets, distortions, and adversarial attacks.The authors state that this may affect how generalizable results obtained with SRB are.Source: §5 · p. 10
  • The benchmark has so far been tested only on English and Spanish, which may not capture challenges in other languages and dialects.The authors say SRB can be extended to incorporate novel approaches for quantifying ASR robustness.Source: §5 · p. 10
  • The authors identify possible effects of these scope boundaries on the generalizability of SRB results.They also note the benchmark's extensibility for adding approaches to quantify ASR robustness.Source: §5 · p. 10

6 CONCLUSION

The authors propose SRB to standardize ASR robustness evaluations and demonstrate its utility through examples, including fairness assessment. They release transformed English and Spanish test sets and anticipate that SRB will encourage more comparable robustness evaluations.

6 CONCLUSION

  • The authors propose SRB as a standardized benchmark for evaluating ASR robustness and demonstrating its utility through concrete examples, including fairness assessment.They intend it to support easy comparisons between existing and new approaches.Source: §6 · p. 10
  • SRB's transformed test sets are released in English and Spanish to support researchers and model developers conducting robustness evaluations.The authors anticipate that this will make such evaluations more prevalent.Source: §6 · p. 10
  • The authors anticipate that more prevalent evaluations will encourage model developers to consider robustness as a key metric for improvement.This is presented as an expectation, not a demonstrated effect.Source: §6 · p. 10

A PERTURBATION GENERATION/APPLICATION PROCEDURE

The appendix describes how SRB's perturbations are generated or applied, including noise, acoustic effects, speech transformations, and adversarial attacks. It also points to tables listing perturbation parameters and speech-quality scores, and dataset statistics.

A PERTURBATION GENERATION/APPLICATION PROCEDURE

  • Table 2 lists perturbation parameters, while Table 3 reports normalized DNSMOS and PESQ scores for each perturbation.Table 4 gives distributional statistics for the speech and noise datasets used in SRB.Source: A PERTURBATION GENERATION/APPLICATION… · p. 16, Table 3 · p. 17, Table 4 · p. 17
  • Environmental noise is sampled per test utterance from ESC-50, MS-SNSD, MUSAN, or WHAM and added at a specified SNR; longer noise is clipped and shorter noise repeated.Crosstalk and music use MUSAN audio, added with torchaudio's add_noise function at a given SNR.Source: A PERTURBATION GENERATION/APPLICATION… · pp. 17–18
  • Room impulse responses are applied to clean recordings, with severity based on increasing RT60 for synthetic responses and decreasing SRMR for real responses.At evaluation, an RIR at the given severity is sampled for each test recording; resampling, speed, pitch, and gain changes use torchaudio transforms, while other effects use SoX filters.Source: A PERTURBATION GENERATION/APPLICATION… · p. 18
  • Accent recordings come from clean Common Voice 17 samples with non-American accents and DNSMOS of at least 3.4; the resulting subset contains 640 recordings.Voice conversion uses YourTTS to synthesize LibriSpeech test-clean transcripts in the style of randomly selected VCTK speakers from 12 accents.Source: A PERTURBATION GENERATION/APPLICATION… · p. 18
  • Inter-personal communication scenarios use CHiME-6 and AMI recordings, with separate near- and far-field results and recordings under three words removed.The recordings represent dinner parties and meetings and come from lapel and room microphones.Source: A PERTURBATION GENERATION/APPLICATION… · p. 18
  • Utterance-specific attacks use untargeted PGD to optimize a per-recording noise perturbation that maximizes divergence between true and predicted transcriptions.Utterance-agnostic attacks instead compute one perturbation expected to attack any input; evaluation perturbations are applied to test audio at the specified SNR, using development and test splits described for LibriSpeech, TEDLIUM, and Multilingual LibriSpeech.Source: A PERTURBATION GENERATION/APPLICATION… · p. 19

B ADDITIONAL DEFINITIONS

This appendix defines WER and its aggregation, notes the analogous character-level CER, and describes the utterance-agnostic attack algorithm. It also introduces LWERR for comparing prediction accuracy across male and female speakers and motivates robustness disparities as a fairness dimension.

B ADDITIONAL DEFINITIONS

  • WER is word-level edit distance between predicted and reference transcripts, normalized by reference length, with punctuation removed and text lowercased.The edit distance counts substitutions, deletions, and additions; across transcript pairs, counts are summed and divided by the summed reference lengths.Source: B ADDITIONAL DEFINITIONS · p. 19
  • Across transcript pairs, WER sums substitutions, deletions, and additions and divides by the total reference length; CER uses character-level edit distance instead.The appendix identifies edit distance as the basis for the computation.Source: B ADDITIONAL DEFINITIONS · p. 19
  • The utterance-agnostic attack algorithm iteratively updates a perturbation using speech data, an ASR model and loss, an allowed SNR, and target success-rate and CER thresholds.Its stated inputs also include a learning rate and limits on epochs and per-sample iterations.Source: B ADDITIONAL DEFINITIONS · p. 20
  • LWERR measures male–female disparity in prediction accuracy using the female and male utterance subsets.The appendix names the measure Log WER Ratio and defines the subsets by speaker gender.Source: B ADDITIONAL DEFINITIONS · p. 21
  • The paper assesses fairness through robustness disparities across English and Spanish speech and male and female speakers, alongside the more common focus on predictive-performance disparities.It notes prior work that considered robustness disparities as an alternative fairness notion.Source: B ADDITIONAL DEFINITIONS · p. 21
  • The authors argue that robustness may better reflect expected subgroup disparities under diverse noisy conditions in deployment.They connect this motivation to the possibility of differing robustness across population subgroups.Source: B ADDITIONAL DEFINITIONS · p. 21

C MODELS

The evaluation includes unilingual and multilingual models, with multilingual-model WER reported separately for English and Spanish.

C MODELS

  • Multilingual models are pretrained and fine-tuned on multiple languages, allowing the same DNN to transcribe several languages.Some unilingual models are pretrained on multilingual data but fine-tuned on one language and cannot transcribe other languages.Source: C MODELS · p. 21
  • The model names correspond to their pretrained Hugging Face checkpoint names, with abbreviations shown in parentheses.Source: C MODELS · p. 21
  • For multilingual models, WER is presented as English/Spanish.Source: C MODELS · p. 21

D FINE GRAINED ANALYSES

The appendix figures provide fine-grained views of model accuracy and robustness, including breakdowns by perturbation, language, and severity.

D FINE GRAINED ANALYSES

  • Figure 8 overviews model accuracy and robustness for English and Spanish, using clean-data WER, perturbed-data NWERD, and adversarial-data WERD.Its markers identify datasets or perturbation categories, and the x-axes are ordered by ascending y-axis values.Source: D FINE GRAINED ANALYSES · p. 21
  • Figures 9 and 10 present robustness breakdowns by perturbation for models on English and Spanish, respectively, while Figure 11 presents a separate breakdown of robustness by severity.Source: D FINE GRAINED ANALYSES · p. 21
  • Figure 11 presents English-data NWERD as augmentation severity increases.Source: D FINE GRAINED ANALYSES · p. 21

E COMPUTE RESOURCES

The experiments were run on the Bridges-2 cluster at the Pittsburgh Supercomputing Center using Nvidia V-100 GPUs.

E COMPUTE RESOURCES

F DATASET LICENSES

The appendix identifies licenses for the SRB datasets and includes tables and figures covering evaluated models, accuracy, and robustness analyses.

F DATASET LICENSES

  • The licenses of the datasets considered in SRB are described in Table 6.The listed licenses include CC-BY-4.0 for LibriSpeech, CC BY 4.0 for Multilingual Librispeech, and CC-BY-NC-ND 3.0 for TEDLIUM.Source: F DATASET LICENSES · pp. 21–22, Table 6 · p. 22
  • Table 5 summarizes the models used in the evaluations.Source: Table 5 · p. 22
  • Table 7 covers accuracy and robustness of ASR models on all datasets.Source: Table 7 · p. 24
  • Figure 8 encodes clean-speech WER, non-adversarial NWERD, and adversarial WERD for English and Spanish models.Markers indicate datasets or perturbation categories; x-axes are ordered by ascending y-axis values.Source: Figure 8 · p. 25
  • Figures 9 and 10 show English and Spanish models' perturbation-level NWERD averaged over severities.Source: Figure 9 · p. 26, Figure 10 · p. 27
  • Figures 11 and 12 show English NWERD across increasing augmentation severity and male-female log WER ratios for English and Spanish data.Source: Figure 11 · p. 27, Figure 12 · p. 28
Loading 2403.07937v3…