Source-linked AI summary
Acoustic domain shift in spoken language identification from systematic domain generalization evaluation to real-world application
Francois Derrida, Raphaël Duroselle, Thomas Courtat, Jean-François Bonastre
TL;DR
Systematic evaluation of domain generalization for spoken language identification under acoustic shifts remains limited, while existing speech datasets are often designed for specific scenarios and confound controlled comparisons. This paper introduces a compact dataset and DomainBed-based protocol, then finds that in-domain performance does not reliably predict transfer and explicit invariance methods do not outperform ERM. The findings are also examined on a 126-language system.
Problem
Spoken language identification lacks systematic evaluation across controlled, diverse acoustic domain shifts, while existing speech datasets often focus on specific scenarios and include confounding variation.
Source: §I · p. 1Method
The paper builds a compact speech dataset and extends the DomainBed evaluation protocol to speech, testing three domain-generalization algorithms and validating findings on a 126-language system.
Source: §I · p. 1Results
In-domain performance does not reliably predict out-of-domain robustness, and MMD or DANN do not outperform ERM; these patterns are broadly consistent across FLEURS and VoxLingua107.
Source: C. Results · p. 3, B. Results · p. 4Takeaways & Limitations
The paper concludes that robustness is driven primarily by exposure to diverse acoustic conditions, while transfer depends strongly on the source-target relationship.
Source: §V · p. 4Corruption transfer is asymmetric, and difficult unseen distortions can remain challenging even after multi-domain training.
Source: §V · p. 4
Abstract
from arXiv · showhide
Domain Generalization (DG) aims to develop models that remain robust to conditions unseen during training. While DG has been systematically studied in computer vision through controlled benchmarks and diverse distribution shifts, its evaluation in spoken language recognition remains less structured. Existing speech datasets provide valuable benchmarks for robustness to real-world acoustic conditions, but are primarily designed around specific scenarios and large scale rather than as general-purpose tools for systematically constructing and evaluating domain shifts. In this work, we introduce a smallscale speech dataset and evaluation protocol for controlled studies of acoustic domain shifts. It enables the evaluation of spoken language recognition models under a variety of acousitc domain shifts. We introduce the speech modality into the DomainBed Domain Generalization framework and evaluate three domain generalization algorithms. We show that in-domain performance is not a reliable predictor of cross-domain robustness and verify that explicit domain invariant algorithms such as MMD or DANN algorithms do not outperform ERM. We further validate the generality of these findings on MMS-LID-126, a state-of-the-art spoken language identification system. We release the code.
I. INTRODUCTION
The paper addresses the limited systematic evaluation of domain generalization for spoken language identification under controlled acoustic shifts. It proposes a compact speech dataset and DomainBed-based evaluation, then checks whether findings extend to a state-of-the-art system.
I. INTRODUCTION
- Existing speech noise-robustness datasets are often large and confound noise effects with speaker, channel, and environment variability.The authors identify this as a reason those datasets are less suitable for controlled studies of noise-induced domain shift.Source: §I · p. 1
- Prior LID studies target specific corpus, channel, or corruption shifts, leaving controlled evaluation across diverse acoustic shifts and target conditions less explored.The paper frames this as a broader question of how to characterize shifts, assess their impact, and select DG strategies.Source: §I · p. 1
- The work extends DomainBed, a unified framework for evaluating DG algorithms across image datasets, to application-relevant acoustic conditions in speech.The paper notes that existing DG benchmarks do not include small speech-related datasets.Source: §I · p. 1
- The paper introduces a compact speech dataset suited to systematic studies of corruption shifts in language identification.The authors also present a protocol for evaluating three DG algorithms under acoustic corruption shifts.Source: §I · p. 1
- The authors validate the generality of their findings on a state-of-the-art language-identification system covering 126 languages.This extends the evaluation beyond the small-scale benchmark setting.Source: §I · p. 1
II. SPEECH CORRUPTION SHIFT DATASET GENERATION
This section introduces a methodology for generating CWWS, a speech dataset designed for domain-generalization studies of language identification under corruption shifts.
II. SPEECH CORRUPTION SHIFT DATASET GENERATION
- CWWS is a speech dataset generated for domain-generalization classification studies in language identification.Its stated focus is evaluation under corruption shifts.Source: §II · p. 2
A. Corruption shift domains
The study treats each domain as a speech-and-language distribution and focuses on shifts created by corrupting speech while retaining its label. It evaluates clean, noisy, and distorted conditions, with a larger application also including codec and demand-noise shifts.
A. Corruption shift domains
- A domain is a distribution over speech utterances and language labels; domain generalization trains on source domains to transfer to an unseen target domain.The setup follows the definition in.Source: A. Corruption shift domains · p. 2
- The study assumes each shifted domain results from applying a potentially stochastic speech transformation while keeping the original language label.This isolates acoustic corruption effects, but is a simplification because acoustic conditions can alter speech production, as in the Lombard effect.Source: A. Corruption shift domains · p. 2
- The described speech conditions are clean original speech, WHAM! background conversational-speech additive noise, wind-simulator-generated additive noise, and saturation that induces signal clipping and distortion.Saturation uses a distortion gain sampled uniformly in [0.5, 4.5].Source: A. Corruption shift domains · p. 2
- The large-scale application also includes DEMAND noise and LPC10 codec distortion; additive-noise SNRs span [-10, 10] dB.The full set of corruptions is used in Section IV, while the DomainBed study uses clean, WHAM, wind, and saturation.Source: A. Corruption shift domains · p. 2
B. Source speech data
The dataset construction controls comparisons by preserving the original corpus splits and matching test utterances across corruption domains. It also equalizes training-set size across methods and assigns each speaker to one domain.
B. Source speech data
- Dataset design preserves the source corpus splits, uses the same test utterances across domains, and gives each method the same number of training utterances.Each speaker is assigned to only one domain to support independent sampling across domains.Source: B. Source speech data · p. 2
III. SYSTEMATIC ANALYSIS OF CORRUPTION DOMAINS AND ADAPTATION ALGORITHMS USING LID TASK
The study applies DomainBed to controlled speech corruption shifts, evaluating three algorithms across source–target configurations. Its protocol includes fair hyperparameter selection and confidence intervals based on test-set size.
A. Evaluation with DomainBed
- DomainBed evaluates three domain-generalization algorithms, with multiple source domains treated as one aggregated source in a binary source–target problem.The algorithms are ERM, MMD, and DANN.Source: A. Evaluation with DomainBed · p. 2
- Each algorithm is tested on four target domains using keep-one-out configurations and compared with mono-source experiments.The four targets are clean, wham, wind, and saturation; mono-source training uses one domain.Source: A. Evaluation with DomainBed · p. 2
- Each configuration receives a 20-experiment grid search, with models selected using source-domain validation or target-domain validation as an Oracle.This amounts to 80 runs per algorithm.Source: A. Evaluation with DomainBed · p. 2
- Uncertainty is measured with 95% bootstrap confidence intervals over test-set size rather than random-seed variation.The authors note this differs from the original DomainBed framework.Source: A. Evaluation with DomainBed · p. 2
B. Data and model architecture
The study uses a language-balanced CommonVoice dataset and a fixed training budget for domain-generalization and mono-source configurations. Its fine-tuned wav2vec2 model uses a pretrained VoxPopuli backbone.
B. Data and model architecture
- All configurations use 4,500 training utterances, split across three domains for domain generalization or drawn from one domain for mono-source training.The domain-generalization split uses 1,500 utterances per domain; the dataset statistics are summarized in Table II.Source: B. Data and model architecture · p. 3, Table II · p. 3
- The dataset uses CommonVoice v22 speech in French, Spanish, German, and Mandarin Chinese.The dataset sets are language-balanced.Source: B. Data and model architecture · p. 3, Table II · p. 3
- The model combines a pretrained wav2vec2 encoder with mean pooling and a classification head, using a VoxPopuli-pretrained backbone disjoint from CommonVoice.The disjoint corpora are used to reduce potential information leakage.Source: B. Data and model architecture · p. 3
- Backbone-only pretraining yields near-random classification, making fine-tuning essential for the considered shifts.Those results are omitted for brevity.Source: B. Data and model architecture · p. 3
C. Results
The results show that multi-source domain generalization often improves on average mono-source out-of-domain performance, but explicit invariance objectives do not consistently outperform ERM. In-domain accuracy and cross-domain usefulness can diverge, with asymmetric transfer between corruption domains.
C. Results
- Wham-to-wind accuracy is 61.1%, while wind-to-wham accuracy is 82.6%, demonstrating asymmetric out-of-domain generalization.The authors note both are additive corruptions.Source: C. Results · p. 3
- In-domain performance exceeds out-of-domain performance, and domain difficulty differs from usefulness as a source: wind is difficult in-domain but useful for generalization.Clean is relatively easy in-domain, while wind is more difficult and wham lies between them.Source: C. Results · p. 3
- Multi-source domain generalization outperforms average mono-source out-of-domain performance and, for some domains, in-domain performance, but MS-clean on wham exceeds domain generalization.ERM is not outperformed by MMD or DANN, and Oracle selection adds only marginal gains.Source: C. Results · p. 3
- These findings motivate testing whether the same behavior persists in a realistic large-scale language-identification setting.Source: C. Results · p. 3
IV. LARGE-SCALE ROBUST LANGUAGE IDENTIFICATION
The paper extends its controlled analysis to the reference MMS-LID-126 system to assess whether earlier findings generalize at larger scale. The controlled setting does not reflect the scale of systems handling hundreds to thousands of languages.
IV. LARGE-SCALE ROBUST LANGUAGE IDENTIFICATION
- The controlled experimental setting does not reflect modern language-identification systems that handle hundreds to thousands of languages.Source: §IV · p. 3
- The large-scale system results report test accuracy on FLEURS and VoxLingua107.Table III identifies the evaluated corpora but the supplied passage does not provide its values.Source: Table III · p. 4
- The authors extend their analysis to the reference MMS-LID-126 system to validate earlier findings on a large-scale system.The experiments also aim to produce a robust language-recognition system.Source: §IV · p. 4
- The LID training recipe uses hyperparameters from the reference system, while domain-generalization hyperparameters come from the preceding section.Source: §IV · p. 4
A. Domain generalization with mms-lid-126
The MMS-LID-126 evaluation uses a multilingual model trained on FLEURS and VoxLingua107 and assesses performance across six acoustic-corruption domains.
A. Domain generalization with mms-lid-126
- MMS-LID-126 replaces the wav2vec2 encoder with MMS-1B while retaining the architecture described in Section III-B.MMS was pretrained on a diverse corpus spanning more than 44.7K hours and 126 languages.Source: A. Domain generalization with… · p. 4
- Training on the union of FLEURS and VoxLingua107 yields a 126-language system spanning audiobook and noisier YouTube speech.The two datasets correspond to different speech styles and acoustic conditions.Source: A. Domain generalization with… · p. 4
- The evaluation covers six domains, naming clean, wind, and saturation as sources and wind, LPC10, and DEMAND as unseen during finetuning.The passage names wind in both lists.Source: A. Domain generalization with… · p. 4
B. Results
The results extend earlier qualitative patterns to a larger language inventory and across two corpora, while showing that acoustic-corruption transfer depends on source-target relationships and can remain difficult after multi-domain training.
B. Results
- MMS-LID-126 loses performance under every corruption shift and is less robust than MS-clean despite similar clean-domain performance.The authors hypothesize that MMS-LID-126's greater training-data exposure explains its greater robustness.Source: B. Results · p. 4
- The source-performance and transferability relationship, Wham's relative transferability to close domains such as DEMAND, and the broad ERM/MMD/DANN pattern persist across FLEURS and VoxLingua107.The qualitative observations also persist with the larger language inventory, despite differences in absolute performance between the corpora.Source: B. Results · p. 4
- LPC10 results indicate that domain diversity helps but does not always ensure robustness to arbitrary unseen corruptions.Source: B. Results · p. 4
V. CONCLUSION
- Robustness is associated primarily with exposure to diverse acoustic conditions rather than explicit domain invariance, but transfer is asymmetric and difficult unseen distortions remain challenging.The conclusion identifies source-target relationships as an important boundary on corruption transfer.Source: §V · p. 4