Source-linked AI summary
Benchmarking_Fast_Domain_Adaptation_for_Unsupervised_Speech_Units
Robin San Roman, Manel Khentout, Tu Anh Nguyen, Paul Michel, Yossi Adi, Emmanuel Dupoux
TL;DR
Self-supervised speech representations must handle accent variation despite limited evidence about few-shot out-of-domain adaptation. The paper introduces ABX-Accent for ten English accents and evaluates adaptive domain normalization applied to pretrained CPC. On the benchmark, the method improves average across-speaker ABX scores by 23% relative to non-adapted models.
Problem
Self-supervised speech models have limited evidence for adapting to out-of-domain accented speech with few unlabelled examples, an issue linked to a long tail of accents.
Method
The paper introduces ABX-Accent, a ten-accent English benchmark, and evaluates adaptive domain normalization for fine-tuning pretrained CPC models.
Results
23% relative improvement in average across-speaker ABX scores is reported for the proposed method on the accent dataset.
Takeaways & Limitations
The benchmark highlights that accent adaptation remains difficult and leaves room for further domain-adaptation work.
Abstract
from arXiv · showhide
Representation learning has attracted great atten- tion and managed to reach good performances as a pretraining method for downstream tasks or as a first step towards unsu- pervised speech modeling. Yet, little is known about how such methods deal with out-of-domain speech and how could they be adapted in a few shot to new domains. This is important especially for accented speech where one observes a long tail of accents that diverge from the standard ones. We introduce ABX- Accent, a benchmark based on the AESRC dataset that features 10 different accents of English. It includes a small (< 10 hours) unlabelled training set in each of the accents and adaptations of the Zero Resources Challenge ABX evaluation metrics to each of the accents. We illustrate this benchmark with a baseline model that uses adaptive domain normalization to fine tune a pretrained Contrastive Predictive Coding model on the accents. This method is first developed on LibriSpeech using a male/female split. When applied to the new benchmark, the proposed method yields a relative improvement of 23.6% on across-speaker ABX scores on average compared to non adapted models. The data and metrics will be open sourced upon paper acceptance
CNRS, EHESS
Self-supervised speech representations are promising for downstream and textless applications, but their robustness to accent variation remains limited. The paper targets adaptation to new accents using little unlabelled data.
- Self-supervised speech learning uses unlabelled audio for downstream tasks and textless speech modeling.
- Accent distributions have a long tail, including geographically, temporally, and generationally varying accents and second-language accents.
- Accent variation creates inclusiveness and bias concerns for downstream applications built from speech datasets.
- ABX-Accent evaluates unsupervised adaptation to ten English accents using little unlabelled data and speaker-level adaptation data.The benchmark is built on AESRC and uses ABX to assess phonetic representation quality.
A. Representation learning
The paper uses contrastive predictive coding as an example of self-supervised speech representation learning and situates it within broader domain-adaptation approaches.
- Self-supervised speech methods learn features from unlabelled audio for applications including ASR, voice conversion, and emotion recognition.
- ABX-Accent organizes ten accents into separate training, development, and test sets with different speakers.
- Contrastive predictive coding is used as an illustration of the benchmark’s representation-learning baseline.
- Domain adaptation applies models trained on data-rich source domains to new domains with limited data, including different accents or speaking conditions.
A. Dataset
ABX-Accent is built from AESRC’s ten English accents with speaker-disjoint splits, limited unlabelled training data, and phoneme-aligned evaluation resources.
- The benchmark divides each of ten regional accents into balanced train, development, and test sets.Development and test sets each contain two hours per accent from six female and six male speakers.
- Each development and test speaker has an additional 2:00-minute adaptation set for speaker-specific statistics or embeddings.These sets support methods designed to compensate for variability among non-native speakers.
- The training set contains the remaining speakers and does not have transcriptions.
- Development and test transcriptions are phonemized and force-aligned to obtain phoneme timestamps.
- ABX error averages discrimination over minimal pairs of phone trigrams using representations of sounds such as “bap” and “bop.”
IV. ADAPTATION BASELINE
The baseline uses contrastive predictive coding to learn speech representations, with speaker-matched negative examples and domain-normalization variants evaluated on LibriSpeech splits.
- The baseline system is based on contrastive predictive coding and illustrates the benchmark’s adaptation challenges.
- CPC combines an encoder that produces embeddings with an autoregressive network that produces contextual representations.
- The training objective uses a linear classifier and a set of negative examples.
- Table II reports across-speaker ABX scores for models trained on one LibriSpeech sex and tested on the other, with lower scores better.The baseline averages 9.06% on old-domain tests and 11.86% on new-domain tests.
- Using negatives from the same speaker encourages representations to encode speech content rather than speaker identity.
B. Adaptive Domain normalization
The method uses adaptive normalization to make speech embeddings more robust to domain and speaker variation. It compresses frame information into domain-specific statistics before normalizing layer outputs.
- Adaptive normalization is applied in multiple settings to test resilience to new speakers.
- A nonlinear transformation reduces the computational cost of normalization.
- A low-dimensional projection uses a dg×p weight matrix with dg << p.
- Softmax weighting combines frames from domain d to form a domain context vector.
- The resulting domain information is used to produce the normalized output of layer l −1.
C. Domain Adaptation
Domain adaptation applies a data-rich pretrained model to new domains with limited data, including speech from new speakers, accents, or conditions. The paper targets rapid adaptation without the cost of retraining from scratch.
- Domain adaptation transfers a model pretrained on data-rich domains to new domains with limited available data.
- In speech processing, the new domains may involve different speakers, accents, or speaking conditions.
- The proposed fine-tuning approach aims to adapt quickly to out-of-domain data with limited computation time and data.
- Retraining from scratch may perform well but is costly and does not support fast adaptation to new speakers.
A. Librispeech Male/Female Experiment
The preliminary experiment adapts CPC models across male and female LibriSpeech domains using controlled fine-tuning splits. It compares new-domain-only fine-tuning with resampling that retains original-domain data and evaluates domain normalization.
- Experimental setup: The experiment trains models on LibriSpeech using male and female speakers as labeled domains.
- Experimental setup: The adaptation data uses 16 splits with different amounts of data and speakers.
- Fine-tuning strategy: The simple baseline fine-tunes using only data from the new domain.
- Fine-tuning strategy: Resampling oversamples the new domain so half of batches come from male speakers and half from female speakers.
- Evaluation: Table III reports across-speaker ABX scores for fine tuning with resampling and domain normalization on accented English test data.
- Results: Using original-domain data with even sampling stabilizes female-domain performance and enables more training epochs while improving average accuracy.
B. ABX-Accent Adaptation experiments
The ABX-Accent experiments transfer LibriSpeech-pretrained models to accented speech using the selected resampling strategy. Results favor domain-normalization warm-up followed by complete fine-tuning, while specialization and pooled data offer different advantages across accents.
- Experimental setup: The experiments fine-tune LibriSpeech-clean models with accented speech while sampling LibriSpeech according to the resampling strategy.
- Evaluation: ABX scores compare across-speaker performance within accents, including jointly trained models and separate models for each accent.
- Experimental setup: Joint experiments alternate samples across eleven domains: ten accents and LibriSpeech.
- Results: Domain-normalization warm-up followed by complete fine-tuning tends to achieve the best performance.
- Results: Models specialized to a specific domain achieve better results on average, although additional data from other domains can help in some cases.
- Results: For some accents, such as Russian, a single-domain specialized model achieves a better ABX score than a model trained on the complete training set.
VI. CONCLUSION
The paper presents an accent adaptation benchmark and shows adaptive normalization improves ABX scores, with smaller gains on accents than on a toy male/female split.
- The benchmark evaluates self-supervised speech representation adaptation to accents using baseline systems with adaptive normalization.
- 33% relative ABX-score improvement was observed in a toy male/female LibriSpeech domain split.
- 23% relative improvement was observed on the accent dataset, indicating greater difficulty and remaining room for improvement.