Source-linked AI summary
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, Michael Auli
TL;DR
The paper addresses limited evidence about domain mismatch across self-supervised pre-training, supervised fine-tuning, and testing. It experimentally varies these domains with wav2vec 2.0 and finds that in-domain unlabeled pre-training improves performance, closes 66%-73% of a key performance gap, and multi-domain pre-training improves unseen-domain generalization.
Problem
The impact of mismatch among unlabeled pre-training, labeled fine-tuning, and test domains in self-supervised speech representation learning is less studied.
Method
The paper runs controlled wav2vec 2.0 experiments varying pre-training domains and amounts, fine-tuning domains and resources, and evaluations on seen and unseen domains.
Results
Pre-training on unlabeled in-domain data improves performance across setups and closes 66%-73% of the gap between ideal in-domain-labeled and competitive supervised out-of-domain models.
Takeaways & Limitations
Unlabeled target-domain data can support fine-tuning with existing out-of-domain labels, while multi-domain pre-training improves generalization to unseen domains.
Abstract
from arXiv · showhide
Self-supervised learning of speech representations has been a very active research area but most work is focused on a single domain such as read audio books for which there exist large quantities of labeled and unlabeled data. In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain. Our experiments show that using target domain data during pre-training leads to large performance improvements across a variety of setups. On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%. This has obvious practical implications since it is much easier to obtain unlabeled target domain data than labeled data. Moreover, we find that pre-training on multiple domains improves generalization performance on domains not seen during training. Code and models will be made available at https://github.com/pytorch/fairseq.
1 Introduction
Prior self-supervised speech-representation studies largely assume little domain mismatch across pre-training, fine-tuning, and testing. This paper examines those mismatches and finds benefits from target-domain and multi-domain unlabeled pre-training.
- Most prior studies focus on settings with little mismatch between unlabeled pre-training, labeled fine-tuning, and test domains.
- Domain mismatch is well known to degrade conventionally supervised ASR, but its impact on self-supervised speech representation learning is less studied.
- Experiments vary in-domain and out-of-domain unlabeled data while fine-tuning on both in-domain and out-of-domain labeled data.
- Unlabeled data matching the test domain always improves performance, even when fine-tuning uses out-of-domain labeled data.
- Pre-training on multiple domains increases robustness to domains unseen during both pre-training and fine-tuning.
2 Related Work
The paper situates its self-supervised approach among feature-based and model-based robust ASR methods. It extends prior domain-shift work by dissecting pre-training-data domains while updating representations during fine-tuning.
- Robust ASR research includes feature-based approaches using signal-processing or learned features.
- Model-based approaches expose models to diverse data through augmentation, target-domain self-training, adversarial training, or joint training.
- This paper’s self-supervised approach is model-based and requires no labeled data during pre-training.
- The closest prior work studied domain shift but did not separate the domains used for pre-training.
- Unlike that prior work, this paper examines pre-training-data domains and updates the pre-trained feature extractor during supervised fine-tuning.
3 Experimental Setup
The experiments use wav2vec 2.0 across varied English speech domains, pre-training mixtures, fine-tuning resource levels, and domain-shift evaluations. The setup includes controlled ablations and scaled comparisons with prior work.
- Model: wav2vec 2.0 maps raw audio through a convolutional encoder and Transformer, using quantized latent representations as masked-training targets.
- Experiment design: The study defines in-domain and out-of-domain relative to test data and evaluates mismatches across pre-training, fine-tuning, and testing.
- Experiment design: Experiments vary domain similarity at fixed pre-training size, quantify effects of in-domain pre-training amount, and scale data and model size.
- Data: The datasets cover six English domains spanning audiobooks, conferences, telephone conversations, crowdsourced speech, read news, and parliamentary speech.
- Data: Most experiments pre-train on combinations of LS, TD, and SF, while scaling experiments combine LL, SF, and CV.
- Training: BASE is used throughout except for the final scaling section, which uses LARGE; models are trained for 400K or 800K steps depending on the experiment.
- Training: Fine-tuning uses 10-hour LS, SF, and TD subsets for low-resource experiments, a 100-hour TD subset for mid-resource evaluation, and full LS and SB for high-resource evaluation.
- Evaluation: Models are fine-tuned with connectionist temporal classification and evaluated using target-domain word error rate.
4 Results
Results show that adding in-domain unlabeled data improves validation performance when fine-tuning and testing match that domain. The experiments also compare out-of-domain pre-training additions using Table 2’s validation WER setups.
- 4.1 Does adding in-domain pre-training data help?: The table marks fine-tuning setups as out-of-domain when applicable and identifies models additionally pre-trained on in-domain data.
- 4.1 Does adding in-domain pre-training data help?: Adding in-domain pre-training data lowers every compared WER when fine-tuning and testing use the same domain.
- 4.1 Does adding in-domain pre-training data help?: Table 2 reports validation WER on TD, LS, and SB for seven pre-training combinations paired with TD-10h, LS-10h, or SB-10h fine-tuning.
4.2 Does adding pre-training data help if out-of-domain?
Adding out-of-domain unlabeled data generally improves performance when in-domain pre-training data is unavailable, while adding more domains after an initial match yields mixed results. Models pre-trained on more domains also tend to generalize better to unseen domains.
- Adding a second out-of-domain pre-training dataset consistently improves performance when the original pre-training data contains no in-domain data.The comparison is between models pre-trained on one versus two out-of-domain datasets.
- Adding a second pre-training domain usually improves results when the original pre-training data already includes in-domain data.The only reported exception is SB RT03 WER when fine-tuning on SB-10h.
- Increasing pre-training from two to three domains produces mixed results, with improvements in about half of the cases.
- Pre-training on more domains tends to improve performance on domains unseen during both pre-training and fine-tuning.The unseen domains are Wall Street Journal, Common Voice, and VoxPopuli.
4.4 Is it still effective and robust with more labeled data?
With a larger TD-100h labeled fine-tuning set, both in-domain and out-of-domain pre-training additions remain effective on LS dev-other.
- Adding in-domain unlabeled data remains effective when fine-tuning on TD-100h and testing on LS dev-other.
- Adding out-of-domain unlabeled data also remains effective with the larger TD-100h labeled set.The comparison covers successive pre-training configurations in Table 4.
- Table 4 reports the effect of more labeled data using LS dev-other WER.
4.5 Effect of pre-training data similarity to target domain
Controlling pre-training size shows that target-domain similarity affects performance, but matching the fine-tuning domain also matters when labeled and target domains differ.
- Fixing pre-training at 450 hours and varying the TD/LS ratio isolates the effect of domain similarity to LS dev-other.
- Increasing in-domain unlabeled data improves performance up to 50% of all pre-training data.
- With perfect domain matching across unlabeled, labeled, and target data, more unlabeled data consistently improves performance.
- When the labeled domain differs, performance saturates at 50% in-domain data for TD-10h fine-tuning and 75% for SB-10h fine-tuning.
- The authors believe some unlabeled data should match the fine-tuning domain for fine-tuning to be most beneficial.This effect is reported as particularly strong for TD-10h.
4.6 Effect of in-domain pre-training data size
More in-domain unlabeled data continually improves performance under both joint and continual pre-training, with similar results from the two strategies.
- Joint training pre-trains on TD and LS together, whereas continual training adds LS-only pre-training after initial TD pre-training.Joint training uses TD + LS for 400k steps; continual training first uses TD for 400k steps, then LS for additional steps.
- Adding more in-domain unlabeled data continually improves performance for both joint and continual training.
- Joint and continual training achieve similar performance.
- Using all in-domain unlabeled data remains effective with TD-10h fine-tuning because pre-training includes all TD data.
4.7 Larger model, more pre-training and fine-tuning data
The paper scales wav2vec 2.0 pre-training to a 300M-parameter model trained across three domains, then evaluates fine-tuning with limited and full labeled data. The resulting model performs strongly in-domain and generalizes to out-of-domain test sets, including domains absent from training.
- Larger model, more pre-training and fine-tuning data: A 300M-parameter wav2vec 2.0 model was pre-trained on LL, SF, and CV for 800K steps.It was fine-tuned on 10 hours and all available data from LS and SB, then evaluated on LS, SB, CV, and TD validation and test splits.
- Larger model, more pre-training and fine-tuning data: 35% to 50% relative WER reductions were obtained on SB, CV, and TD when fine-tuning on LS versus the full-LS single-dataset baseline.The comparison uses the baseline supervised model trained on full LS data.
- Larger model, more pre-training and fine-tuning data: The model improved out-of-domain performance even when fine-tuned on only 10 hours of LS data.The same trend held for SB-10h, compared with a baseline trained on all SF, which used 200x more labeled data.
- Larger model, more pre-training and fine-tuning data: Without TD pre-training or fine-tuning, the LS-fine-tuned model outperformed supervised systems trained on TD and on five labeled datasets including TD.These comparisons concern TD performance.
- Larger model, more pre-training and fine-tuning data: On LS/SB in-domain data, the model outperformed all single- and multi-domain models in except on CallHome (H-CH).The comparison is against the models reported in.
4.8 Effectiveness of pre-training
The paper evaluates whether unlabeled in-domain pre-training can compensate for unavailable in-domain labels while fine-tuning on labeled out-of-domain data. Across earlier and larger competitive systems, this approach closes most of the gap to ideal in-domain supervision.
- Effectiveness of pre-training: Unlabeled in-domain data is often easier to obtain than transcribed in-domain data.This creates a practical alternative to collecting labeled data for a new domain.
- Effectiveness of pre-training: Pre-training on in-domain unlabeled data closes at least 73% of the performance gap across varied settings.The gap is measured between ideal in-domain labeled training and out-of-domain labeled training, with the proposed setup using in-domain unlabeled pre-training and OOD labeled fine-tuning.
- Effectiveness of pre-training: 66%-73% of the performance gap is closed on larger competitive systems by in-domain unlabeled pre-training with OOD fine-tuning.The comparison is between the ideal Topline and a competitive supervised Baseline evaluated on Switchboard sets.
- Effectiveness of pre-training: WERR is defined as (proposed −baseline)/(topline −baseline).This ratio measures how much of the gap between the OOD-only baseline and ideal in-domain topline is closed by the proposed setup.
5 Conclusion
The paper presents a controlled study of domain shift in self-supervised ASR and finds that in-domain unlabeled pre-training improves performance despite OOD fine-tuning. Multi-domain representations also generalize to unseen domains, while retaining some fine-tuning-domain unlabeled data remains beneficial.
- Conclusion: The study examines domain shift in self-supervised learning for ASR through controlled experiments.It focuses on mismatches among pre-training, fine-tuning, and test domains.
- Conclusion: Adding unlabeled in-domain data improves performance even when fine-tuning data does not match the test domain.This finding is reported across the paper’s domain-shift experiments.
- Conclusion: 66-73% of the performance gap is closed without access to in-domain labeled data.The comparison is between ideal in-domain labeled data and a competitive supervised out-of-domain model.
- Conclusion: Representations trained on multiple domains improve generalization to domains unseen during both pre-training and fine-tuning.The conclusion characterizes these representations as robust across such unseen domains.
- Conclusion: Retaining some unlabeled data from the fine-tuning domain remains beneficial.This is stated as an additional conclusion of the study.