Source-linked AI summary
Self-training and Pre-training are Complementary for Speech Recognition
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, Michael Auli
TL;DR
It was unclear whether self-training and unsupervised pre-training learn similar patterns or can be effectively combined. This paper combines pseudo-labeling with wav2vec 2.0 pre-training and finds them complementary across labeled-data setups, achieving 3.0%/5.2% WER with 10 minutes of labeled data and 1.5%/3.1% with all Librispeech labels.
Problem
Self-training and unsupervised pre-training both use unlabeled speech effectively, but whether they learn similar patterns or are complementary was unclear.
Method
The paper combines wav2vec 2.0 unsupervised pre-training with pseudo-labeling, using a pre-trained model to label unlabeled data before training the final acoustic model.
Results
3.0%/5.2% WER is achieved with 10 minutes of labeled data, while using all 960 hours of Librispeech labeled data achieves 1.5%/3.1% WER.
Takeaways & Limitations
Self-training and unsupervised pre-training are complementary, enabling strong speech recognition with as little as 10 minutes of transcribed speech.
Takeaways & Limitations
The study uses a single pseudo-labeling iteration and leaves iterative pseudo-labeling and distribution-based filtering for future work.
Abstract
from arXiv · showhide
Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data. However, it is not clear whether they learn similar patterns or if they can be effectively combined. In this paper, we show that pseudo-labeling and pre-training with wav2vec 2.0 are complementary in a variety of labeled data setups. Using just 10 minutes of labeled data from Libri-light as well as 53k hours of unlabeled data from LibriVox achieves WERs of 3.0%/5.2% on the clean and other test sets of Librispeech - rivaling the best published systems trained on 960 hours of labeled data only a year ago. Training on all labeled data of Librispeech achieves WERs of 1.5%/3.1%.
1 Introduction
The paper asks whether self-training and unsupervised pre-training learn complementary patterns and shows that combining them improves speech recognition across labeled-data regimes.
- Supervised speech recognition requires substantial labeled data, limiting its practicality for most of the world’s 7,000 languages.
- Self-training pseudo-labels unannotated audio before retraining, whereas pre-training learns speech representations from unlabeled audio before supervised fine-tuning.
- The paper combines wav2vec 2.0 pre-training with self-training and tests training from scratch versus fine-tuning on pseudo-labeled data.
- The methods are complementary across full-Librispeech and Libri-light labeled-data setups.
- 3.0%/5.2% WER is achieved with 10 minutes of labeled data and LibriVox audio on Librispeech clean/other test sets.
- 1.5%/3.1% WER is achieved when all Librispeech labeled data are used.
2 Background
The paper reviews wav2vec 2.0 representation learning and defines self-training as pseudo-labeling unlabeled audio before final acoustic-model training, then combines both pipelines.
- wav2vec 2.0 maps raw audio to latent representations, transforms them with a Transformer, and discretizes encoder outputs to training targets.
- Its training objective identifies the true quantized latent for a masked time step among 100 distractors using cosine similarity.
- Self-training first trains an acoustic model on labeled data, uses it with a language model to label unlabeled audio, then trains a new model on both datasets.
- The experiments use one pseudo-labeling iteration because it is less computationally demanding, while iterative labeling and distribution filtering are left for future work.
- The combined pipeline pre-trains wav2vec 2.0, fine-tunes it on labeled data, pseudo-labels unlabeled audio, and trains the final model on the resulting labels.
3 Experimental Setup
Experiments vary labeled and unlabeled speech resources, using wav2vec 2.0 models to generate pseudo-labels and comparing sequence-to-sequence training with CTC fine-tuning.
- The study uses 960 hours of Librispeech audio or 53.2k hours of LibriVox audio as unlabeled data across multiple labeled-data setups.
- The pre-trained LARGE model has about 300M parameters and uses 24 Transformer blocks with dimension 1,024 and 16 attention heads.
- Pseudo-labels are generated with wav2vec 2.0 LARGE fine-tuned on each labeled split, followed by 4-gram beam search and Transformer-language-model rescoring.
- The final acoustic model uses a convolutional frontend followed by 36 Transformer blocks and contains about 300M parameters.
- Word-piece vocabularies use 10k units for full Librispeech training and 5k units otherwise, with 4-gram and Transformer language models used in decoding.
4 Results
Across low- and high-resource settings, combining pre-training with self-training improves speech recognition, with gains depending strongly on available unlabeled data and final-model design.
- 4.1 Low-Resource Labeled Data: wav2vec 2.0 plus self-training outperforms pre-training alone across all Libri-light low-resource setups.
- 4.1 Low-Resource Labeled Data: 5.2% WER on test-other is achieved with 10 minutes of labeled data and LibriVox pseudo-labeling.
- 4.1 Low-Resource Labeled Data: 3.0%/5.2% WER results from increasing unlabeled audio from LS-960 to LV-60k, a 25-28% relative WER reduction.
- 4.1 Low-Resource Labeled Data: Fine-tuning generally beats scratch sequence-to-sequence training in smaller setups, while scratch sequence-to-sequence training is stronger with all labeled data.
- 4.2 High-Resource Labeled Data: With 100 hours of labeled data, LS-960 unlabeled audio does not beat the baseline, whereas LV-60k produces a 10% relative WER reduction on test-other.
- 4.2 High-Resource Labeled Data: 1.5%/3.1% WER is achieved on Librispeech when the full labeled benchmark is combined with wav2vec 2.0 and pseudo-labeling.
- 4.3 Results without a Language Model at Inference Time: Combined models retain strong performance without an inference-time language model because pseudo-labeling partly distills language-model information into training data.
5 Analysis
The analysis finds that self-training gains depend more on the amount of unlabeled data relative to labeled data than on labeled data alone. With a fixed unlabeled-to-labeled ratio, relative improvements become comparable across labeled-data setups.
- Data scaling: Larger improvements appear in the 10-minute labeled split when the amount of unlabeled data is not held at a fixed ratio.The analysis reports that improvements decreased as more labeled data became available.
- Data scaling: The main driver of performance is the ratio between labeled and unlabeled data, rather than the amount of labeled data alone.Table 4 adds 8.6 times as much unlabeled data to each labeled setup.
6 Conclusion
The conclusion is that unsupervised pre-training and pseudo-labeling are complementary for speech recognition. Their combination enables systems trained with as little as 10 minutes of transcribed speech to reach strong word error rates.
- Unsupervised pre-training and pseudo-labeling are complementary for speech recognition.
- With as little as 10 minutes of transcribed speech, the approach achieves word error rates that were recently associated with systems trained on 960 hours of labeled data.