Source-linked AI summary
Self-Training for End-to-End Speech Recognition
Jacob Kahn, Ann Lee, Awni Hannun
TL;DR
End-to-end speech recognition performance degrades when less transcribed data is available, motivating methods that use abundant unpaired audio and text. This paper applies self-training with filtered pseudo-labels and sample ensembles, achieving higher WER recovery than prior sequence-to-sequence semi-supervised methods on LibriSpeech.
Problem
End-to-end speech recognition needs substantial transcribed data, while self-training has not been carefully studied for sequence-to-sequence models.
Method
The approach generates pseudo-labels with a strong acoustic model and language model, filters sequence-to-sequence errors, and combines bootstrapped models through sample ensembles.
Results
93.8% relatively higher WER recovery rate is achieved on the clean test set than other semi-supervised sequence-to-sequence methods.
Takeaways & Limitations
Self-training with tailored filtering and ensembles substantially improves end-to-end systems and establishes a reproducible LibriSpeech benchmark for future semi-supervised methods.
Abstract
from arXiv · showhide
We revisit self-training in the context of end-to-end speech recognition. We demonstrate that training with pseudo-labels can substantially improve the accuracy of a baseline model. Key to our approach are a strong baseline acoustic and language model used to generate the pseudo-labels, filtering mechanisms tailored to common errors from sequence-to-sequence models, and a novel ensemble approach to increase pseudo-label diversity. Experiments on the LibriSpeech corpus show that with an ensemble of four models and label filtering, self-training yields a 33.9% relative improvement in WER compared with a baseline trained on 100 hours of labelled data in the noisy speech setting. In the clean speech setting, self-training recovers 59.3% of the gap between the baseline and an oracle model, which is at least 93.8% relatively higher than what previous approaches can achieve.
1. INTRODUCTION
The paper revisits self-training for end-to-end speech recognition, using pseudo-labels from strong models, tailored filtering, and ensembles to improve performance on LibriSpeech.
- End-to-end speech recognition requires substantial transcribed data, while transcription is expensive and time-consuming.
- The approach trains a strong baseline acoustic model and language model to generate pseudo-labels from unpaired audio and text.
- Filtering methods target looping, early stopping, and other sequence-to-sequence errors, while an ensemble increases pseudo-label diversity.
- 59.3% of the baseline-to-oracle gap is recovered in the clean speech setting through self-training.
- 93.8% relatively higher WER recovery rate is achieved than other semi-supervised sequence-to-sequence methods on the clean test set.
2. MODEL
The model is an attention-based encoder-decoder whose inference combines sequence-to-sequence and language-model scores during beam search, with decoding controls for stability.
- The sequence-to-sequence model uses an encoder-decoder architecture with attention.
- A fully convolutional encoder with time-depth separable blocks maps speech frames to key-value hidden representations.
- The RNN decoder encodes the previous token and query vector to produce the next query vector, while attention produces a summary vector.
- 2.1. Inference: Beam search finds the most likely hypothesis using sequence-to-sequence and external language-model scores.
- 2.1. Inference: The language-model weight α and token insertion term β help control decoding, with β addressing early stopping.
3. SEMI-SUPERVISED SELF-TRAINING
Semi-supervised self-training expands a labelled speech-recognition dataset with filtered pseudo-labels generated from unlabelled audio. The approach also uses sample ensembles to increase pseudo-label diversity while avoiding heavier joint decoding.
- Supervised learning: A supervised acoustic model is trained on paired data by maximizing the likelihood of ground-truth transcriptions given utterances.
- Semi-supervised self-training: An acoustic model and language model generate pseudo-labels for unlabelled audio, producing a pseudo-paired dataset for subsequent acoustic-model training.The new model is trained using both the original paired dataset and the pseudo-paired dataset.
- Filtering: Filtering balances pseudo-labelled dataset size against transcription noise using sequence-to-sequence-specific heuristics and conventional confidence filtering.The methods are applied at the sentence level.
- Filtering: Looping errors are filtered by removing pseudo-labels with excessively repeated n-grams, while early-stopping errors are addressed through EOS-probability and complete-hypothesis checks.
- Filtering: Length-normalized log likelihood from the sequence-to-sequence model serves as the confidence score for each pseudo-label.The utterance length is represented by the number of tokens in the pseudo-label.
- Ensembles: Sample ensemble generates separate pseudo-labelled datasets from M bootstrapped acoustic models in parallel and combines them with uniform weights during training.This increases pseudo-label diversity without the heavyweight decoding required when combining model scores during inference.
4. EXPERIMENTS
Experiments on LibriSpeech evaluate self-training with clean and noisy unpaired speech, focusing on filtering, ensembles, and comparison with prior semi-supervised methods. Filtering improves pseudo-label quality, while the strongest gains combine filtering with model ensembles.
- Experimental setup: Experiments use 100 hours of paired clean speech, plus 360 hours of unpaired clean speech or 500 hours of unpaired noisy speech.Results are reported on standard clean and noisy development and test sets.
- Filtering: Filtering improves pseudo-label quality as the confidence threshold becomes more selective, using heuristic “no EOS + n-gram” filtering followed by confidence-based filtering.The heuristic uses c = 2 and n = 4.
- Filtering: 5.2% relative WER improvement on dev clean results from removing the heuristic-filtered 1.8% of data plus the worst 10% by confidence.More aggressive filtering further improves label quality but worsens model performance.
- Filtering: 22.7% relative lower WER on dev other is achieved by filtering 60% of labels in the noisy setting compared with no filtering.Removing the worst 10% already significantly reduces WER, while filtering beyond the optimum degrades performance.
- Model ensembles: 13.7% relative improvement with six models and heuristic filtering is obtained in noisy speech, where ensembles are especially effective.Combining both filtering methods with ensembles yields 27.0% relative lower WER than a single unfiltered model.
- Comparison with literature: The best model achieves over 50% WER recovery rate with external LM decoding in both clean and noisy speech settings.WRR measures the gap bridged between supervised baseline and oracle WER.
- Comparison with literature: 65.1% relatively lower WER than previous sequence-to-sequence semi-supervised results is achieved, while WRR is at least 93.8% relatively higher.The reported gains use a strong TDS-based baseline and a larger unpaired text corpus.
5. RELATED WORK
Prior end-to-end semi-supervised speech-recognition methods use synthetic speech, cycle consistency, or shared speech-text embeddings, whereas this work studies direct self-training.
- Prior self-training: Earlier hybrid-system work emphasizes confidence-, agreement-, and multi-level data filtering for self-training.Selection can operate from frames to utterances, and soft pseudo-labels have also been explored.
- End-to-end methods: Recent end-to-end methods generate synthetic speech with TTS, impose ASR–TTS cycle consistency, or constrain speech and text in a shared embedding space.The paper positions its self-training approach as simple and effective for end-to-end systems.
6. CONCLUSION
The paper shows that self-training can substantially improve end-to-end speech recognition over a strong baseline by exploiting large unlabelled datasets. Sequence-to-sequence-specific filtering and model ensembles further improve the gains and support a reproducible benchmark.
- Conclusion: Self-training substantially improves end-to-end systems over a strong baseline by leveraging a large unlabelled dataset.The experiments establish a reproducible semi-supervised setting for future evaluation.
- Conclusion: Filtering tailored to sequence-to-sequence errors and ensembles of models further improve self-training accuracy gains.The conclusion identifies both mechanisms as complementary contributors to performance.
A.1. Supervised Baseline
The paper establishes a strong, practical supervised baseline on LibriSpeech’s train-clean-100 subset, improving on prior dev-set results while matching prior test performance.
- The TDS baseline achieves better WER on the development sets than the best prior result limited to train-clean-100.
- The TDS baseline has similar test WER to the best prior result under the same train-clean-100 constraint.
- The authors position this baseline as a challenging yet practical starting point for semi-supervised experiments.
- The baseline is intended to make improvements from adding unlabelled audio or text data more meaningful.
A.2. Evaluating Beam Search
The study compares greedy, simple-beam, and stable-beam decoding for pseudo-label generation. Language-model-assisted decoding improves pseudo-label quality, while stable beam search further improves resulting WER.
- Stable beam search combines the acoustic and language models using the EOS threshold and hard attention limit described in Section 2.1.
- Using an LM in simple beam search improves pseudo-label quality and the resulting trained model compared with acoustic-model-only greedy decoding.
- Table 4 compares AM greedy, AM+LM simple, and AM+LM stable pseudo-labels using dev clean and other WER averaged over three models.
- Stable beam search further improves pseudo-label quality and the resulting model WER beyond simple beam search.
A.3. Importance of the LM
The experiments vary language-model perplexity to assess its effect on self-training. Lower perplexity consistently corresponds to lower development-set WER, while even a higher-perplexity LM improves effectiveness over acoustic-model-only decoding.
- The experiment trains three models per pseudo-label set and reports average WER without decoding with an LM.
- Lower LM perplexity produces lower development-set WER for self-training.
- Without an LM, AM Greedy yields WERs of 12.27 on dev clean and 33.42 on dev other.
- Using an LM improves self-training effectiveness even when its perplexity is relatively high.