Source-linked AI summary
Iterative Pseudo-Labeling for Speech Recognition
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, Ronan Collobert
TL;DR
Semi-supervised ASR needs effective ways to exploit large amounts of unlabeled speech and additional text. The paper proposes Iterative Pseudo-Labeling, which repeatedly fine-tunes an acoustic model using refined labels, language-model decoding, augmentation, and sampled unlabeled data. IPL achieves state-of-the-art results on LibriSpeech test sets in standard and low-resource settings while improving training efficiency.
Problem
Semi-supervised ASR seeks to better utilize unlabeled data, while pseudo-labeling requires an efficient way to perform multiple refinement iterations as the acoustic model evolves.
Method
IPL repeatedly fine-tunes an existing acoustic model using language-model decoding, data augmentation, and subsets of unlabeled data rather than retraining from scratch.
Results
IPL achieves state-of-the-art results on LibriSpeech test sets in both standard and low-resource settings and outperforms a three-round pseudo-labeling baseline.
Takeaways & Limitations
IPL can efficiently accumulate gains across many pseudo-labeling updates and leverage additional unpaired text through language models.
Takeaways & Limitations
A mismatch between decoding parameters optimized on dev-other and those optimal for unlabeled audio can limit further acoustic-model improvement.
Abstract
from arXiv · showhide
Pseudo-labeling has recently shown promise in end-to-end automatic speech recognition (ASR). We study Iterative Pseudo-Labeling (IPL), a semi-supervised algorithm which efficiently performs multiple iterations of pseudo-labeling on unlabeled data as the acoustic model evolves. In particular, IPL fine-tunes an existing model at each iteration using both labeled data and a subset of unlabeled data. We study the main components of IPL: decoding with a language model and data augmentation. We then demonstrate the effectiveness of IPL by achieving state-of-the-art word-error rate on the Librispeech test sets in both standard and low-resource setting. We also study the effect of language models trained on different corpora to show IPL can effectively utilize additional text. Finally, we release a new large in-domain text corpus which does not overlap with the Librispeech training transcriptions to foster research in low-resource, semi-supervised ASR
1 Introduction
The paper studies how to make semi-supervised ASR benefit more effectively from unlabeled speech as acoustic-model architectures improve. It proposes iterative pseudo-labeling, which repeatedly refines labels while fine-tuning the existing model on subsets of unlabeled data.
- Recent ASR progress has shifted research toward using self- and semi-supervised techniques to exploit unlabeled data.Transformer-family acoustic models have delivered state-of-the-art benchmark results, motivating further work on unlabeled-data utilization.
- Iterative pseudo-labeling extends pseudo-labeling by conducting more labeling rounds as the model trains, continuously refining generated transcripts.
- IPL fine-tunes the existing model and labels only a subset of unlabeled data at each iteration instead of retraining from scratch on the entire collection.These choices address the computational cost of repeated relabeling and scratch training.
- Figure 1 compares fine-tuning with training a new model from scratch using pseudo-labels from train-clean-360 and true labels from train-clean-100.Both strategies are evaluated by WER on dev-other.
2 Related Work
The work builds on semi-supervised ASR and recent end-to-end pseudo-labeling methods. Prior approaches include representation learning, pseudo-labeling, and related techniques for low-resource recognition.
- Semi-supervised learning has been extensively studied in ASR, and this work primarily builds on recent end-to-end systems.
- Earlier pseudo-labeling work showed effectiveness with only 100h of labeled audio, using a sequence-to-sequence loss and additional pseudo-label filtering.
- Connectionist Temporal Classification loss was used in related work to generate pseudo-labels and mitigate instability in sequence-to-sequence decoding.
3 Method
IPL repeatedly generates pseudo-labels for sampled unlabeled data, then fine-tunes the acoustic model on labeled and pseudo-labeled examples with language-model decoding and data augmentation. Experiments show these components sustain WER improvements, while moderate unlabeled-data subsampling can preserve performance.
- Iterative pseudo-labeling: IPL initializes an acoustic model on labeled data, repeatedly samples unlabeled data, decodes it with a language model, and fine-tunes on labeled plus pseudo-labeled data.The iterations continue until convergence or a maximum iteration count.
- Iterative pseudo-labeling: The combined loss is the labeled-data loss plus a weighted unlabeled-data loss.The weighting factor is denoted λ.
- Language-model decoding: Language-model decoding replaces greedy pseudo-label generation with transcriptions incorporating external language-model knowledge, allowing further model updates.The method uses beam-search-style decoding and may incorporate lattice or beam rescoring.
- Data augmentation: Data augmentation changes the optimized likelihood because augmented inputs can make previously optimized model weights non-optimal; SpecAugment is used experimentally.This provides another mechanism for continued parameter updates during IPL.
- Empirical analysis: Using both augmentation and language-model decoding causes a dramatic initial WER drop and continued improvement as IPL progresses, whereas removing either degrades convergence.Removing both components makes unlabeled data provide no benefit in the reported experiment.
- Empirical analysis: 20% to 40% of the unlabeled data per iteration reaches the same WER as using 100%, while using only 10% significantly hurts convergence.The study frames this as a trade-off between estimating the unlabeled-data distribution and performing more pseudo-labeling rounds.
4 Experiments
Experiments evaluate IPL across labeled-data regimes, decoding strategies, language-model corpora, and iterative training schedules. IPL achieves state-of-the-art semi-supervised WER, accumulates gains efficiently through repeated updates, and benefits from additional non-overlapping in-domain text, while decoding-parameter mismatch limits late-stage improvement.
- Experimental setup: The experiments use LIBRISPEECH and LIBRIVOX audio, with labeled settings of 960 hours, 100 hours, or 10 hours.LIBRIVOX contributes 54K hours of audio, while LIBRISPEECH contributes 960 hours with paired transcriptions.
- Results: 26.02%, 19.92%, 8.95%, 7.11%, and 4.01% test-other WERs establish state-of-the-art results across three semi-supervised settings.The figures correspond to 10-hour or 100-hour labeled setups with LIBRISPEECH or LIBRIVOX unlabeled data, plus the full-LIBRISPEECH labeled setting.
- Effectiveness: IPL outperforms a three-round train-from-scratch pseudo-labeling baseline, accumulating gains through up to 80 rounds of pseudo-label updates.The baseline retrains from scratch after three rounds, whereas IPL fine-tunes the evolving model with repeated pseudo-labeling updates.
- Efficiency: 10.69%, 8.50%, and 4.14% WER are achieved by IPL within the same 4-, 11-, and 17-day budgets used by three-round training.IPL reaches the same post-round-3 WER in 0.7, 3.3, and 8 days, using fine-tuning and unlabeled-data downsampling; a 20% rate gives 5× labeling speedup.
- Language-model study: Better language models improve WER and let IPL use additional in-domain text, but decoding-parameter mismatch can prevent dev-set gains from transferring to unlabeled audio.Non-overlapping GB \ LV \ LS text can achieve similar or better WER than LS \ LV, while parameters optimized on dev-other may be unsuitable for unlabeled-audio decoding.
5 Conclusion
Iterative pseudo-labeling achieves superior results in standard and low-resource settings while offering an efficient alternative to conventional pseudo-labeling. Combined with beam-search language-model decoding, data augmentation, and dataset subsampling, it reaches state-of-the-art results on LibriSpeech test sets.
- IPL gives superior results in both standard and low-resource settings.
- IPL provides an efficient training algorithm compared with conventional pseudo-labeling approaches.
- Beam-search decoding with a language model, data augmentation, and dataset subsampling contribute to IPL’s effectiveness and efficiency.
- IPL achieves state-of-the-art results on LibriSpeech test sets with the Transformer acoustic model.
A.1 Typical IPL curves
IPL training curves across full, 100-hour, and 10-hour labeled-data settings show sharp WER reductions after unlabeled data is introduced, followed by continued marginal gains.
- WER decreases dramatically during the first IPL rounds as newly generated pseudo-labels and data are consumed.
- IPL continues accumulating marginal gains through the end of training across three labeled-data scenarios.The scenarios use full LibriSpeech, train-clean-100, and LibriLight-10 labeled data.
- IPL can bootstrap from an immature acoustic model with approximately 80% WER.
- The language model and lexicon limit the word search space during pseudo-label generation.
A.2 IPL v.s. Training From Scratch
Compared with repeatedly training acoustic models from scratch as pseudo-labels evolve, IPL avoids diminishing gains and achieves better WER within the same acoustic-model training time.
- Training from scratch after each pseudo-labeling round produces diminishing gains.
- IPL can always outperform scratch training and reach better WER given the same acoustic-model convergence time.This comparison ignores the time required to generate labels.
- Training from scratch and labeling the entire 54K-hour dataset are both time-consuming.
A.3 IPL v.s. Retraining
Retraining a new acoustic model from scratch on pseudo-labels generated by a converged IPL model does not outperform IPL, with both approaches reaching similar results even with beam-search decoding.
- A scratch-trained model using pseudo-labels from a fully converged IPL model does not outperform IPL.
- Beam-search decoding still yields similar results for IPL and retraining models.
- 8.83 and 8.87 WER are reported for IPL and retraining on train-clean-100, respectively.
- 25.69 and 25.62 WER are reported for IPL and retraining on LibriLight-10, respectively.
B.1 LM weights for different LMs
The study examines how language-model weighting affects IPL decoding and rescoring. It finds that reasonable beam-search LM weights preserve candidates for later neural-LM rescoring.
- A 4-gram LM generates beam candidates, which an NN-LM then rescores.
- For beam-search decoding, LM weights between 0 and 2 preserve good candidates for later rescoring, leaving final WER insensitive to the exact weight.
- WER is evaluated across different LM weights for beam-search decoding and rescoring.
B.2 LM weights on different dataset
The optimal LM weight is aligned between development and training data for both immature and converged acoustic models. This supports using the development set to tune decoder parameters.
- Parameter sweeps compare decoding and rescoring settings on development and training data.
- The optimal LM weight is aligned between development and training for both immature and converged acoustic models.
- The development set can serve as a proxy for decoder-parameter tuning.
C Comparison With The Latest Methods
The paper compares IPL with recent semi-supervised methods on LibriSpeech and LibriVox. The compared methods achieve better results than IPL using stronger acoustic modeling, pseudo-label filtering, or unsupervised pre-training, while the authors identify combining these advantages as future work.
- Two compared methods achieve better results than IPL using multi-round pseudo-labeling or unsupervised pre-training.
- The compared multi-round method adds a LAS acoustic model and pseudo-label filtering, while the pre-training method learns features from large-scale clean speech data.
- Table 5 compares WER with recent semi-supervised methods on LibriSpeech and LibriVox data.
- Combining better acoustic models and pre-trained features with IPL is identified as future work.