Source-linked AI summary
Improved Noisy Student Training for Automatic Speech Recognition
Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, Quoc V. Le
TL;DR
The paper addresses semi-supervised ASR by adapting noisy student training, an iterative teacher-student method that uses augmentation and unlabeled data. It introduces speech-specific augmentation, transcript fusion and filtering, balancing, and gradational strategies, achieving improved LibriSpeech WERs on both low- and high-supervision tasks.
Problem
The paper seeks to improve semi-supervised learning for ASR by adapting noisy student training from image classification.
Method
The method combines SpecAugment, language-model fusion, normalized transcript filtering, sub-modular balancing, and gradational filtering and augmentation in iterative self-training.
Results
4.2%/8.6% and 1.7%/3.4% test-clean/test-other WERs are achieved on LibriSpeech 100-860 and LibriSpeech-LibriLight, respectively.
Takeaways & Limitations
The gradational methods benefit the low-supervised-performance task, while showing little effect on the high-supervised-performance task.
Abstract
from arXiv · showhide
Recently, a semi-supervised learning method known as "noisy student training" has been shown to improve image classification performance of deep networks significantly. Noisy student training is an iterative self-training method that leverages augmentation to improve network performance. In this work, we adapt and improve noisy student training for automatic speech recognition, employing (adaptive) SpecAugment as the augmentation method. We find effective methods to filter, balance and augment the data generated in between self-training iterations. By doing so, we are able to obtain word error rates (WERs) 4.2%/8.6% on the clean/noisy LibriSpeech test sets by only using the clean 100h subset of LibriSpeech as the supervised set and the rest (860h) as the unlabeled set. Furthermore, we are able to achieve WERs 1.7%/3.4% on the clean/noisy LibriSpeech test sets by using the unlab-60k subset of LibriLight as the unlabeled set for LibriSpeech 960h. We are thus able to improve upon the previous state-of-the-art clean/noisy test WERs achieved on LibriSpeech 100h (4.74%/12.20%) and LibriSpeech (1.9%/4.1%).
1. Introduction
The paper adapts noisy student training to ASR with SpecAugment, language-model fusion, normalized filtering, balancing, and progressively adjusted filtering and augmentation. These methods achieve improved WERs on both low- and high-supervision LibriSpeech tasks.
- 1. Introduction: Noisy student training iteratively uses teacher-generated labels on unlabeled data while students learn from heavily augmented inputs.Teacher outputs are filtered and balanced to preserve label integrity and distributional alignment.
- 1. Introduction: SpecAugment, language-model fusion, normalized transcript filtering, and sub-modular sampling adapt noisy student training to speech recognition.The normalized filtering score depends on fusion score and transcript token count.
- 1. Introduction: Gradational filtering relaxes the acceptance criterion as model performance improves, growing the semi-supervised dataset across iterations.This strategy is described as effective when supervised performance is low.
- 1. Introduction: Gradational augmentation increases augmentation strength with noisy student iterations.The method adjusts the student’s augmentation as self-training progresses.
- 1. Introduction: 4.2%/8.6% test-clean/test-other WERs are achieved on LibriSpeech 100-860, versus 5.5%/16.9% baseline WERs.The task uses 100 hours labeled and 860 hours unlabeled LibriSpeech data.
- 1. Introduction: 1.7%/3.4% test-clean/test-other WERs are achieved on LibriSpeech-LibriLight, versus 1.9%/4.1% baseline WERs.The task uses LibriSpeech 960h as labeled data and LibriLight unlab-60k as unlabeled data.
- 1. Introduction: The work extends prior ASR self-training and consistency-based research with new augmentation, normalized filtering, and gradational filtering contributions.The related-work passage identifies three main contributions.
2. Noisy Student Training for ASR
The proposed NST pipeline repeatedly generates teacher transcripts, filters and balances them, mixes them with labeled data, and retrains an augmented student. Its filtering score normalizes shallow-fusion scores using transcript length and dev-set statistics.
- 2. Noisy Student Training for ASR: The algorithm assumes labeled set S, unlabeled set U, and a fixed language model, then generates successive ASR models.Each cycle produces a new generation for the next self-training iteration.
- 2. Noisy Student Training for ASR: Each iteration trains an initial model, fuses it with the language model, generates transcripts, filters and balances them, then retrains a new model.The retrained model becomes the teacher for the next cycle.
- 2. Noisy Student Training for ASR: The normalized filtering score uses shallow-fusion score S and transcript token length ℓ.The score is designed for transcripts produced by the fused teacher model.
- 2. Noisy Student Training for ASR: Parameters µ, β, and σ are fit on dev-set transcripts and refit for each teacher-model generation.µ and β come from linear regression, while σ is computed from normalized score residuals.
- 2. Noisy Student Training for ASR: Gradational filtering lowers the cutoff across self-training cycles, retaining more generated utterance-transcript pairs as iterations proceed.The cutoff keeps pairs whose normalized scores exceed the current threshold.
- 2. Noisy Student Training for ASR: Sub-modular sampling balances filtered data by greedily reducing KL divergence between sampled and target token distributions.The procedure selects batches using a cost-benefit score.
- 2. Noisy Student Training for ASR: The balanced semi-supervised data is combined with labeled data using either fixed batch ratios or uniform sampling across datasets.These are the two specified mixing strategies.
3. Experiments
The experiments evaluate noisy student training on two ASR settings: LibriSpeech 100-860 and LibriSpeech-LibriLight, using progressively trained models and task-specific configurations.
- LibriSpeech 100-860: LibriSpeech 100-860 uses clean 100h as supervised data and the remaining 860h, including clean and noisy audio, as unlabeled data.The task uses 360h of clean and 500h of noisy unlabeled audio.
- LibriSpeech 100-860: Six generations, numbered 0 to 5, are trained for LibriSpeech 100-860, with the supervised baseline counted as generation 0.Each generation uses Adam optimization on 32 Google Cloud TPU chips for 10 days.
- LibriSpeech 100-860: The generation-4 model is the best trained model, and its performance is compared with the baseline and language-model-fused variants in Table 1.The fused models’ performance is also plotted across generations in Figure 1.
- LibriSpeech-LibriLight: LibriSpeech-LibriLight uses all LibriSpeech training data as supervised data and the unlab-60k LibriLight subset as unlabeled data.The LibriLight audio is derived from audiobooks and is not accompanied by transcripts for this task.
- LibriSpeech-LibriLight: The LibriSpeech-LibriLight models use ContextNet architectures across five generations, with model width increasing from generation 1 through generation 4.Each generation is trained on 128 Google Cloud TPU chips for 1–3 days.
- LibriSpeech-LibriLight: For LibriSpeech-LibriLight, adaptive time masking is used without time warping or filtering, while the semi-supervised data fraction increases across generations.The supervised-to-semi-supervised ratio changes from 4:6 to 2:8 by generation 4.
4. Discussion
For LibriSpeech 100-860, gradually expanding the filtered dataset across slower noisy-student generations improves performance, while balancing can reduce dataset size and performance. For LibriSpeech-LibriLight, balancing is retained, but gradational filtration and augmentation show little benefit because the baseline is already strong.
- Filtering: The filtering score reasonably ranks generated transcript quality, except for a generation-4 blank transcript on a long utterance.The generation-4 anomaly makes cumulative WER large at high scores.
- Filtering: Filtered dev-set WERs remain comparable across generations while the filtered dataset grows, which is the intended effect of gradational filtration.The filtering criterion is gradually relaxed as model performance improves.
- Generation schedule: 6-generation pipelines outperform 3-generation pipelines at comparable semi-supervised dataset sizes.The authors report that slower evolution improves performance within the explored range.
- Augmentation: Stronger augmentation is favored as the dataset grows in the LibriSpeech 100-860 proxy task.Time-mask sizes for generations 2 and 4 were selected using proxy-task WERs, prioritizing clean-set performance without LM fusion.
- Mixing and balancing: Balancing halves the semi-supervised dataset at each generation in LibriSpeech 100-860 and diminishes trained-network performance.The control experiments tested balancing with 1:1 batch-wise mixing.
- LibriSpeech-LibriLight: For LibriSpeech-LibriLight, balancing is used without filtering because its token-distribution benefit outweighs the dataset-size reduction.This trade-off is attributed to the large size of the LibriLight dataset.
- LibriSpeech-LibriLight: Gradational filtration and augmentation are less effective for LibriSpeech-LibriLight than for LibriSpeech 100-860.Both filtered and unfiltered approaches reach dev-clean/dev-other WERs of 1.7%/3.7% after LM fusion, consistent with the already strong baseline.
5. Conclusion
The paper adapts and improves noisy student training for ASR with augmentation, language-model fusion, normalized transcript filtering, sub-modular sampling, and gradational self-training. These methods benefit the low-supervised LibriSpeech 100-860 task, while gradational methods have little effect for the high-supervised LibriSpeech-LibriLight task.
- Contributions: The ASR pipeline combines SpecAugment, language-model fusion, sub-modular sampling, normalized filtering, and gradational self-training.The components are mixed and matched across the two studied tasks.
- Findings: Gradational methods benefit LibriSpeech 100-860 but show little effect on LibriSpeech-LibriLight.The paper attributes this difference to the high supervised performance of the LibriSpeech-LibriLight setting.