Source-linked AI summary
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, Yonghui Wu
TL;DR
The paper addresses how unlabeled audio can improve supervised automatic speech recognition. It combines iterative self-training with pre-training and scaled-up Conformer models, achieving state-of-the-art LibriSpeech performance with 1.4%/2.6% WER on test/test-other.
Problem
Semi-supervised learning seeks to use large unlabeled datasets to improve performance on supervised automatic speech recognition tasks.
Method
The approach combines iterative self-training and pre-training, using unlabeled audio for both pre-training and pseudo-label generation, with scaled-up Conformer models.
Results
1.4%/2.6% WER on the LibriSpeech test/test-other sets establishes state-of-the-art performance.
Takeaways & Limitations
Combining recent advances in architecture, augmentation, and semi-supervised learning pushes the state of the art on LibriSpeech speech recognition.
Takeaways & Limitations
The study does not use quantization or a diversity loss and leaves settings that could improve Conformer performance for future investigation.
Abstract
from arXiv · showhide
We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.
1 Introduction
The paper combines pre-training and iterative self-training to exploit Libri-Light’s unlabeled audio for supervised ASR on LibriSpeech. Using giant Conformers, wav2vec 2.0 pre-training, noisy student training, and SpecAugment yields state-of-the-art LibriSpeech performance.
- Semi-supervised learning uses large unlabeled datasets to improve supervised automatic speech recognition.
- The approach pre-trains a series of models, then uses them to initialize iterative self-training on the same unlabeled dataset.The unlabeled data serves both as the pre-training set and as the source of pseudo-labels for student training.
- The pipeline combines Conformer architectures, wav2vec 2.0 pre-training, noisy student training, and adaptive SpecAugment.
- 1.1 Related Work: The work combines self-training, which generates labels for unlabeled data, with consistency-oriented representation learning through pre-training.The authors use wav2vec 2.0 pre-training within a noisy student self-training loop.
2 Methods
The method uses Conformer transducer models with wav2vec 2.0-style pre-training and noisy student training. It scales the architecture to 600M- and 1B-parameter models while reusing unlabeled Libri-Light audio for pre-training and pseudo-label generation.
- 2.1 Model Architecture: Conformer: The ASR network is a sequence transducer with a Conformer encoder and an LSTM decoder.The encoder contains conformer blocks built from self-attention, depth-wise convolution, and feed-forward layers.
- 2.1 Model Architecture: Conformer: Conformer XL and XXL contain 600M and 1B parameters, respectively.Both models use two-layer LSTM decoders, while the original Conformer L uses a single-layer decoder.
- 2.1 Model Architecture: Conformer: Conformer XXL+ adds a Conformer block and stacking layer to Conformer XXL, reducing time length by half and doubling channel size.The resulting model has approximately 50M more parameters than Conformer XXL.
- 2.2 Wav2vec 2.0 Pre-training: Wav2vec 2.0-style pre-training masks encoded features, produces context and target context vectors, and optimizes their contrastive loss.The paper uses log-mel spectrograms and replaces wav2vec 2.0’s quantization layer with a linear layer.
- 2.2 Wav2vec 2.0 Pre-training: During fine-tuning, a projection block converts context vectors into features passed to the transducer.Conformer XXL and XXL+ share the same pre-trained feature encoder and context network but use different projection blocks.
- 2.3 Noisy Student Training: Noisy student training uses a language-model-fused teacher to generate transcripts, filters and balances them, and trains the next model with adaptive SpecAugment.The procedure mixes teacher-labeled Libri-Light data with labeled LibriSpeech data before fine-tuning a new pre-trained model.
3 Experiments
The experiments use LibriSpeech’s labeled data with Libri-Light’s unlabeled audio to train successive pre-trained Conformer models through noisy student training. Across generations, the combined pipeline is evaluated with WERs and language-model fusion, reaching 7–15% relative improvement over the pre-trained baseline.
- The task uses 960 hours of transcribed LibriSpeech audio and unlabeled Libri-Light audio for the LibriSpeech ASR experiments.
- The pipeline trains four generations, using pre-trained Conformer XL at generations 0–1, Conformer XXL at generation 2, and Conformer XXL or XXL+ at generation 3.
- Student models use adaptive SpecAugment, while each training batch contains supervised and teacher-labeled data at a fixed 1:9 ratio.
- The experiments compare baseline, NST-only, pre-training-only, and combined semi-supervised-learning models using LibriSpeech WERs.
- Figure 3 tracks performance across generations with and without LM fusion, using Conformer XXL+ as the final generation model.
- 7–15% relative WER improvement is obtained by the generation-3 Conformer XXL over the pre-trained baseline across LibriSpeech development and test sets.
4 Discussion
The discussion examines how pre-training, input processing, and noisy-student data policies affect performance. It finds that pre-training makes model scaling effective, while aggressive segmentation and some transcript-processing choices can hurt or fail to improve results.
- 4.1 Model Size and Pre-training: Scaling from 100M to 1B parameters alone does not improve performance on the supervised dataset.
- 4.1 Model Size and Pre-training: Pre-training produces consistent gains as model size increases up to 1B parameters, transferring model-size growth into performance.
- 4.2 Ablations for Pre-training: The pre-training procedure uses log-mel spectrograms rather than waveforms and omits quantization and diversity loss for simplicity.
- 4.2 Ablations for Pre-training: Longer pre-training segments require more aggressive convolutional subsampling to reduce input length.
- 4.2 Ablations for Pre-training: VAD tends to degrade performance because segmented inputs may lack enough context to benefit from pre-training.
- 4.3 Ablations for Fine-tuning: A 1:9 supervised-to-teacher-labeled mix and non-batch-wise mixing are comparable, while a 2:8 mix performs worst.
- 4.3 Ablations for Fine-tuning: Neither transcript balancing nor LM-based filtering improves generation-1 development performance for this model series.
5 Conclusion
The work combines recent advances in architecture, augmentation, and semi-supervised learning to push state-of-the-art performance on LibriSpeech.
- The authors combine advances in architecture, augmentation, and semi-supervised learning to push state-of-the-art LibriSpeech speech recognition performance.