Source-linked AI summary
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, Ronan Collobert
TL;DR
The paper studies whether simpler end-to-end ASR systems can match more complex pipelines and how pseudo-labeling can improve their semi-supervised training. It evaluates multiple architectures and losses on LibriSpeech with additional LibriVox audio, finding state-of-the-art performance and reduced reliance on external language models.
Problem
The paper asks whether end-to-end ASR systems can perform as well as more complex pipelines despite optimization and regularization challenges.
Method
The study trains ResNet, time-depth separable convolutional, and Transformer acoustic models with CTC or Seq2Seq losses, using pseudo-labels from a 53.8K-hour LibriVox corpus alongside LibriSpeech.
Results
Semi-supervised training improves end-to-end models across architectures and losses, reaching state-of-the-art results without decoding or an external language model.
Takeaways & Limitations
With enough pseudo-labeled audio, acoustic models learn language-model-level information and generally benefit less from external language-model decoding.
Takeaways & Limitations
The 10k word-piece vocabulary limits the amount of architectural striding available, motivating acoustic models that output words with arbitrary lexicons.
Abstract
from arXiv · showhide
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.
1 Introduction
The paper examines whether simpler end-to-end ASR systems can approach classical pipelines and studies pseudo-labeling as a semi-supervised strategy for improving them.
- Motivation: End-to-end ASR models are simpler to implement and train than bootstrapped systems, but classical ASR models still dominate common benchmarks.The paper investigates whether end-to-end systems can match more complex training pipelines without an algorithmic contribution.
- Motivation: CTC and Seq2Seq acoustic-model optimization remains difficult, while classical pipelines offer more readily available regularization techniques.
- Semi-supervision: Self- and semi-supervised ASR uses pretrained networks to generate and train on their own labels, yielding improvements in related domains.
- Research setting: Transformer acoustic-model architectures outperform other models by a large margin on LibriSpeech, whereas the gap is much smaller with LibriVox data.
2 Models
The study compares ResNet, TDS convolutional, and Transformer acoustic models that output distributions over 10k word pieces, using CTC or Seq2Seq objectives and external language models for decoding.
- Common design: All acoustic models take 80-channel log-mel filterbanks as input and output probability distributions over a fixed set of 10k word pieces.The fixed word-piece set supports a controlled architectural comparison.
- ResNet: The ResNet encoder contains 42 one-dimensional convolutional layers with skip connections and performs 160 milliseconds of total input pooling.Nearly identical encoders feed CTC and Seq2Seq losses, with architecture changes for Seq2Seq and larger models for unlabeled data.
- TDS convolution: TDS blocks combine a two-dimensional convolution with two fully connected layers, while increased internal channel capacity and three stride-2 subsampling layers expand capacity and context.
- Transformer: Transformer models use convolutional front-ends followed by self-attention, with Seq2Seq variants adding a six-layer Transformer decoder with four attention heads.
- Language models: External language models include n-gram, GCNN, and Transformer variants, with word-piece compatibility maintained between acoustic-model outputs and LM-scored candidates.
3 Dataset and Language Models for Pseudo-Labeling
The pseudo-labeling pipeline constructs a large, speaker-disjoint LibriVox corpus, generates labels with a CTC Transformer and filtered 4-gram LM, and evaluates overlap effects.
- Unlabeled audio: After filtering duplicate-text readings and corrupted audio and removing LibriSpeech speaker overlap, LibriVox contains 53.8K hours of segmented read speech.The corpus begins with 72K hours selected from English book listings and is segmented into chunks no longer than 36 seconds.
- Pseudo-label generation: A LibriSpeech-trained CTC Transformer achieving 6.20% WER on dev-other generates pseudo-labels for all selected LibriVox audio using 4-gram-LM beam search.
- Text-corpus preparation: The LM corpus is filtered to remove text corresponding to unlabeled audio, including exact and low-similarity book-title matches, to avoid embedding ground-truth information.Manual matching follows title normalization and word-level Levenshtein comparisons.
- Text-corpus preparation: The filtered corpus preserves 88% of the original normalized text, while tokenizer differences can change sentence boundaries and abbreviations.
- Language-model training: The resulting unpruned 4-gram LM uses the top 200k words as vocabulary and produces only a 0.05% absolute WER regression versus the full-corpus LM.
4 Decoding
Decoding combines acoustic-model posteriors with an external language model through beam search, optionally followed by neural-LM rescoring of N-best hypotheses.
- Beam search: One-pass beam search selects transcriptions using acoustic-model posteriors and external-LM scores, with optional stronger neural LMs used for rescoring.
- Decoder variants: Lexicon-based decoding serves CTC models with word-level LMs, while lexicon-free decoding serves Seq2Seq models with word-piece LMs.
- Search stabilization: Seq2Seq beam search adds an EOS penalty to prevent early stopping and uses hypothesis merging, score caching, and batched LM forwarding for efficiency.
- Rescoring: N-best hypotheses are rescored with word-level GCNN and Transformer LMs using validation-optimized weights for each LM probability and transcription length.
5 Experiments
The experiments evaluate end-to-end acoustic models across architectures, loss functions, supervised and pseudo-labeled training settings, and language-model decoding. Semi-supervised training improves results substantially, while additional audio reduces reliance on external language models.
- LIBRISPEECH Results: Transformer acoustic models reach 6.98% WER on test-other without decoding and 5.17% with decoding and rescoring, matching traditional bootstrapped systems.These results are reported for supervised LIBRISPEECH training.
- LIBRIVOX Results: Semi-supervised Transformer models reach 2.28% WER on test-clean and 4.88% on test-other without decoding or an LM.The combined training uses 960 hours of labeled LIBRISPEECH audio and pseudo-labeled LIBRIVOX audio.
- LIBRIVOX Results: Decoding and rescoring further improve the semi-supervised results to 2.09% WER on test-clean and 4.11% on test-other.The paper reports these as further improvements on the state of the art.
- Varying the amount of unlabeled audio: Increasing the amount of pseudo-labeled audio strictly improves performance in the reported Transformer Seq2Seq experiments.WERs are measured without decoding after 800k training iterations.
- Generating pseudo-labels with an LM containing overlapping text: Models trained with pseudo-labels generated using an overlapping language-model corpus generalize similarly to models trained with non-overlapping pseudo-labels.The overlapping-corpus condition gives only a small improvement on dev-other.
- Training on pseudo-labels only: Training on LIBRIVOX pseudo-labels alone outperforms LIBRISPEECH-only training, achieving 2.38% versus 2.99% on dev-clean and 5.43% versus 7.31% on dev-other.Combining LIBRISPEECH with LIBRIVOX pseudo-labels further reaches 2.28% and 4.99% on dev-clean and dev-other, respectively.
6 Related Work
Prior end-to-end ASR work spans multiple loss functions and architectures, including biLSTM, TDS convolutional, and Transformer-based models. Semi-supervised pseudo-labeling has also been explored across resource settings.
- End-to-end ASR research has used CTC, ASG, LF-MMI, sequence-to-sequence, transduction, and differentiable decoding objectives.
- Listen Attend and Spell models use biLSTM architectures and achieved state-of-the-art results with improved regularization from data augmentation.
- Time-depth separable convolutions have produced strong sequence-to-sequence speech-recognition results.
- Transformer-based acoustic models have shown strong results in sequence-to-sequence settings, including without external language models.
- Semi-supervised pseudo-label-style self-training has been studied in end-to-end ASR for both low-resource and large-scale setups.
7 Discussion
The paper reports state-of-the-art end-to-end results on LibriSpeech and argues that a simple semi-supervised pipeline can support strong performance without external language models. It also identifies tokenization and shrinking language-model gains as directions for future work.
- The study presents state-of-the-art LibriSpeech results using end-to-end methods.
- 10k word-piece tokens limit the amount of architectural striding, while word-output acoustic models could support arbitrary lexicons.
- With LibriVox data, relative WER gains from language models shrink from approximately 20% to approximately 10% for GCNN decoding.
- The authors identify differentiable decoding as a possible route toward joint single-stage acoustic-model and language-model training.
- Semi-supervised training without decoding or an external language model is presented as viable for end-to-end state-of-the-art results.
A Experiment Details
The experiments report comprehensive LibriSpeech and LibriVox WER evaluations and tune beam-search decoding through random search, with separate parameter ranges for language models and rescoring.
- WER results cover LibriSpeech and LibriVox acoustic models under greedy and beam-search decoding with different language models and rescoring.
- Beam-search hyperparameters are tuned by random search using dev-clean and dev-other as validation sets.
- 64 to 128 runs are used per random search, with hyperparameters sampled uniformly from specified ranges.
- Tables 6 and 7 document the searched hyperparameter ranges and optimal language-model weight ranges for n-gram and GCNN decoding.
- Rescoring dumps beam-search hypotheses with enlarged beam settings before grid-searching rescoring parameters for CTC models.
B Generating Shuffled Audio
The shuffled-audio experiments assess potential systematic error by repeating evaluations, while the TTS procedure creates synthetic speech from randomly reordered dev-other sentences. This procedure can amplify TTS and rare-word errors.
- The shuffled-utterance experiments repeat dev-other evaluations five times and report means with standard-deviation error bars.
- Synthetic shuffled audio is generated by randomly reordering each dev-other sentence and passing the result through a TTS model.
- The TTS ablation can introduce and amplify intrinsic synthesis errors, particularly for rare words in dev-other.
- The TTS-generated speech differs from human speech, adding another boundary to interpreting this ablation.
B.2 Segmentation
The segmentation method force-aligns dev-other, splits audio only at silences longer than 130 milliseconds, and then randomly reassembles the chunks. Because boundaries are not always handled, resulting segments may contain incomplete words and LM-unfriendly context.
- Audio is force-aligned with a letter-based stridetwo ASG model to obtain each word’s beginning timestamp and duration.
- Segments are split only at silences longer than 130 milliseconds, with each split placed in the middle of the silence.
- The resulting audio chunks are randomly shuffled and reassembled into new utterances.
- Force alignment and segmentation do not handle every word boundary, so resulting audio may contain incomplete words and LM-unfriendly context.
- Figure 4 shows the distribution of word n-grams across 16,362 segments from 1,969 filtered dev-other samples.
B.3 Recording
The recording experiment compares original and shuffled dev-other transcriptions using language-model perplexity and recognition performance. Shuffling creates large perplexity gaps, removes important word context, and makes the WER gap a proxy for acoustic models’ implicit language modeling.
- The unshuffled recorded audio has the lowest WER among the three recording methods.
- The authors recorded 184 randomly selected dev-other sentences and one set of shuffled utterances, with additional unshuffled and shuffled recordings planned.
- Large perplexity gaps separate original and shuffled transcription sets across all evaluated language-model settings.
- Shuffling removes important word context and breaks alignment between the audio-word distribution and the language model.
- The WER gap between original and shuffled sets serves as a proxy for how much language modeling an acoustic model may implicitly perform.