Source-linked AI summary
A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition
Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, Karen Livescu
TL;DR
Encoder-decoder ASR models cannot directly exploit the much larger text-only corpora available to conventional systems, and prior integration methods lacked a thorough comparison. This paper compares fusion approaches and proposed alternatives across Switchboard and large-scale Google data. Unpaired text generally improves results; shallow fusion is best for first-pass decoding, while cold fusion has lower oracle error and benefits most from second-pass rescoring on Google voice search.
Problem
Encoder-decoder ASR training requires paired speech and text, leaving the use of much larger unpaired text resources and the relative merits of integration methods insufficiently established.
Method
The paper compares shallow, deep, and cold fusion plus proposed approaches across Switchboard and Google voice search and dictation data.
Results
Shallow fusion performs best for first-pass decoding across data sets, while cold fusion yields lower oracle error and outperforms other approaches after second-pass rescoring on Google voice search.
Takeaways & Limitations
Using unpaired text benefits encoder-decoder ASR across the evaluated data sets, with the strongest approach depending on decoding stage and evaluation criterion.
Takeaways & Limitations
Cold fusion is computationally costlier than shallow and deep fusion when language-model and ASR models change frequently.
Abstract
from arXiv · showhide
Attention-based recurrent neural encoder-decoder models present an elegant solution to the automatic speech recognition problem. This approach folds the acoustic model, pronunciation model, and language model into a single network and requires only a parallel corpus of speech and text for training. However, unlike in conventional approaches that combine separate acoustic and language models, it is not clear how to use additional (unpaired) text. While there has been previous work on methods addressing this problem, a thorough comparison among methods is still lacking. In this paper, we compare a suite of past methods and some of our own proposed methods for using unpaired text data to improve encoder-decoder models. For evaluation, we use the medium-sized Switchboard data set and the large-scale Google voice search and dictation data sets. Our results confirm the benefits of using unpaired text across a range of methods and data sets. Surprisingly, for first-pass decoding, the rather simple approach of shallow fusion performs best across data sets. However, for Google data sets we find that cold fusion has a lower oracle error rate and outperforms other approaches after second-pass rescoring on the Google voice search data set.
1. INTRODUCTION
Encoder-decoder ASR models integrate acoustic, pronunciation, and language modeling but are trained only on paired speech-text data, leaving the use of unpaired text unresolved. This paper compares fusion methods across Switchboard and large-scale Google data, finding broad gains from unpaired text and distinct strengths for shallow and cold fusion.
- 1. INTRODUCTION: Encoder-decoder ASR models fold acoustic, pronunciation, and language models into one end-to-end network trained on paired speech and text.Their decoder is exposed only to audio transcripts, unlike conventional ASR systems that can train a separate language model on much larger text collections.
- 1. INTRODUCTION: Unpaired text is difficult to use because encoder-decoder training requires both input and output sequences.Conventional ASR can exploit text-only data through a separately trained language model.
- 1. INTRODUCTION: The paper compares shallow, deep, and cold fusion on Switchboard, Google voice search, and Google dictation data, while also testing proposed methods.The comparison addresses both method differences and scaling with data size.
- 1. INTRODUCTION: Almost all integration approaches improve over a baseline encoder-decoder model across the evaluated data sets, confirming benefits from unpaired text.The paper evaluates both prior and proposed approaches.
- 1. INTRODUCTION: Shallow fusion performs best for first-pass decoding across all data sets, while cold fusion has lower oracle error and wins after second-pass rescoring on Google voice search.Deep fusion is comparable to cold fusion on Switchboard but shows no or negligible gains on large-scale Google data.
2. RELATED WORK
Prior work on using unpaired text for encoder-decoder models falls into two broad research directions.
- 2. RELATED WORK: Previous approaches to unpaired text for encoder-decoder models can be categorized along two major themes.
Using an external language model
One research direction trains an external language model on unpaired text and integrates it with the encoder-decoder model. This paper focuses on shallow, deep, and cold fusion within that direction.
- Using an external language model: External-LM approaches train a language model on unpaired text and integrate it into the encoder-decoder model.
- Using an external language model: Shallow fusion combines model scores, whereas deep and cold fusion integrate the external language model more directly into the encoder-decoder model.
Generating paired data from unpaired text
A second research direction uses unpaired text to synthetically generate matching input sequences, thereby expanding the paired training data.
- Generating paired data from unpaired text: Synthetic-data methods generate matching input sequences from unpaired text to expand the paired data set.In machine translation, this process is called backtranslation; the analogous ASR approach uses text-to-speech synthesis.
3. MODEL
The model section reviews LAS and compares several ways to incorporate an external language model or unpaired text into encoder-decoder speech recognition.
- LAS Model: LAS jointly trains an encoder, decoder, and attention network to predict wordpiece sequences from acoustic feature frames.The encoder produces high-level speech features, while attention supplies context to the decoder at each step.
- Shallow Fusion: Shallow fusion combines separately trained ASR and language-model scores only during inference.Its criterion extends baseline beam-search decoding with an external LM score and may include penalty terms such as coverage.
- Deep Fusion: Deep fusion combines pretrained ASR-decoder and language-model hidden states during late integration while keeping most model parameters fixed.Only fusion parameters are learned, reducing backpropagation cost and enabling faster convergence than baseline training.
- Cold Fusion: Cold fusion integrates a pretrained external LM from the start of ASR training and uses fine-grained gating based on both model states.Unlike deep fusion’s coarse gating, cold fusion uses a vector gate; these experiments use the LM hidden state rather than logits.
- Other LM Integration Approaches: Additional approaches place a pretrained LM below the decoder or train the shared decoder jointly for ASR and language modeling.The multitask approach uses no external LM: ASR updates the full model, while LM updates leave the encoder and attention components unaffected.
4.1. Switchboard
The Switchboard experiments use a medium-scale conversational speech corpus, carefully prepared external text, and recurrent encoder-decoder models with external language-model integration.
- Data: Switchboard contains roughly 300 hours of conversational telephone speech, with about 192K utterances retained after preprocessing.The first 4K training utterances are reserved for validation, and duplicate short utterances are removed beyond a threshold of 300.
- Data: The external language model is trained on approximately 2 million Switchboard and Fisher utterances after domain-mismatch filtering.Filtering removes noise and hesitation markers and excludes utterances unsupported by the Switchboard wordpiece model.
- Model details: The encoder is a 4-layer pyramidal bidirectional LSTM that reduces time resolution by 8-fold.Each direction uses 256 hidden units, with max-pooling used for the reductions below the topmost layer.
- Model details: The baseline decoder is a single-layer unidirectional LSTM with 256 hidden units and a 1K wordpiece output vocabulary.The vocabulary includes all characters for open-vocabulary coverage and uses jointly learned 256-dimensional embeddings.
- Inference and training: Inference uses beam search with beam size 10 and a development-set-tuned wordpiece insertion reward.The reward offsets insertion errors, allowing larger beams to outperform smaller beams across models.
- Language model: The external language model is a single-layer 512-unit LSTM RNN with a 256-unit projection and approximately 15 development-set perplexity.It is trained for 20 epochs with early stopping and shares the LAS output vocabulary.
- Inference and training: All models use Adam and development-set WER early stopping, with separate learning-rate schedules for early and late language-model integration.Baseline and early-integration models train for 12 epochs, whereas late-integration models train for 8 epochs.
4.2. Google Voice Search and Dictation
The Google experiments evaluate models on large-scale noisy voice-search and dictation data, using large recurrent architectures and substantial distributed training resources.
- Data: Google training uses approximately 22 million anonymized, human-transcribed utterances representing live Voice Search and dictation traffic.Clean utterances are artificially corrupted with noise and reverberation spanning 0dB to 30dB SNR, averaging 12dB.
- Language model: The external language model combines untranscribed voice queries, typed Google Search queries, and transcribed training utterances in a 1:1:1 mixture.The component sources are up- and downsampled to equalize their contribution.
- Model details: The Google LAS model uses a 5-layer unidirectional LSTM encoder, four-headed additive attention, and a 2-layer unidirectional LSTM decoder.The encoder layers have 1400 hidden units each, decoder layers have 1024, and the output vocabulary contains 16384 wordpieces.
- Inference: Inference uses beam search with beam size 8, while shallow-fusion results tune language-model weight and coverage penalty on a development set.The development set contains about 10K Voice Search utterances.
- Training: LAS models are trained in cross-entropy and minimum-word-error-rate stages using 128 synchronous TPU replicas followed by 16 synchronous GPU replicas.The authors report that the effective batch size of 4096 was critical for observing cold-fusion improvement.
- Training: All models use Adam and TensorFlow, with the external language model also trained on a 4x4 TPU topology.Deep fusion effectively requires four training phases across LAS and deep-fusion cross-entropy and MWER training.
5. RESULTS
Fusion approaches improve encoder-decoder ASR over baseline across Switchboard and Google data, with shallow fusion strongest for first-pass decoding. Cold fusion offers distinct advantages for second-pass rescoring, while deep fusion scales poorly on large Google data.
- Fusion Approaches: 3-7% relative WER reduction on Eval2000 is achieved by all fusion approaches over the baseline on Switchboard and CallHome.
- Fusion Approaches: Shallow fusion is the clear winner on Switchboard, achieving almost double the baseline gains of deep and cold fusion.
- Fusion Approaches: Shallow fusion is best on both Google data sets, tied with cold fusion on VS14K, while deep fusion provides no or negligible gain on large-scale data.
- Proposed Approaches: The LM multitask approach achieves minor gains over the LAS baseline without introducing new parameters.
- Proposed Approaches: Introducing a pretrained LM as a lower decoder layer performs comparably to deep and cold fusion, while a randomly initialized RNN yields only marginal gains.
- Second Pass Rescoring: Cold fusion has significantly better oracle WER on VS14K and lowers WER from 5.3 to 5.0 after second-pass rescoring, unlike shallow fusion.
- Second Pass Rescoring: Cold fusion has the lowest oracle WER on D15K, but none of the models benefit much from second-pass rescoring, so shallow fusion remains best.
- Second Pass Rescoring: Shallow fusion has higher oracle WER than the baseline on both Google data sets, possibly because the LM can pull poor transcripts into the beam.
6. CONCLUSION
The study compares language-model integration methods for encoder-decoder ASR across Switchboard and Google data sets, finding that the best method depends on decoding stage and data set.
- Shallow fusion performs best for first-pass scoring across all evaluated data sets.The authors characterize shallow fusion as a simple approach.
- Cold fusion yields lower oracle error rates among the top-8 decoded transcripts and surpasses shallow fusion after second-pass rescoring on Google voice search.
- Deep fusion is comparable to cold fusion on Switchboard but provides no or negligible gains over baseline on Google data sets.The authors suggest this indicates poor scaling with data.
- Using a pretrained language model as a lower decoder layer performs comparably to cold and deep fusion on Switchboard.The authors suggest this approach warrants further investigation.