Source-linked AI summary
An analysis of incorporating an external language model into a sequence-to-sequence model
Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N. Sainath, Zhifeng Chen, Rohit Prabhavalkar
TL;DR
Attention-based ASR trains its internal language model only on transcribed audio-text pairs, motivating external text-only language models. The paper evaluates shallow fusion across tasks, decoding units, and LM types, finding a 9.1% relative WERR with a wordpiece RNN LM on Google Voice Search and eliminating second-pass rescoring.
Problem
The LAS decoder language model is trained only on transcribed audio-text pairs, raising concerns about learning rare words and phrases in long-tail Voice Search queries.
Method
The paper evaluates shallow fusion, which incorporates separately trained n-gram or RNN language models at each beam-search step across tasks and decoding units.
Results
9.1% relative WERR is achieved on a competitive unidirectional Voice Search system with a wordpiece-level RNN LM.
Takeaways & Limitations
Shallow fusion with the RNN LM eliminated the need for second-pass rescoring despite being 70 times smaller than the second-pass LM.
Takeaways & Limitations
Bayesian interpolation and dictionary constraints are described as techniques currently available only for n-gram models, leaving analogous RNN-LM methods for future work.
Abstract
from arXiv · showhide
Attention-based sequence-to-sequence models for automatic speech recognition jointly train an acoustic model, language model, and alignment mechanism. Thus, the language model component is only trained on transcribed audio-text pairs. This leads to the use of shallow fusion with an external language model at inference time. Shallow fusion refers to log-linear interpolation with a separately trained language model at each step of the beam search. In this work, we investigate the behavior of shallow fusion across a range of conditions: different types of language models, different decoding units, and different tasks. On Google Voice Search, we demonstrate that the use of shallow fusion with a neural LM with wordpieces yields a 9.1% relative word error rate reduction (WERR) over our competitive attention-based sequence-to-sequence model, obviating the need for second-pass rescoring.
1. INTRODUCTION
The paper investigates whether shallow fusion with separately trained language models can improve attention-based ASR across language-model types, decoding units, and task scales. It reports stronger gains from RNN LMs and a 9.1% relative WERR on Google Voice Search with a wordpiece RNN LM.
- Motivation: LAS trains its decoder language model only on transcribed audio-text pairs, unlike state-of-the-art language models trained on a billion words or more.The authors identify rare words and phrases in the long tail of Google Voice Search queries as a particular concern.
- Prior work: Shallow fusion incorporates an external language model through log-linear interpolation during beam-search decoding.Earlier work also used n-best rescoring, but shallow fusion applies the external model at each decoding step.
- Scope: Earlier shallow-fusion studies focused on n-gram LMs, grapheme outputs, and the relatively small WSJ task.The paper extends this setting to RNN LMs, wordpieces, and large-scale Voice Search.
- Findings: RNN LMs reduce error more effectively than n-gram LMs on the small-corpus task, with the reduction consistent across sub-word units.This is the paper’s first stated goal.
- Findings: 9.1% relative WERR results from shallow fusion with a wordpiece-level RNN LM on a competitive unidirectional Voice Search baseline.The result addresses a large-scale, large-vocabulary task where the LAS decoder might otherwise be strong enough to limit external-LM benefits.
2. SHALLOW FUSION WITH LAS MODELS
LAS combines an acoustic encoder, attention-based alignment, and decoder, whose internal language model is trained only with audio-text pairs. Shallow fusion adds a separately trained LM at inference by interpolating its score during beam search.
- 2.1. Listen, attend, and spell: LAS consists of an LSTM encoder, an attention mechanism, and an LSTM decoder conditioned on the attention context.The encoder processes acoustic feature vectors, attention selects relevant encoder features, and the decoder predicts output symbols.
- 2.1. Listen, attend, and spell: The LAS decoder acts as a neural LM conditioned on acoustic output but can only be trained on audio-text pairs.This structure motivates adding an external LM trained on text-only data.
- 2.2. Integrating a language model: Shallow fusion incorporates an external LM during inference only, leaving the dotted-box LAS components as the trained model.The external LM is combined with LAS through log-linear interpolation at every beam-search step.
- 2.2. Integrating a language model: The decoding criterion can include the external LM probability and tuned weights alongside the LAS score and coverage penalty.The coverage penalty is designed to penalize incomplete transcripts.
- 2.2. Integrating a language model: Coverage measures how extensively attention weights cover input frames and is applied only during decoding.It addresses truncated-output failures by promoting transcripts requiring attention to more audio frames.
- 2.2. Integrating a language model: Shallow fusion reduced error more than n-best rescoring in initial WSJ experiments because correct prefixes could be pruned before entering the n-best list.Applying the LM during beam search avoids relying exclusively on already-generated hypotheses.
3. EXPLORING SHALLOW FUSION ACROSS TASKS, DECODING UNITS, AND TYPES OF LANGUAGE MODELS
The study tests shallow fusion across task scale, vocabulary, domain data, decoding units, and LM architectures. It compares graphemes with wordpieces and n-gram LMs with RNN LMs while considering domain matching and dictionary constraints.
- Task properties: Training-corpus size, vocabulary size, and availability of same-domain LM data are identified as task properties affecting external-LM gains.A larger training corpus can strengthen the LAS decoder, while larger vocabularies increase exposure to unseen words and phrases.
- Tasks: WSJ provides substantial same-domain text-only data and a relatively small speech corpus, making it suitable for comparing external-language-model effects.The setup also supports direct comparison with previous work.
- Tasks: Voice Search tests transfer to a much larger task with a large vocabulary and a very long tail of queries.Its larger training set means gains observed on WSJ may not necessarily transfer.
- Decoding Units: Wordpieces vs. Graphemes: Wordpieces range from single graphemes to complete words, avoid out-of-vocabulary terms, and capture more context per decoding step than graphemes.The resulting shorter dependency length is expected to make wordpiece LMs more effective in shallow fusion.
- Language Models: RNNs vs. n-gram: The study compares n-gram and RNN LMs at the LAS decoding-unit level for shallow fusion.RNN LMs are trained on graphemes or wordpieces, while n-gram setups can additionally exploit word-level constraints and dictionary information.
- Language Models: RNNs vs. n-gram: RNN LMs are hypothesized to reduce error more than n-gram LMs because they achieve lower perplexity, particularly on rare words.The paper notes that Bayesian interpolation and dictionary constraints were, at the time, available only for n-gram models.
4. EXPERIMENTAL DETAILS
The experiments evaluate external language models on WSJ and Google Voice Search using grapheme and wordpiece LAS systems with task-specific training and text data.
- Wall Street Journal: WSJ experiments use si284 for training, dev93 for validation, and eval92 for evaluation.
- Wall Street Journal: WSJ external LMs include two-layer 512-unit LSTM RNNs and Katz-smoothed, pruned word-, grapheme-, and wordpiece-level n-gram models.The word-level LM uses a speller for grapheme- or wordpiece-level decoding.
- Google Voice Search: The Voice Search task contains approximately 12,500 training hours and 15M anonymized English utterances, with results from two sets of approximately 14,800 utterances.Training data are artificially corrupted with noise and reverberation, spanning 0dB to 30dB SNR.
- Google Voice Search: Voice Search uses a streaming LAS baseline with five unidirectional 1,400-unit encoder LSTM layers, two 1,024-unit decoder layers, and multi-headed attention.The unidirectional encoder is required for streaming operation.
- Common setup: All experiments use 80-dimensional log-mel features with 25-ms windows, 10-ms shifts, frame stacking, and downsampling to a 30-ms frame rate.Models are trained with cross-entropy and asynchronous stochastic gradient descent.
- Google Voice Search: Voice Search text data include billions of sentences from anonymized spoken, typed, and transcribed queries; production LMs are 4M-vocabulary 5-gram models.The RNN LM is trained on about half a billion sampled sentences using two 2,048-unit LSTM layers.
5. RESULTS
Across WSJ and Voice Search, shallow fusion consistently favors RNN language models over n-gram alternatives for both grapheme and wordpiece LAS systems. On Voice Search, the compact RNN-WP model captures the benefits of a much larger production LM without requiring second-pass rescoring.
- 5.1. Comparing LMs for shallow fusion: 3-GRAM-W barely outperforms 20-GRAM-G, so word constraints and an implicit dictionary provide only a slight benefit at matched context.
- 5.1. Comparing LMs for shallow fusion: RNN-G outperforms both n-gram LMs when fused with LAS-G on WSJ.The compared systems are RNN-G, 20-GRAM-G, and 3-GRAM-W.
- 5.2. Extending shallow fusion to wordpiece models: RNN-WP is significantly better than 3-GRAM-W on LAS-WP, matching the WSJ trend observed with grapheme decoding units.The relative improvement from RNN fusion is roughly consistent across graphemes and wordpieces, but LAS-WP and LAS-G LM results are not directly comparable because their baselines differ.
- 5.3. Scaling up to Voice Search: The Voice Search LAS-WP baseline becomes comparable with LAS-G when abundant training data are available, so the analysis focuses on LAS-WP.
- 5.3. Scaling up to Voice Search: The 80GB PRODLM2 offers only slightly more improvement than the compact PRODLM1 and is operationally unwieldy for low-latency first-pass decoding.PRODLM1 fits in memory and minimizes the search space for real-time requirements.
- 5.3. Scaling up to Voice Search: RNN-WP provides greater benefit than PRODLM2 with a 1.1GB memory footprint, and adding PRODLM2 afterward yields no further gain.This eliminates the need for second-pass rescoring in the reported Voice Search setup.
- 5.3. Scaling up to Voice Search: RNN-WP is 1.5% the size of PRODLM2 and avoids out-of-vocabulary words because it is trained on wordpieces.Its data-source mixing weights remain ad hoc, leaving more principled mixing as future work.
6. CONCLUSIONS
Shallow fusion augments LAS with an external language model at inference time. RNN LMs outperform n-gram LMs, with consistent gains across units and a 9.1% relative WERR on Voice Search.
- 9.1% relative WERR enabled shallow fusion with an RNN LM on a competitive unidirectional Voice Search system.This improvement eliminated the need for second-pass rescoring, despite the second-pass LM being 70 times larger.
- RNN LMs yielded greater improvement than n-gram LMs on the small WSJ task.The gain was consistent across grapheme and wordpiece decoding units.
- The LAS decoder was already competitive on Voice Search, so the production first-pass LM provided little additional benefit.