Source-linked AI summary
Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition
Zhong Meng, Sarangarajan Parthasarathy, Eric Sun, Yashesh Gaur, Naoyuki Kanda, Liang Lu, Xie Chen, Rui Zhao, Jinyu Li, Yifan Gong
TL;DR
External LM integration is challenging for E2E ASR because acoustic and language components are not explicitly separated. ILME estimates an E2E model’s internal LM by removing acoustic components and subtracts its scores during external-LM fusion. In evaluations with 30K-hour RNN-T and AED models, ILME reduced WER relative to Shallow Fusion across cross-domain and in-domain tests.
Problem
E2E ASR lacks a clear acoustic/language-model division, making external LM integration challenging.
Source: §1 · p. 1Method
Results
ILME reduced Shallow Fusion WER by 8.1%-15.5% relatively in cross-domain evaluations and 2.4%-6.8% in intra-domain evaluations.
Source: §6 · p. 6, §5.4 · p. 6Takeaways & Limitations
ILME integrates external LMs with pre-existing E2E models without additional training and consistently outperforms Density Ratio with fewer run-time parameters.
Source: §1 · p. 1, §6 · p. 6The Density Ratio derivation assumes domain-invariant acoustic conditions.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
The external language models (LM) integration remains a challenging task for end-to-end (E2E) automatic speech recognition (ASR) which has no clear division between acoustic and language models. In this work, we propose an internal LM estimation (ILME) method to facilitate a more effective integration of the external LM with all pre-existing E2E models with no additional model training, including the most popular recurrent neural network transducer (RNN-T) and attention-based encoder-decoder (AED) models. Trained with audio-transcript pairs, an E2E model implicitly learns an internal LM that characterizes the training data in the source domain. With ILME, the internal LM scores of an E2E model are estimated and subtracted from the log-linear interpolation between the scores of the E2E model and the external LM. The internal LM scores are approximated as the output of an E2E model when eliminating its acoustic components. ILME can alleviate the domain mismatch between training and testing, or improve the multi-domain E2E ASR. Experimented with 30K-hour trained RNN-T and AED models, ILME achieves up to 15.5% and 6.8% relative word error rate reductions from Shallow Fusion on out-of-domain LibriSpeech and in-domain Microsoft production test sets, respectively.
1. INTRODUCTION
E2E ASR is vulnerable to domain shift and lacks an explicit acoustic/language-model separation, complicating external LM integration. ILME estimates and subtracts an E2E model’s internal LM scores during inference, enabling integration without additional model training.
1. INTRODUCTION
- E2E ASR combines acoustic, language, and pronunciation modeling in a single network, making it more susceptible to domain shift than conventional DNN-HMM hybrid models.The cited adaptation methods require audio or audio-transcript pairs when applied to E2E models.Source: §1 · p. 1
- External LMs trained on target-domain text can support cross-domain adaptation and multi-domain in-domain recognition, but integration is challenging because E2E models do not explicitly separate acoustic and language models.External LMs may use much more text-only data than E2E training transcripts.Source: §1 · p. 1
- Density Ratio extends Shallow Fusion by subtracting a source-domain LM score, but assumes the E2E posterior factors into separately parameterized acoustic and language models.HAT estimates an internal LM by removing the encoder effect, but requires a special architecture that existing E2E models do not have.Source: §1 · p. 1
- ILME estimates an E2E model’s internal LM scores and subtracts them from the log-linear combination of E2E and external LM scores during inference.The method is designed for pre-existing E2E models, including RNN-T and AED, without additional model training.Source: §1 · p. 1
- Up to 15.5% and 6.8% relative WER reductions from Shallow Fusion were reported for cross-domain and intra-domain evaluations, respectively.The evaluation used 30K-hour-trained RNN-T and AED models; ILME also outperformed Density Ratio with 26.8%-29.8% fewer parameters.Source: §1 · p. 2
2. RELATED E2E MODELS
E2E models predict output-token sequences from speech features. RNN-T combines encoder and prediction-network representations, while AED uses attention over encoded speech to inform its decoder.
2. RELATED E2E MODELS
- An E2E model predicts a posterior distribution over output-token sequences given input speech features.RNN-T and AED prepend a start-of-sentence token to the output sequence.Source: §2 · p. 2
2.1. Recurrent Neural Network Transducer (RNN-T)
- RNN-T uses an encoder, prediction network, and joint network; the prediction network conditions on previous non-blank tokens, and the joint network combines encoder and prediction outputs to predict the next token or blank.Its output probability sums over possible blank-augmented alignments of the token sequence to speech features.Source: §2.1 · p. 2
2.2. Attention-Based Encoder-Decoder (AED)
- AED uses an encoder, attention network, and decoder; attention weights over encoded speech produce a context vector used with the previous token embedding to predict the next token or end-of-sentence symbol.The attention-weight vector has dimension T, the number of encoded speech frames.Source: §2.2 · p. 2
3. RELATED LM INTEGRATION METHODS
Shallow Fusion combines E2E and external LM probabilities during inference. Density Ratio extends this approach by subtracting a source-domain LM score, under a domain-invariant acoustic-condition assumption.
3.1. Shallow Fusion
- Shallow Fusion uses left-to-right beam search to optimize a log-linear interpolation of source-domain E2E and target-domain external LM probabilities.The external LM is trained on target-domain text, and its weight is λT.Source: §3.1 · p. 2
3.2. Density Ratio Method
- Density Ratio subtracts a source-domain LM score from the E2E and external LM score combination, assuming domain-invariant acoustic conditions and a factorization into acoustic and language models.The source-domain LM is trained on the E2E training transcript and has separate parameters from the E2E model.Source: §3.2 · pp. 2–3
4. INTERNAL LM ESTIMATION FOR E2E MODELS
ILME estimates an E2E model’s implicit internal LM by removing acoustic representations, then subtracts that estimate during external-LM fusion; JSA applies to RNN-T and AED without additional training.
4. INTERNAL LM ESTIMATION FOR E2E MODELS
- JSA approximates an E2E model’s internal LM by zeroing its acoustic representation and applying the model’s output function to its language representation.ILME subtracts this estimate from the log-linear combination of E2E and external-LM scores during inference.Source: §4 · p. 3
4.1. Internal LM Estimation for RNN-T Model
- For RNN-T, JSA removes the acoustic representation and blank from the estimated distribution; beam search subtracts its score for each non-blank candidate.The procedure uses source-domain audio-transcript pairs, a target-domain text-only LM, and the RNN-T score alone for the blank candidate.Source: §4.1 · pp. 3–4
4.2. Internal LM Estimation for AED Model
- For AED, removing the encoder-derived context leaves the decoder driven by the token sequence, and beam search subtracts its estimated internal-LM score.The AED procedure combines a source-domain audio-transcript model with a target-domain text-only LM during inference.Source: §4.2 · p. 4
5. EXPERIMENT
Experiments use beam search with a fixed beam size and word-piece output tokens across E2E and RNN-LM models.
5. EXPERIMENT
- Beam search inference uses a beam size of 25 in all evaluations.The paper uses PyTorch for all experiments.Source: §5 · p. 4
- All E2E models and RNN-LMs predict word-piece output tokens.The vocabulary contains 3999 word pieces generated by byte-pair encoding on acoustic training text.Source: §5 · p. 4
- Training transcripts and LM training text are tokenized with the 3999-piece vocabulary generated from acoustic training text.The pieces are generated using byte-pair encoding.Source: §5 · p. 4
5.1. E2E Models
The RNN-T and AED are trained on the same 30K hours of anonymized, transcribed Microsoft-service data, with distinct encoder and decoder configurations.
5.1.1. Training data
- The shared training set contains 30K hours of anonymized, transcribed Microsoft-service speech across voice search, dictation, command-and-control, and conversations.Recordings include close-talk and far-field conditions; speech features are 80-dimensional log Mel filter banks extracted every 10 ms over 25 ms, stacked in groups of three and strided by 30 ms.Source: §5.1.1 · p. 4
5.1.2. RNN-T Model
- The 30K-hour RNN-T uses a six-layer unidirectional LSTM encoder and a two-layer unidirectional LSTM prediction network, with 76M parameters.Each layer has 1024 hidden units; the model predicts 3999 word pieces and the blank token.Source: §5.1.2 · p. 4
5.1.3. AED Model
- The 30K-hour AED uses a six-layer bidirectional LSTM encoder and a two-layer unidirectional LSTM decoder, with 97M parameters.The encoder has 780 hidden units per layer and the decoder 1280; the model predicts 3999 word pieces and <eos>.Source: §5.1.3 · p. 4
5.2. Cross-Domain Evaluation
On both LibriSpeech test sets, ILME outperforms Shallow Fusion and Density Ratio for the evaluated RNN-T and AED models.
5.2. Cross-Domain Evaluation
- The source-domain E2E models are evaluated on LibriSpeech, while the external word-piece LM uses LibriSpeech text-only data.The target-domain LM combines transcripts from 960 hours of training speech with 813M words of text-only data collected from 14.5K books.Source: §5.2 · pp. 4–5
- 15.5% and 5.6% relative WER reductions over Shallow Fusion and Density Ratio, respectively, are reported for RNN-T on test-clean; AED reductions are 43.4%, 8.6%, and 4.3% versus its baseline, Shallow Fusion, and Density Ratio.The test-clean set has 53K words; dev-clean, with 54K words, is used to tune LM weights.Source: Table 1 · p. 5, §5.2 · p. 5
- 9.9% and 5.9% relative WER reductions over Shallow Fusion and Density Ratio, respectively, are reported for RNN-T on test-other; AED reductions are 44.1%, 8.6%, and 4.3% versus its baseline, Shallow Fusion, and Density Ratio.The test-other set has 52K words; dev-other, with 51K words, is used as the validation set.Source: Table 2 · p. 5, §5.2 · p. 5
5.3. Intra-Domain Evaluation
The 30K-hour models were evaluated on in-domain dictation and meeting speech using a multi-domain LM trained on 2B words; ILME performed best in both evaluations.
5.3. Intra-Domain Evaluation
- On dictation, ILME reduced RNN-T WER by 6.8% vs Shallow Fusion and 6.0% vs Density Ratio.The test set contained 15K words of in-house keyboard voice-input dictation.Source: §5.3 · p. 5, Table 3 · p. 6
- On dictation, ILME reduced AED WER by 12.2% vs its baseline, 4.6% vs Shallow Fusion, and 4.1% vs Density Ratio.The AED results were described as similar to the RNN-T results, with ILME performing best.Source: §5.3 · p. 5, Table 3 · p. 6
- On meeting speech, ILME reduced RNN-T WER by 2.4% vs Shallow Fusion and 3.1% vs Density Ratio.The test set contained 5K words of conversation speech from real meetings.Source: §5.3 · p. 5, Table 4 · p. 6
- Table 4 reports WERs for baseline (BS), Shallow Fusion (SF), Density Ratio (DR), and ILME on the in-domain in-house meeting test set; the external LM was trained with 2B-word multi-domain text, and WERR is the relative WER reduction from BS.Source: Table 4 · p. 6
5.4. Result Analysis
Across the evaluated test sets, ILME improved on Shallow Fusion and Density Ratio in both cross-domain and intra-domain evaluations, with gains varying by setting and model.
5.4. Result Analysis
- ILME reduced Shallow Fusion WER by 8.1%–15.5% cross-domain and 2.4%–6.8% intra-domain for both E2E models.The comparison summarizes all test sets in the two evaluation settings.Source: §5.4 · p. 6
- ILME consistently achieved lower WER than Density Ratio with 29.8% fewer RNN-T and 26.8% fewer AED run-time parameters.ILME does not require an additional source-domain LSTM-LM during beam search.Source: §5.4 · p. 6
- AED had lower baseline WER than RNN-T, while subtracting the internal LM gave larger relative gains for RNN-T than AED.The paper attributes AED's lower baseline WER to its bidirectional encoder and larger parameter count.Source: §5.4 · p. 6
- Most gains from all three LM integration methods transferred from development sets to test sets for both E2E models.The authors describe this as good generalization capability.Source: §5.4 · p. 6
- Relative WER reductions from the baseline were greater cross-domain than intra-domain, where training data included language styles similar to testing.Even in-domain, ILME reduced RNN-T WER by 3.2%–9.0%, compared with 0.2%–3.2% for Shallow Fusion and Density Ratio.Source: §5.4 · p. 6
5.5. Internal LM Perplexity
Internal-LM perplexity varied with training and differed substantially between AED and RNN-T, reflecting how each model's decoder relates to its encoder.
5.5. Internal LM Perplexity
- Both models' internal-LM perplexities first decreased and then increased as training progressed, with AED beginning to rise and converge earlier than RNN-T.Perplexity was evaluated on the validation set of the 30K-hour E2E training data.Source: §5.5 · p. 6, Figure 1 · p. 6
- AED's larger internal-LM perplexity suggests its decoder needs encoder input, whereas RNN-T's prediction network is trained to act like an individual LM.The RNN-T prediction network is later combined with the encoder by the joint network.Source: §5.5 · p. 6
- AED's internal LM perplexity was 796.7, versus 99.4 for RNN-T; the source-domain transcript LM had perplexity 30.1 on the same data.The maximum perplexity under a uniform distribution over tokens was 4000.Source: §5.5 · p. 6
6. CONCLUSION
ILME estimates an E2E model's internal LM by zeroing its acoustic components, then subtracts that score from the combined E2E and external-LM scores during inference.
6. CONCLUSION
- ILME integrates external LMs with pre-existing E2E models without additional training, estimating the internal LM by zeroing acoustic components.During inference, its internal-LM score is subtracted from the log-linear combination of E2E and external-LM scores.Source: §6 · p. 6
- ILME reduced Shallow Fusion WER by 8.1%–15.5% cross-domain and 2.4%–6.8% intra-domain.The evaluation used 30K-hour RNN-T and AED models on LibriSpeech and Microsoft production test sets.Source: §6 · p. 6
- ILME consistently outperformed Density Ratio in WER with 26.8%–29.8% fewer run-time parameters.The parameter reduction reflects not requiring an additional source-domain LSTM-LM during beam search.Source: §6 · p. 6