Source-linked AI summary
Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition
Zhong Meng, Yashesh Gaur, Naoyuki Kanda, Jinyu Li, Xie Chen, Yu Wu, Yifan Gong
TL;DR
E2E ASR adaptation under domain mismatch often requires audio, although target-domain text is easier to collect in large quantities. ILMA fine-tunes the E2E model’s internal LM using text-only data, with internal LM training and KLD regularization. On 30K-hour trained transformer transducers, it achieves up to 34.9% and 6.6% relative WER reductions on cross-domain and intra-domain test sets, respectively.
Problem
E2E ASR performance degrades under source-to-target domain mismatch, and existing adaptation methods require audio despite the greater availability of target-domain text.
Source: §1 · p. 1Method
Results
Up to 34.9% and 6.6% relative WER reductions from the baseline T-T were achieved on cross-domain and intra-domain test sets, respectively.
Source: §5 · p. 4Takeaways & Limitations
ILMA enables fast text-only adaptation without increasing run-time computational cost during inference.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Text-only adaptation of an end-to-end (E2E) model remains a challenging task for automatic speech recognition (ASR). Language model (LM) fusion-based approaches require an additional external LM during inference, significantly increasing the computation cost. To overcome this, we propose an internal LM adaptation (ILMA) of the E2E model using text-only data. Trained with audio-transcript pairs, an E2E model implicitly learns an internal LM that characterizes the token sequence probability which is approximated by the E2E model output after zeroing out the encoder contribution. During ILMA, we fine-tune the internal LM, i.e., the E2E components excluding the encoder, to minimize a cross-entropy loss. To make ILMA effective, it is essential to train the E2E model with an internal LM loss besides the standard E2E loss. Furthermore, we propose to regularize ILMA by minimizing the Kullback-Leibler divergence between the output distributions of the adapted and unadapted internal LMs. ILMA is the most effective when we update only the last linear layer of the joint network. ILMA enables a fast text-only adaptation of the E2E model without increasing the run-time computational cost. Experimented with 30K-hour trained transformer transducer models, ILMA achieves up to 34.9% relative word error rate reduction from the unadapted baseline.
1. Introduction
Text-only data offers a plentiful route to adapting E2E ASR models, but existing approaches bring inference or adaptation costs. ILMA adapts the E2E model’s internal LM and reaches up to 34.9% relative WER reduction from the unadapted T-T.
1. Introduction
- E2E ASR performance degrades under source-to-target domain mismatch, while existing adaptation methods require audio and target-domain text is easier to collect.Text-only data can be collected in orders of magnitude greater quantities than target-domain audio-transcript pairs.Source: §1 · p. 1
- LM fusion methods require an external LM at inference, increasing run-time computational cost.Fusion methods include shallow, density-ratio, internal-LM-estimation, and structural approaches.Source: §1 · p. 1
- TTS-based adaptation requires transcribed speech to train TTS and costly synthesis, while other text-only approaches add training steps or model costs.These costs make frequent on-device adaptation difficult.Source: §1 · p. 1
- ILMA fine-tunes the E2E components excluding the encoder using text-only data, with internal LM training and KLD regularization.For transducers, the internal LM comprises the prediction and joint networks; for AED models, it is the decoder.Source: §1 · p. 1
- Up to 34.9% relative WER reduction from the unadapted T-T was achieved on Microsoft production test sets.The evaluation used 30K-hour trained transformer transducers.Source: §1 · p. 2
2. Related Work
This section describes E2E ASR components, the internal LM implicit in an E2E model, and ILMT, which trains that LM alongside the standard E2E objective. ILMT strengthens the internal LM while maintaining ASR performance.
2.1. E2E Models for ASR
- A transducer uses an encoder, prediction network, and joint network; the joint network combines encoder and predictor outputs to predict the next token.The output distribution includes non-blank tokens and the blank symbol.Source: §2.1 · p. 2
2.2. Internal LM of E2E Model
- The E2E model implicitly learns an internal LM over token sequences, estimated by zeroing the encoder contribution; a transducer’s internal LM uses its prediction and joint networks.The internal LM output is obtained by feeding previous tokens through those networks.Source: §2.2 · p. 2
2.3. Internal LM Training (ILMT)
- ILMT adds an internal LM loss to the standard E2E loss, updating internal LM parameters to maximize transcript probability while the full model learns speech-conditioned probabilities.ILMT strengthens the internal LM and reduces perplexity without losing ASR performance.Source: §2.3 · p. 2
3. Internal LM Adaptation (ILMA)
ILMA adapts an E2E model’s internal LM using text-only data, with prior ILMT preserving its ability to collaborate with the encoder and KLD regularization constraining adaptation.
3. Internal LM Adaptation (ILMA)
- ILMA retrains the internal LM on text-only adaptation sentences by minimizing cross-entropy.The sentences in the text-only adaptation set supply the internal-LM training data.Source: §3 · p. 2
- Prior ILMT teaches a dual-mode internal LM to act as a standalone LM while retaining its ability to collaborate with the encoder during E2E inference.During ILMA, the LM mode adapts toward text-only data while the E2E-component mode remains unchanged.Source: §3 · p. 2
- KLD regularization penalizes divergence between adapted and unadapted internal-LM output distributions to guard against overfitting adaptation text.The regularized loss reduces to cross-entropy ILMA when ρ = 0.Source: §3 · pp. 2–3
- Updating only the joint network is most effective for transducers; because adaptation text lacks blank tokens, this preserves blank prediction while adapting non-blank prediction.The proposed explanation is that the adapted parameters do not contribute to blank-token probability.Source: §3 · p. 3
- The workflow applies ILMT on paired training data, ILMA on text-only data, then decodes with the adapted E2E model without added inference cost.Compared with FTA, ILMA also avoids training an additional output-LM component.Source: §3 · p. 3
4. Experiments
Experiments evaluate text-only ILMA on cross-domain Command & Control and intra-domain Voice Search, using transformer transducers trained with standard or ILMT loss and no external LM at inference. Results show the value of ILMT training and favor updating only the joint-network output layer.
4.1. Data Preparation
- T-T models trained on 30K hours were adapted with 50M-word Command & Control plus 400K real-text data or 2B-word Multi-Domain data; tests used 1K Command & Control and 18K Voice Search utterances.The Command & Control test set is cross-domain, while Voice Search is intra-domain.Source: §4.1 · p. 3
4.2. Baseline Systems
- ILMT reduced internal-LM perplexity from 131.2 to 60.7 and slightly improved baseline WER; FTA and Shallow Fusion served as comparison systems.FTA achieved 10.04% and 4.15% WER on Command & Control and Voice Search, respectively; Shallow Fusion achieved 8.71% and 4.04%.Source: §4.2 · p. 3
4.3.1. Cross-Domain Evaluation
- On Command & Control adaptation, text-only ILMA updating the joint network alone at ρ=0.2 reduced baseline T-T WER by 34.9% and yielded 33.5% and 16.9% lower WER than ILMT-ed T-T and FTA, respectively.This scheme also reduced WER by 13.8% and 16.8% relative to adapting the whole internal LM and prediction network alone, respectively.Source: §4.3.1 · p. 4
4.3.2. Intra-Domain Evaluation
- When adapting the joint network’s internal-LM component with Multi-Domain text at ρ = 0.5, ILMA achieved 6.6% relative WER reduction from baseline T-T and 6.4% and 4.8% from ILMT-ed T-T and FTA.Against Shallow Fusion, ILMA achieved a 2.2% relative WER reduction with 46.4% fewer model parameters and 25.6% shorter inference time.Source: §4.3.2 · p. 4
4.4. Result Analysis
- Standard-loss ILMA may degrade performance, whereas ILMT-trained ILMA is effective; KLD regularization with ρ > 0 reduces WER, and joint-output-only updates consistently perform best.Compared with FTA, ILMA saves one output-LM training step and achieves 4.8%–16.9% relatively lower WERs.Source: §4.4 · p. 4
5. Conclusion
ILMA fine-tunes an E2E model’s internal LM on text-only data, with internal LM training and KLD regularization supporting adaptation. On a 30K-hour trained T-T model, it reduces relative WER on cross-domain and intra-domain test sets.
5. Conclusion
- ILMA fine-tunes the internal LM with adaptation text using cross-entropy, while KLD regularization is added to avoid overfitting.Source: §5 · p. 4
- Internal LM training before ILMA helps it behave like a standalone NN-LM while remaining functional as an inseparable E2E component.Source: §5 · p. 4
- 34.9% and 6.6% relative WER reductions from the baseline T-T on cross-domain and intra-domain test sets, respectively.The reported results use a 30K-hour trained T-T model.Source: §5 · p. 4
- Updating only the last linear layer of the joint network consistently achieves the best ILMA performance, and ILMT is necessary for effective and robust ILMA.Source: §5 · p. 4