Source-linked AI summary
Factorized Neural Transducer for Efficient Language Model Adaptation
Xie Chen, Zhong Meng, Sarangarajan Parthasarathy, Jinyu Li
TL;DR
Neural Transducers make text-only language-model adaptation difficult because their predictor must coordinate with acoustic outputs and predict blanks. The paper separates blank and vocabulary prediction, using a standalone language model as the vocabulary predictor. With target-domain text adaptation, the factorized model reports relative WER improvements on LibriSpeech and an in-house call-center test set.
Problem
The Transducer predictor is not a standalone language model, making it difficult to adapt the model with text-only data.
Source: §1 · p. 1Method
Results
15.4% to 19.4% relative WER improvement on LibriSpeech dev and test sets, and 15.6% relative WER improvement on the in-house call-center test set, followed target-domain text adaptation.
Source: §4.1 · p. 4Takeaways & Limitations
The vocabulary predictor can be adapted with text-only data using conventional neural language-model adaptation techniques.
Source: §1 · p. 1, §5 · p. 4The additional vocabulary predictor slightly increases model parameters.
Source: §3 · p. 3
Abstract
from arXiv · showhide
In recent years, end-to-end (E2E) based automatic speech recognition (ASR) systems have achieved great success due to their simplicity and promising performance. Neural Transducer based models are increasingly popular in streaming E2E based ASR systems and have been reported to outperform the traditional hybrid system in some scenarios. However, the joint optimization of acoustic model, lexicon and language model in neural Transducer also brings about challenges to utilize pure text for language model adaptation. This drawback might prevent their potential applications in practice. In order to address this issue, in this paper, we propose a novel model, factorized neural Transducer, by factorizing the blank and vocabulary prediction, and adopting a standalone language model for the vocabulary prediction. It is expected that this factorization can transfer the improvement of the standalone language model to the Transducer for speech recognition, which allows various language model adaptation techniques to be applied. We demonstrate that the proposed factorized neural Transducer yields 15% to 20% WER improvements when out-of-domain text data is used for language model adaptation, at the cost of a minor degradation in WER on a general test set.
1. INTRODUCTION
Neural Transducers suit streaming speech recognition, but their integrated predictor is not a standalone language model, making text-only adaptation difficult. The paper proposes separating blank and vocabulary prediction so language-model adaptation can be applied to the vocabulary predictor.
1. INTRODUCTION
- Neural Transducers are a streaming-focused end-to-end ASR approach, reported to outperform traditional hybrid systems in some scenarios.The paper focuses on Transducers for streaming ASR.Source: §1 · p. 1
- Because the predictor must coordinate with the acoustic encoder, it does not function as a standalone language model, complicating text-only domain adaptation.The authors identify effective and efficient language-model adaptation for end-to-end ASR as an open problem.Source: §1 · p. 1
- Prior approaches include target-text-to-speech synthesis, external language-model fusion, and language-model-loss fine-tuning, each with practical constraints.The paper notes synthesis cost and inflexibility, dev-set tuning and weight sensitivity for fusion, and the predictor’s continued need to coordinate with acoustic outputs and predict blanks.Source: §1 · p. 1
- The proposed factorized Transducer separates blank and vocabulary prediction, making its vocabulary predictor a standalone language model for adaptation with text.The authors expect language-model improvements to transfer to speech recognition.Source: §1 · p. 1
2. NEURAL TRANSDUCER
The neural Transducer combines acoustic and label-history representations to predict output sequences over alignments that include blanks. Its predictor also handles blank emission and acoustic coordination, so it does more than predict the next vocabulary token as a language model.
2.1. Neural Transducer Architecture
- The encoder processes acoustic features while the predictor represents label history; a joint network combines their outputs to produce token probabilities.The paper uses a Transformer acoustic encoder and LSTM predictor in its Transformer Transducer backbone.Source: §2.1 · pp. 1–2
- A blank symbol φ represents a null output, and the objective minimizes negative log probability across valid alignments, with β removing blanks to recover the label sequence.The backbone is chosen with performance and computational cost in mind.Source: §2.1 · p. 2
2.2. Rethinking the Blank Symbol in Neural Transducer
- Unlike a standalone language model, the predictor must coordinate with encoder outputs and prevent repeated labels across multiple acoustic frames.The internal language model extracted from the standard Transducer has relatively low perplexity but does not operate as a standalone language model.Source: §2.2 · p. 2
- In the example alignment, the predictor emits “A” at time t but emits blank φ at t + 1, whereas a language model would predict the next vocabulary token “T”.The many blank symbols in an alignment mean the predictor’s role includes generating blanks as well as vocabulary tokens.Source: §2.2 · p. 2
3. FACTORIZED NEURAL TRANSDUCER
The factorized neural Transducer separates blank prediction from vocabulary prediction, allowing the vocabulary branch to function as a standalone language model. Its architecture supports language-model training and adaptation while retaining a separate blank predictor.
3. FACTORIZED NEURAL TRANSDUCER
- The model separates blank and vocabulary prediction, using a blank predictor and a vocabulary predictor that functions as a standalone language model.The blank branch combines acoustic and label information early through a joint network, while the vocabulary branch combines them at the logit level.Source: §3 · pp. 2–3
- The training objective combines Transducer loss with cross-entropy language-model loss, weighted by λ.λ tunes the effect of the language-model loss.Source: §3 · p. 3
- The vocabulary predictor can be replaced by an LSTM or n-gram language model trained with the same vocabulary, and avoiding large joint-network matrix computation can improve training speed and memory consumption.The vocabulary predictor has the structure of a standalone language model and outputs log probabilities over the vocabulary.Source: §3 · p. 3
- The additional vocabulary predictor causes a slight increase in model parameters.Source: §3 · p. 3
- At adaptation time, the vocabulary language model is fine-tuned on target-domain text for a specified number of sweeps.The model is trained from scratch with the factorized Transducer loss before adaptation.Source: §3 · p. 3
4. EXPERIMENTS
Experiments evaluate factorized Transducer benchmark performance and text-based language model adaptation across general, Librispeech, and in-house call-center tests. Target-domain text adaptation improves WER on both domain-specific evaluations, while adaptation-step results connect vocabulary-predictor improvement with Transducer WER gains.
4.1. Experiment Setup
- Experiments use 64 thousand hours of Microsoft speech; the general benchmark spans 13 scenarios and 1.8 million words, alongside Librispeech and in-house call-center adaptation tests.The general test averages WER over the scenarios; the domain-specific sets evaluate language model adaptation.Source: §4.1 · p. 3, §4 · p. 3
- Factorized Transducer models slightly worsen general-set WER, while adding LM loss sharply lowers PPL; the same-structure LSTM-LM has PPL 27.5.The reported PPL is computed over a 4,000-token sentence-piece vocabulary.Source: §4.1 · p. 4, Table 2 · p. 4
- 15.4%–19.4% relative WER improvement on Librispeech and 15.6% on call-center follow target-domain text adaptation; adapted factorized T-T beats shallow fusion on most Librispeech sets and call-center.Librispeech gains are reported on dev and test sets; the call-center result is from an in-house test set.Source: §4.1 · p. 4, Table 3 · p. 4, Table 4 · p. 4
- Four adaptation-text sweeps, about 10k steps each, suffice for Librispeech PPL and WER improvements to converge, and WER gain is highly correlated with vocabulary-predictor PPL improvement.The correlation is reported for factorized Transducer adaptation on Librispeech.Source: §4.1 · p. 4, Figure 4 · p. 4
5. CONCLUSION
The conclusion presents factorized neural Transducer as a way to adapt language models with target-domain text. Experiments report WER improvements and performance above shallow fusion on the standard Transducer.
5. CONCLUSION
- The factorized neural Transducer separates blank and vocabulary prediction and uses a standalone language model as its vocabulary predictor.This allows its vocabulary predictor to be adapted with text-only data.Source: §5 · p. 4
- Target-domain text adaptation improves speech-recognition WER and outperforms shallow fusion on the standard neural Transducer.The conclusion reports this outcome across its experiments without specifying a single test-set value.Source: §5 · p. 4
- The conclusion states that language-model improvement can transfer to Transducer speech-recognition performance.Source: §5 · p. 4