Source-linked AI summary
Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion
Duc Le, Mahaveer Jain, Gil Keren, Suyoun Kim, Yangyang Shi, Jay Mahadeokar, Julian Chan, Yuan Shangguan, Christian Fuegen, Ozlem Kalinli, Yatharth Saraf, Michael L. Seltzer
TL;DR
End-to-end ASR methods have struggled to handle large lists of rare contextual words in open-domain scenarios. This paper combines WFST shallow fusion, trie-based deep biasing, and NNLM contextualization; the system improves over existing contextual biasing approaches and a strong hybrid baseline. It remains lightweight and modular for quick modification without retraining the main ASR model.
Problem
Existing dynamic-context methods either depend on strong context prefixes, struggle with large biasing lists, or underperform on rare words, leaving open-domain end-to-end ASR with large rare-word lists challenging.
Source: §1 · p. 1Method
The approach combines WFST shallow fusion, trie-based deep biasing, and neural network language model contextualization.
Source: §1 · p. 1Results
19.5% relative WER improvement over existing contextual biasing approaches; on in-house open-domain and constrained tasks, 5.4%–9.3% improvement over a strong hybrid baseline.
Source: §1 · p. 1, §5.2 · p. 4Takeaways & Limitations
The final system remains lightweight and modular, allowing quick modification without retraining the main ASR model.
Source: §1 · p. 1Improving trie-based deep-biasing robustness across different biasing-list sizes remains future work.
Source: §6 · p. 4, §5.1 · p. 4
Abstract
from arXiv · showhide
How to leverage dynamic contextual information in end-to-end speech recognition has remained an active research area. Previous solutions to this problem were either designed for specialized use cases that did not generalize well to open-domain scenarios, did not scale to large biasing lists, or underperformed on rare long-tail words. We address these limitations by proposing a novel solution that combines shallow fusion, trie-based deep biasing, and neural network language model contextualization. These techniques result in significant 19.5% relative Word Error Rate improvement over existing contextual biasing approaches and 5.4%-9.3% improvement compared to a strong hybrid baseline on both open-domain and constrained contextualization tasks, where the targets consist of mostly rare long-tail words. Our final system remains lightweight and modular, allowing for quick modification without model re-training.
1. Introduction
End-to-end ASR offers simplicity and competitive generic transcription, while hybrid systems retain practical appeal through modularity and flexible language-model integration. Handling large lists of rare contextual words in open-domain use remains a challenge; this work combines three contextualization techniques to address it.
1. Introduction
- Existing shallow-fusion methods rely on strong context prefixes and generalize poorly to open-domain use, while deep-biasing methods struggle with larger lists and rare words.The paper identifies handling large biasing lists of rare words in open-domain end-to-end ASR as an ongoing challenge.Source: §1 · p. 1
- 19.5% relative WER improvement over existing contextual biasing approaches; on in-house open-domain and constrained tasks, the system improves 5.4%–9.3% over a strong hybrid baseline.The LibriSpeech system also averages 34.5% relative WER improvement over the baseline.Source: §1 · p. 1
- The proposed system combines WFST shallow fusion, trie-based deep biasing, and neural network language model (NNLM) contextualization.It remains lightweight and modular, allowing modification without retraining the main ASR model.Source: §1 · p. 1
2. Data
The experiments use LibriSpeech with constructed utterance-specific biasing lists and large-scale in-house video and assistant data. The two settings cover contextualization targets ranging from rare words to video-title terms and contact names.
2.1. LibriSpeech
- LibriSpeech evaluation adds rare reference words and 100, 500, 1000, or 2000 distractors to each test utterance’s biasing list.Rare words fall outside the 5,000 most common training words; the goal is to improve B-WER without significantly degrading U-WER.Source: §2.1 · p. 1
2.2. Large-Scale In-House Data
- In-house training data comprise 1.7M hours of public English video and 50K hours of manually transcribed, de-identified assistant-domain speech.The evaluation sets include 14.0K videos and 20.8K assistant utterances; video titles and descriptions supply open-domain biasing words, while assistant lists contain contact names.Source: §2.2 · p. 2
- Video lists average 11 words and cover 1.5% of reference words, whereas assistant lists average 876 contact names and cover 13.4%.The video task is open-domain; assistant biasing words appear mostly in limited contexts.Source: §2.2 · p. 2
3. Methods
The methods combine decoder-side WFST shallow fusion with trie-based deep biasing integrated into RNNT and NNLM models. The trie supplies WordPiece context signals, while the two neural models use those signals in different ways.
3.1. WFST Shallow Fusion with Dynamic Replacement
- A dynamically constructed generic WFST applies biasing at the WordPiece level before decoder pruning, without requiring strong context prefixes or domain-specific sentence patterns.It can also be modified to incorporate context prefixes while retaining generalizability.Source: §3.1 · p. 2
3.2. Trie-Based Deep Biasing
- The non-trainable trie encodes biasing words as WordPieces and returns a binary vector marking tokens that can start or continue a listed word.A lookup-table implementation gives O(1) query time and O(N) construction time, where N is the total number of WordPieces in the list.Source: §3.2 · p. 2, Figure 2 · p. 2
3.2.1. Deep Biasing RNNT (DB-RNNT)
- DB-RNNT projects the trie output and supplies it as an additional joiner input, giving RNNT implicit access to biasing data during inference.This addresses the limitation that vanilla RNNT may assign rare long-tail words probabilities too small for decoder-side biasing to recover.Source: §3.2.1 · pp. 2–3
3.2.2. Deep Biasing NNLM (DB-NNLM)
- DB-NNLM projects trie output and concatenates it with each NNLM hidden-layer output; the biasing and NNLM parameters are optimized together.The NNLM can be trained on text-only data, and in this work it is trained separately from RNNT to keep the system modular.Source: §3.2.2 · p. 3
4. Experimental Setup
The experiments use a streaming RNNT and a separately trained NNLM, with deep-biasing modules trained using artificial distractor lists on LibriSpeech. In-house experiments adapt the training strategy and include graphemic pronunciation variants.
4.1. LibriSpeech Experiments
- The LibriSpeech baseline is an approximately 83M-parameter streaming RNNT, paired with an approximately 105M-parameter NNLM trained on an 810M-word text corpus.The NNLM is shallow fused after discounting internal-LM scores.Source: §4.1 · p. 3
- DB-RNNT and DB-NNLM use artificial training lists with 400–800 distractors; training sometimes zeros the bias vector or removes a target word from the list.The bias vector is zeroed with 10% probability, and each target reference word is removed with 40% probability; Figure 4 plots DB-NNLM probability for “platterbaff” across list sizes.Source: §4.1 · p. 3, Figure 4 · p. 3
4.2. In-House Data Experiments
- In-house DB-RNNT uses two-stage training, and G2G generates graphemic pronunciation variants during training and decoding for biasing words.The in-house NNLM corpus is only 20% larger than paired audio-text data, potentially limiting WER improvement from NNLM and DB-NNLM.Source: §4.2 · p. 3
5. Results and Discussion
The proposed system improves contextual word error rates on LibriSpeech and in-house data, with the in-house results also outperforming a strong hybrid baseline at a much smaller size. Results also show that deep-biasing gains depend on biasing-list size and the availability of text-only data.
5.1. LibriSpeech Results
- DB-RNNT B-WER gains shrink as biasing lists grow; training with more distractors helps on larger lists but degrades results on smaller ones.The authors suggest choosing the number of sampled training distractors according to the expected test-list size.Source: §5.1 · p. 3, Table 1 · p. 4
- NNLM shallow fusion improves WER over the baseline but has neutral B-WER impact without biasing information; replacing it with DB-NNLM significantly improves B-WER.DB-NNLM is less effective when biasing lists are much larger than those seen during training.Source: §5.1 · pp. 3–4
- 19.5% average relative B-WER improvement over S3 and 34.5% over RNNT + NNLM baseline B2 with DB-NNLM (S5).The improvement is reported on LibriSpeech; Table 1 reports WER as U-WER/B-WER across biasing-list sizes.Source: Table 1 · p. 4, §5.1 · p. 4
- 13.4% average relative B-WER improvement over the best standalone technique and 38.5% over RNNT baseline B1 when combining DB-RNNT and WFST shallow fusion.The averages span all test sets and biasing lists.Source: §5.1 · p. 3, Table 1 · p. 4
5.2. In-House Data Results
- 42.3%–75.8% B-WER improvement with minimal U-WER degradation over RNNT baseline B1; on in-house data, S5 also improves U-WER and B-WER by 5.4%–9.3% over hybrid baseline B0.The in-house system is 200MB versus 4GB for B0, both with 8-bit quantization.Source: Table 2 · p. 4, §5.2 · p. 4
- DB-RNNT and DB-NNLM have larger effects on Video than Assistant, likely because Video has smaller biasing lists; DB-NNLM gains are smaller than on LibriSpeech.The authors suggest DB-NNLM is more applicable when labeled audio is limited relative to text-only data.Source: §5.2 · p. 4
6. Conclusion and Future Work
The paper presents a contextual-biasing approach that combines trie-based deep biasing, WFST shallow fusion, and NNLM contextualization. It reports WER improvements on academic and large-scale in-house data, including over a larger-footprint hybrid baseline, and identifies future directions for improving robustness and extending deep biasing.
6. Conclusion and Future Work
- The approach combines trie-based deep biasing, WFST shallow fusion, and NNLM contextualization for end-to-end ASR contextual biasing.The authors report WER improvement over previous approaches on academic and large-scale in-house data, and over a strong hybrid baseline with a higher disk and memory footprint.Source: §6 · p. 4
- The authors publicly share their LibriSpeech experimental setup to encourage replication and wider adoption.Source: §6 · p. 4
- Future work will target robustness to biasing-list sizes, closing the performance gap with WFST shallow fusion, and extending deep biasing to generic speech recognition accuracy.Source: §6 · p. 4