Source-linked AI summary
Deep context: end-to-end contextual speech recognition
Golan Pundak, Tara N. Sainath, Rohit Prabhavalkar, Anjuli Kannan, Ding Zhao
TL;DR
Contextual speech recognition benefits from information specific to the user’s situation, but traditional contextualization relies on independently trained rescoring components. CLAS embeds context phrases and jointly trains them with an end-to-end LAS recognizer, outperforming shallow-fusion approaches on several tasks while requiring further work for very large phrase sets.
Problem
Speech recognition must incorporate changing contextual information, while independently trained on-the-fly rescoring conflicts with the joint optimization of end-to-end models.
Method
CLAS is an all-neural end-to-end model that embeds full context phrases and uses an additional bias encoder and attention mechanism jointly with LAS.
Results
CLAS outperforms standard shallow-fusion biasing on several test sets, performs significantly better with hundreds of high-OOV phrases, and benefits from bias-conditioning for thousands.
Takeaways & Limitations
Jointly trained contextualization improves contextual ASR across several test sets, while bias-conditioning enables scaling to large phrase lists without degradation.
Takeaways & Limitations
Bias-conditioning depends on a rule-based prefix construction, with full algorithmic treatment left for future work; scaling CLAS to tens of thousands of phrases remains future work.
Abstract
from arXiv · showhide
In automatic speech recognition (ASR) what a user says depends on the particular context she is in. Typically, this context is represented as a set of word n-grams. In this work, we present a novel, all-neural, end-to-end (E2E) ASR sys- tem that utilizes such context. Our approach, which we re- fer to as Contextual Listen, Attend and Spell (CLAS) jointly- optimizes the ASR components along with embeddings of the context n-grams. During inference, the CLAS system can be presented with context phrases which might contain out-of- vocabulary (OOV) terms not seen during training. We com- pare our proposed system to a more traditional contextualiza- tion approach, which performs shallow-fusion between inde- pendently trained LAS and contextual n-gram models during beam search. Across a number of tasks, we find that the pro- posed CLAS system outperforms the baseline method by as much as 68% relative WER, indicating the advantage of joint optimization over individually trained components. Index Terms: speech recognition, sequence-to-sequence models, listen attend and spell, LAS, attention, embedded speech recognition.
1. INTRODUCTION
Context can substantially improve speech recognition, but existing approaches add independently trained components to end-to-end models. This work proposes CLAS, which jointly trains contextualization with ASR and performs especially well with hundreds of context phrases.
- Context such as dialog state, location, contacts, and playlists can improve recognition in mobile and assistant applications.
- Sequence-to-sequence ASR jointly models traditional acoustic, pronunciation, and language-model components in one neural network.
- Prior contextualization methods incorporate independently trained language models through on-the-fly rescoring, shallow fusion, or cold fusion.
- CLAS embeds context phrases and uses attention to summarize context during output prediction, jointly optimizing contextualization with the ASR model.
- CLAS does not require context during training or careful rescoring-weight tuning, while supporting OOV terms and outperforming online rescoring with hundreds of phrases.
2. BACKGROUND
LAS predicts grapheme sequences from acoustic features with jointly trained encoder, decoder, and attention modules. On-the-fly rescoring adds contextual language-model scores during beam search but is limited by word-boundary scoring and large, noun-heavy phrase lists.
- 2.1. The LAS model: LAS models the probability of grapheme sequences conditioned on log-mel acoustic features.
- 2.1. The LAS model: The LAS architecture jointly trains an encoder, decoder, and attention network to predict graphemes from acoustic feature frames.
- 2.1. The LAS model: The encoder is a unidirectional stacked RNN that converts acoustic features into high-level hidden states.
- 2.1. The LAS model: The decoder models output-token sequences, while multi-head attention conditions predictions on the decoder state and encoder-state sequence.
- 2.2. On-the-fly Rescoring: On-the-fly rescoring compiles bias phrases into a WFST and adds contextual-language-model scores during beam search using a tunable weight λ.
- 2.2. On-the-fly Rescoring: Without weight pushing, rescoring applies only at word boundaries and cannot help when the relevant word is absent from the beam; many proper nouns further reduce performance.
3. CONTEXTUAL LAS (CLAS)
CLAS extends LAS with a bias encoder and separate attention over contextual phrases, jointly modeling audio, output history, and context. Training uses randomly generated bias lists, while inference supports new phrases and prefix-based bias conditioning for large lists.
- Architecture: CLAS models P(y|x, z) by adding a bias encoder and attention mechanism to standard LAS.The bias phrases z represent context-relevant items such as contact names and song lists.
- Architecture: A learnable no-bias vector lets CLAS ignore all supplied phrases and fall back to bias-less decoding when none matches the audio.The model can therefore determine whether any contextual phrase is relevant to the current utterance.
- Architecture: The bias encoder embeds each phrase from decoder subword units, using the final LSTM state as the phrase representation.A separate attention mechanism computes a bias context vector from these phrase embeddings.
- Training: During training, bias lists are randomly generated from transcript n-grams, with reference transcripts optionally excluded to exercise the no-bias case.The number of phrases and maximum n-gram order are controlled by Nphrases and Norder.
- Inference: During inference, CLAS embeds user-provided context sequences before streaming and decodes hypotheses with standard LAS beam search.The supplied context phrases may differ from those seen during training.
- Bias-Conditioning: Bias-Conditioning enables a phrase only after its associated prefix appears on the partial hypothesis, reducing overload when thousands of phrases are supplied.Prefix design should balance shared-prefix counts against preserving distinctive phrase embeddings; full algorithmic construction remains future work.
4. EXPERIMENTS
The experiments use large-scale English voice-search data and test CLAS across matched bias-free sets and synthetic contextual sets. Contextual lists range from four to more than three thousand phrases and include songs, contacts, and chatbot requests.
- Training setup: The training data contain approximately 25,000 hours and 33 million anonymized, hand-transcribed English utterances.The utterances represent Google voice-search traffic and are augmented with simulated noise and reverberation.
- Training setup: Models are trained on 8 × 8 TPU slices with a global batch size of 4,096 and shards containing 32 utterances.Bias phrases are randomized within each shard during training.
- Model configuration: The acoustic input uses 80-dimensional log-mel features computed every 10ms over 25ms windows, with stacked frames downsampled by a factor of 3.This downsampling supports a simpler encoder architecture.
- Model configuration: The model has 10 unidirectional 256-node LSTM encoder layers, four attention heads, a 512-node bias encoder, and about 58 million parameters.The decoder contains four 256-node LSTM layers.
- Test sets: Voice Search and Dictation are matched, bias-free test sets, while Songs, Contacts, and Talk-To are generated with noisy text-to-speech audio.Table 1 summarizes the biasing setup, including the bias OOV rate for words absent from training data.
- Test sets: The evaluated contextual sets contain phrase lists ranging from four to more than 3,000 items, varying across utterances.Songs use artist and song names, Contacts use contact names, and Talk-To uses chatbot names.
5. RESULTS
CLAS is evaluated against LAS and on-the-fly rescoring across biasing conditions, including settings with hundreds or thousands of contextual phrases. It performs strongly with hundreds of phrases, degrades as distractors increase, and benefits from bias-conditioning for large phrase sets.
- 5.1. CLAS without bias phrases: With no bias phrases at inference, CLAS-NB performs better than vanilla LAS and is used as the LAS proxy in subsequent comparisons.CLAS-NB was trained with random bias phrases but evaluated with an empty phrase list.
- 5.2. On-the-fly (OTF) Rescoring with LAS Baseline: Biasing each subword unit gives the best OTF-rescoring performance, whereas word-end biasing yields very little improvement over the no-bias baseline.
- 5.3.1. Comparison of Biasing Approaches: CLAS is compared with a LAS baseline and LAS plus on-the-fly rescoring across multiple contextual-biasing test sets.The LAS baseline uses CLAS-NB, while the rescoring baseline estimates λ on the same test sets.
- 5.3.1. Comparison of Biasing Approaches: On Songs and Contacts, which contain hundreds of biasing phrases and high OOV rates, CLAS significantly outperforms traditional approaches without additional hyperparameter tuning.
- 5.3.1. Comparison of Biasing Approaches: On Talk-To, which contains thousands of phrases, CLAS degrades relative to the traditional approaches.The paper identifies this as a scalability issue addressed with bias-conditioning.
- 5.3.2. CLAS with varying number of bias phrases: WER gradually degrades as the number of randomly selected distractor phrases increases alongside phrases appearing in the reference transcript.This experiment evaluates CLAS without rescoring or conditioning.
- 5.3.2. CLAS with varying number of bias phrases: The authors hypothesize that large phrase lists create correlations between bias embeddings; talking pal and talkative ai have correlation 0.6 versus an average correlation of 0.2.
6. CONCLUSIONS
The paper presents CLAS as an all-neural, end-to-end contextualized ASR model that embeds full context phrases. Experiments show gains over shallow-fusion biasing on several test sets, while conditioning further improves handling of large phrase sets.
- 6. CONCLUSIONS: CLAS is an all-neural, end-to-end contextualized ASR model that incorporates context by embedding full context phrases.
- 6. CONCLUSIONS: CLAS outperforms standard shallow-fusion biasing techniques on several test sets.
- 6. CONCLUSIONS: Bias-conditioning is suggested as a method for improving CLAS quality with large context-phrase sets.Scaling CLAS to tens of thousands of bias phrases is identified as future work.