Source-linked AI summary
Streaming End-to-end Speech Recognition For Mobile Devices
Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo-yiin Chang, Kanishka Rao, Alexander Gruenstein
TL;DR
On-device ASR requires streaming, real-time, accurate recognition that handles long-tail utterances and user-specific context, while conventional compact systems remain weak on challenging tasks. The paper builds a streaming RNN-T recognizer with architectural, contextual-biasing, quantization, and synthetic-TTS techniques. It runs twice as fast as real time on a Google Pixel and improves WER by more than 20% over a conventional embedded CTC baseline on voice search and dictation.
Problem
On-device ASR must be accurate, real-time, streaming, context-aware, and robust to long-tail utterances, while compact conventional systems perform significantly worse on challenging voice search.
Method
The paper develops an end-to-end RNN-T recognizer using architectural and runtime optimizations, shallow-fusion contextual biasing, and synthetic TTS data for numeric normalization.
Results
Twice as fast as real time, the proposed system improves WER by more than 20% over a conventional embedded CTC model on voice search and dictation.
Takeaways & Limitations
The results show that an end-to-end all-neural model can perform streaming, high-accuracy, low-latency, contextual speech recognition on device.
Abstract
from arXiv · showhide
End-to-end (E2E) models, which directly predict output character sequences given input speech, are good candidates for on-device speech recognition. E2E models, however, present numerous challenges: In order to be truly useful, such models must decode speech utterances in a streaming fashion, in real time; they must be robust to the long tail of use cases; they must be able to leverage user-specific context (e.g., contact lists); and above all, they must be extremely accurate. In this work, we describe our efforts at building an E2E speech recognizer using a recurrent neural network transducer. In experimental evaluations, we find that the proposed approach can outperform a conventional CTC-based model in terms of both latency and accuracy in a number of evaluation categories.
1. INTRODUCTION
On-device ASR could improve reliability, latency, and privacy, but existing systems lose accuracy on challenging tasks. The paper develops a streaming RNN-T recognizer addressing real-time decoding, context, long-tail normalization, and accuracy.
- On-device recognition avoids sending audio to servers, with implications for reliability, latency, and privacy.
- Shrunk conventional ASR components match accuracy in narrow domains but perform significantly worse than server systems on challenging voice search.
- E2E models replace acoustic, pronunciation, and language components with one end-to-end trained neural model that directly predicts character sequences.
- Production E2E systems must maintain accuracy and real-time latency, use user context, and handle long-tail utterances such as spoken numeric sequences.
- The proposed RNN-T system combines architectural, training, contextual-biasing, quantization, and synthetic-TTS improvements; it runs twice as fast as real time and improves WER by more than 20% over a conventional CTC model.
2. RECURRENT NEURAL NETWORK TRANSDUCER
RNN-T models speech with alignments that condition label emissions on prior non-blank label history, unlike CTC's conditional independence assumption. Both models avoid future acoustic frames, enabling streaming recognition.
- The input is a sequence of 80-dimensional log-Mel acoustic frames, while targets are sequences of phonemes, graphemes, or word-pieces.
- CTC augments the output vocabulary with a blank symbol and sums probabilities over frame-level alignments that collapse to the target sequence.
- CTC assumes labels are conditionally independent given acoustics, whereas RNN-T conditions each alignment label on the history of previous non-blank labels.
- RNN-T alignments contain T blanks and U labels, and removing blanks yields the target sequence.
- Both models prohibit an alignment label from depending on future acoustic frames, allowing streaming systems without waiting for the full utterance.
- CTC uses an encoder and softmax, while RNN-T combines encoder outputs with a label-history prediction network through a feed-forward joint network.
3. REAL-TIME SPEECH RECOGNITION USING RNN-T
The system uses a compact recurrent architecture and runtime optimizations to make RNN-T accurate and efficient on mobile devices. Time reduction, caching, asynchronous execution, and quantization target real-time inference.
- 3.1. Model Architecture: The mobile architecture uses eight unidirectional LSTM encoder layers with projection layers that reduce recurrent and output connections.
- 3.1. Model Architecture: A time-reduction layer concatenates N adjacent frames, reducing the frame rate by a factor of N and lowering computation.
- 3.1. Model Architecture: Placing time reduction after the second RNN-T encoder layer improves speed without accuracy loss, whereas the same change degrades CTC phoneme-model accuracy.
- 3.2. Optimization: Layer normalization stabilizes recurrent hidden-state dynamics, while word-piece units outperform graphemes in the experiments.
- 3.3. Runtime Optimization: Caching identical prediction histories saves 50–60% of prediction-network computation, and separate encoder and prediction-network threads enable asynchronous pipelining.
- 3.4. Parameter Quantization: Eight-bit quantization reduces memory use and speeds execution; on ARM architectures it achieves a 3× speedup over floating-point execution.
4. CONTEXTUAL BIASING
The RNN-T recognizer incorporates user-specific context during inference by interpolating contextual biasing scores with the base model during beam search.
- Contextual biasing injects prior knowledge such as songs, contacts, apps, or location into ASR during inference.
- The system computes contextual scores and combines them with the base model using shallow fusion during beam search.The contextual scores are denoted P_C(y), while λ controls their influence on the overall score.
- The contextual finite-state transducer for “cat” represents the word at the subword-unit level with backoff arcs.
5. TEXT NORMALIZATION
The paper addresses numeric text normalization by directly training the E2E model to produce written-domain outputs rather than adding a separate conversion stage.
- Conventional ASR systems convert spoken numeric sequences into written forms during decoding.Class-based language models and grammar WFSTs map numeric classes to possible written instances.
- Spoken-domain recognition alleviates numeric data sparsity by using class-based language models and hand-crafted grammar WFSTs.
- The E2E model is trained to output hypotheses directly in normalized written form to keep the overall system small.The alternative would be a separate neural-network or FST-based conversion stage.
6. EXPERIMENTAL DETAILS
Experiments use large-scale noisy English speech, multiple voice-search, dictation, contextual-biasing, and numeric test sets, plus specified acoustic features, model architectures, and a strong CTC baseline.
- Data Sets: 35 million English utterances, totaling approximately 27,500 hours, form the anonymized, hand-transcribed training set.The data represent voice-search and dictation traffic and are corrupted with simulated noise and reverberation at 0–30dB SNR.
- Data Sets: The main test sets contain 14.8K voice-search utterances and 15.7K dictation utterances.
- Data Sets: Contextual-biasing evaluation uses Songs, Contacts-Real, Contacts-TTS, and Apps voice-command test sets.Except for Contacts-Real, these sets are created using web-mined names and synthesized TTS utterances.
- Data Sets: Numeric evaluation includes real-data Num-Real and synthesized Num-TTS test sets, with no test utterance or transcript overlapping the numerics TTS training set.
- Model and Baseline: The encoder uses stacked log-Mel features and an eight-layer LSTM architecture with a time-reduction layer after the second layer.The time-reduction factor N = 2 yields a reported 1.7× speed improvement without accuracy loss.
- Model and Baseline: The RNN-T is compared with a larger embedded CTC baseline using a six-layer LSTM acoustic model, 42-phoneme output, a 500K-word lexicon, and language-model rescoring.
7. RESULTS
The RNN-T system improves accuracy over the conventional embedded CTC baseline, supports contextual biasing and numeric normalization, and achieves substantially faster inference through word-piece modeling and quantization.
- Quality Improvements: 27% and 25% relative WER improvements are achieved on voice search and dictation, respectively, over the baseline conventional CTC embedded model.Layer normalization, larger-batch TPU training, and word-piece targets contribute to the improvements.
- Contextual Biasing: E2E contextual biasing outperforms or matches conventional-model biasing on every evaluated set except Songs.The Songs exception is attributed to its higher out-of-vocabulary rate: 1.0%, versus 0.2% for Contacts and 0.5% for Apps.
- Text Normalization: Numeric-set errors primarily arise from text normalization, such as producing “2 double 3 4” instead of “2334.”Training with additional numeric examples substantially mitigates the issue, though with a small degradation on voice search and dictation.
- Real Time Factor: The word-piece RNN-T model outperforms the grapheme model in both WER and RT90, where lower RT90 indicates faster processing and lower latency.RT90 is the real-time factor at the 90th percentile, defined as processing time divided by audio duration.
- Real Time Factor: 64% RT90 reduction from symmetric quantization yields inference twice as fast as real time, while reducing model size by 4×.Asymmetric quantization improves RT90 by 28% with a 0.1% absolute WER degradation; the best model remains over 20% more accurate than the conventional CTC model.
8. CONCLUSIONS
The paper presents a compact RNN-T-based E2E recognizer for on-device speech recognition. On a Google Pixel phone, it runs twice as fast as real time and improves WER by more than 20% over a strong embedded baseline on voice search and dictation.
- 8. CONCLUSIONS: The proposed compact E2E speech recognizer is based on an RNN-T model and uses architectural modifications, quantized inference, and synthesized TTS training data.These design choices target streaming, contextual, accurate, and low-latency on-device recognition.
- 8. CONCLUSIONS: Twice-as-fast real-time inference and more than 20% WER improvement are achieved over a strong embedded baseline on voice search and dictation.The result is reported for a Google Pixel phone.