Source-linked AI summary
A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency
Tara N. Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, Chung-Cheng Chiu, David Garcia, Alex Gruenstein, Ke Hu, Minho Jin, Anjuli Kannan, Qiao Liang, Ian McGraw, Cal Peyser, Rohit Prabhavalkar, Golan Pundak, David Rybach, Yuan Shangguan, Yash Sheth, Trevor Strohman, Mirko Visontai, Yonghui Wu, Yu Zhang, Ding Zhao
TL;DR
Prior E2E ASR models had not been shown to beat conventional systems in both WER and latency. The paper combines a streaming RNN-T with an LAS rescorer and broader training and latency techniques, achieving better tradeoffs than a conventional model.
Problem
The central gap is the lack of evidence that E2E ASR can outperform state-of-the-art conventional models in both WER and latency.
Method
The paper develops a first-pass streaming RNN-T and second-pass LAS rescorer, using multi-domain and accented speech plus endpointing and rescoring optimizations.
Results
RNN-T+LAS surpasses a conventional model in both WER and latency.
Takeaways & Limitations
RNN-T+LAS offers a better WER-latency tradeoff for on-device ASR under streaming and low-latency interaction constraints.
Takeaways & Limitations
Multi-domain training introduces inconsistent transcription conventions across domains, including written versus spoken numeric forms.
Abstract
from arXiv · showhide
Thus far, end-to-end (E2E) models have not been shown to outperform state-of-the-art conventional models with respect to both quality, i.e., word error rate (WER), and latency, i.e., the time the hypothesis is finalized after the user stops speaking. In this paper, we develop a first-pass Recurrent Neural Network Transducer (RNN-T) model and a second-pass Listen, Attend, Spell (LAS) rescorer that surpasses a conventional model in both quality and latency. On the quality side, we incorporate a large number of utterances across varied domains to increase acoustic diversity and the vocabulary seen by the model. We also train with accented English speech to make the model more robust to different pronunciations. In addition, given the increased amount of training data, we explore a varied learning rate schedule. On the latency front, we explore using the end-of-sentence decision emitted by the RNN-T model to close the microphone, and also introduce various optimizations to improve the speed of LAS rescoring. Overall, we find that RNN-T+LAS offers a better WER and latency tradeoff compared to a conventional model. For example, for the same latency, RNN-T+LAS obtains a 8% relative improvement in WER, while being more than 400-times smaller in model size.
1. INTRODUCTION
Prior E2E models had not surpassed conventional ASR in both WER and latency. This paper develops an RNN-T+LAS system that improves both through broader training data and latency-focused endpointing and rescoring optimizations.
- E2E models replace conventional acoustic, pronunciation, and language-model components with one neural network, making them attractive for on-device ASR.On-device recognition avoids streaming audio to a server and returning results.
- RNN-T models support streaming recognition but lag behind conventional server-side ASR under low-latency constraints.
- LAS models can outperform conventional ASR but are non-streaming because they attend to the entire audio segment.
- The paper develops an on-device RNN-T+LAS model intended to surpass a conventional model in both WER and latency.
- Multi-domain audio-text training increases acoustic diversity and vocabulary coverage, while accented English data improves robustness to pronunciation differences.
- Latency improvements use RNN-T end-of-sentence decisions for microphone closure and optimizations that accelerate LAS rescoring.
2. MODEL ARCHITECTURE
The two-pass architecture combines a streaming RNN-T first pass with an LAS second pass that rescores its hypotheses using shared encoder representations.
- The first pass feeds acoustic frames through a shared multilayer LSTM encoder and an RNN-T decoder that predicts word-piece tokens streaming.
- During decoding, LAS rescores multiple top RNN-T hypotheses represented as a lattice using attention and teacher forcing.
- The second pass applies an additional encoder to the shared encoder output before processing it with an LAS decoder.
- The system extracts the highest-probability output sequence from the rescored lattice.
3. QUALITY IMPROVEMENTS
The paper improves E2E quality by expanding training across domains, handling domain-specific variation, adding accented speech, and adapting the learning-rate strategy.
- 3.1. Multi-domain Data: Multi-domain audio-text training spans search, farfield, telephony, and YouTube utterances to increase vocabulary and acoustic diversity.The E2E model uses audio-text pairs, unlike the much larger text-only data used for a conventional language model.
- 3.1. Multi-domain Data: Domain-specific transcription conventions and speaker composition create challenges when combining multi-domain data.Search queries use written numerics and one speaker, whereas YouTube queries may use spoken numerics and multiple speakers.
- 3.1. Multi-domain Data: A domain-id is fed to the E2E model to address variation across the four training domains.
- 3.2. Accented English Data: Conventional lexicons make accent robustness difficult through pronunciation variation, memory overhead, phoneme confusion, and data sparsity.An on-device E2E lexicon can exceed 0.5 GB.
- 3.2. Accented English Data: The E2E model directly predicts word pieces and uses accented English training data so pronunciation variations are handled from data without a lexicon whose size depends on variants.
- 3.3. Learning Rate Schedule: With more multi-domain data, the paper explores a constant learning rate and evaluates exponential-moving-average weights to aid convergence.
4. LATENCY IMPROVEMENTS
Latency improvements target both microphone endpointing and second-pass rescoring. The approach penalizes mistimed RNN-T end-of-query predictions and streamlines LAS computation.
- 4.1. Endpointer: An external VAD-based endpointer declares an end-of-query after speech followed by a fixed silence interval.
- 4.1. Endpointer: The RNN-T can predict an end-of-query token during beam search to signal that the microphone should close.Premature prediction causes deletion errors, while late prediction increases latency.
- 4.1. Endpointer: Training adds early and late penalties to the RNN-T end-of-query probability, with a buffer before late penalties apply.The penalty scales are tuned experimentally.
- 4.1. Endpointer: The end-of-query token is added only for search queries so endpointing can be aggressive there without affecting other domains.
- 4.2. LAS Rescoring: LAS latency is reduced by rescoring tree-based lattices with batched arc inference and caching first-pass encoder and attention computations.
5. EXPERIMENTAL DETAILS
The experiments use quantized streaming RNN-T and LAS models with stacked log-mel inputs, trained on diverse noisy data and evaluated across standard and accented speech sets.
- Model configuration: All models use 128-dimensional log-mel features computed with 32 ms windows and 10 ms hops.Four contiguous frames are stacked into a 512-dimensional representation before 3× subsampling.
- Model configuration: The decoders predict 4,096 word pieces, with LAS using four attention heads followed by two LSTM layers.
- Model configuration: The RNN-T has 120M parameters, while the complete quantized model occupies 177MB.The additional encoder and LAS decoder contribute 57M parameters.
- Training data: Multi-condition training adds noise and reverberation diversity using 0–30 dB SNR and 0–900 ms T60 conditions.Noise comes from YouTube and daily-life environmental recordings, with both 8 kHz and 16 kHz data.
- Evaluation: Evaluation includes approximately 14K voice-search utterances plus numeric, multi-talker, and accented-English test sets.Accented sets cover Australia, the United Kingdom, India, Kenya, Nigeria, and South Africa.
6.1. Quality
Quality improves through multidomain training, domain conditioning, accented speech, and a constant learning rate, with gains varying across evaluation sets.
- Evaluation setting: The reported quality results in this section exclude endpointer processing and LAS rescoring.
- Domain-ID Models: Domain conditioning lets multidomain RNN-T training improve all three evaluated datasets and outperform the conventional baseline on numeric and multi-talker data.Without domain conditioning, numeric performance degrades significantly despite gains on voice-search and multi-talker data.
- Robustness to Accents: Accented-data training improves over the conventional baseline on every accented test set.The multidomain-plus-domain-ID model otherwise degrades significantly on accented sets relative to that baseline.
- Learning Rates: A constant learning rate improves voice-search and multi-talker performance by approximately 7% and 8% relative, respectively, without significantly harming numeric performance.Further learning-rate schedule optimization is left for future work.
6.2. Latency
Latency improvements combine an RNN-T endpointer with optimized LAS rescoring, reducing endpointing delay and improving WER without affecting endpoint latency.
- Latency metrics: Endpoint latency is measured using median EP50 and 90th-percentile EP90 latency.
- E2E Endpointer: The joint RNN-T endpointer achieves 6.8% WER while reducing EP50 and EP90 by 20 ms and 70 ms, respectively.It predicts the end-of-sentence token during RNN-T training and also combines EOQ endpointing.
- E2E Endpointer: An external acoustic endpointer worsens WER from 6.2% to 7.4% while achieving 450 ms EP50 and 860 ms EP90 latency.The degradation can occur when short pauses cause the endpointer to cut off decoding hypotheses.
- Second-Pass LAS Rescoring: LAS rescoring reduces WER by 10% relative, from 6.8% to 6.1%, without affecting endpoint latency.
- Second-Pass LAS Rescoring: Batch inference reduces both median and 90th-percentile LAS computation latency by approximately 32%, reaching 97 ms at the 90th percentile.The benchmark uses 100 search utterances on a Google Pixel 4 CPU.
6.3. Comparison to Conventional Model
The proposed 0.18G RNN-T+LAS model offers a better WER–EP90 tradeoff than the conventional model at EP90 operating points of 550ms or above, while being over 400-times smaller at matched total latency.
- At EP90 operating points of 550ms or above, E2E provides a better WER and EP latency tradeoff than the conventional model.
- 8% relative WER improvement is achieved at matching 90% total latency between E2E and server models.Total latency combines EP90 latency with 90% second-pass rescoring computation latency.
- The RNN-T+LAS model is 0.18G in size, compared with the conventional system's substantially larger component models.The conventional system includes a 0.1GB acoustic model, 2.2GB pronunciation model, 4.9GB first-pass language model, and 80GB second-pass MaxEnt language model.
- At matched 90% total latency, E2E is more than 400-times smaller than the conventional model.