Source-linked AI summary
A Better and Faster End-to-End Model for Streaming ASR
Bo Li, Anmol Gulati, Jiahui Yu, Tara N. Sainath, Chung-Cheng Chiu, Arun Narayanan, Shuo-Yiin Chang, Ruoming Pang, Yanzhang He, James Qin, Wei Han, Qiao Liang, Yu Zhang, Trevor Strohman, Yonghui Wu
TL;DR
E2E models can match or surpass conventional ASR on quality and endpointer latency, but may delay partial predictions toward the utterance end, limiting prefetch time. The paper combines FastEmit for earlier RNN-T emissions with Conformer encoders and a Cascaded Encoders second-pass beam search using the same decoder. The system was better and faster than the previous best E2E system and surpassed the conventional model in quality and all latency metrics; on Voice Search, relative WER fell 10%, with median partial and prefetch latency reductions of 250ms and 170ms versus.
Problem
E2E models can match or surpass conventional ASR on quality and endpointer latency, but may delay partial predictions toward the utterance end, limiting prefetch time.
Source: §1 · p. 1, §5.2 · p. 4Method
The paper combines FastEmit for earlier RNN-T emissions with Conformer encoders and a Cascaded Encoders second-pass beam search using the same decoder.
Source: §1 · p. 1, §2.2 · p. 2, §3.3.2 · p. 3Results
Takeaways & Limitations
The Conformer RNN-T with Cascaded Encoders offers a better quality-and-latency tradeoff for streaming ASR.
Source: §6 · p. 4The Cascaded Encoders model has much higher computation cost than the rescoring approach because it uses beam search.
Source: §5.3 · p. 4
Abstract
from arXiv · showhide
End-to-end (E2E) models have shown to outperform state-of-the-art conventional models for streaming speech recognition [1] across many dimensions, including quality (as measured by word error rate (WER)) and endpointer latency [2]. However, the model still tends to delay the predictions towards the end and thus has much higher partial latency compared to a conventional ASR model. To address this issue, we look at encouraging the E2E model to emit words early, through an algorithm called FastEmit [3]. Naturally, improving on latency results in a quality degradation. To address this, we explore replacing the LSTM layers in the encoder of our E2E model with Conformer layers [4], which has shown good improvements for ASR. Secondly, we also explore running a 2nd-pass beam search to improve quality. In order to ensure the 2nd-pass completes quickly, we explore non-causal Conformer layers that feed into the same 1st-pass RNN-T decoder, an algorithm called Cascaded Encoders [5]. Overall, we find that the Conformer RNN-T with Cascaded Encoders offers a better quality and latency tradeoff for streaming ASR.
1. INTRODUCTION
Streaming E2E speech recognition must balance quality with low latency, yet partial predictions can still arrive late. This work explores FastEmit, Conformer encoders, and a second-pass strategy to improve that tradeoff.
1. INTRODUCTION
- Although E2E models can match or surpass conventional systems on quality and endpointer latency, their partial hypotheses may be delayed toward the utterance end.That delay contrasts with a conventional model trained with alignment.Source: §1 · p. 1
- FastEmit encourages non-blank emissions across the RNN-T sequence and reduces partial latency with less quality degradation than alignment-based approaches.It uses transducer forward-backward probabilities and does not require prior alignment information.Source: §1 · p. 1
- On Voice Search, the explored system achieved 10% relative WER reduction, 250ms lower median partial latency, and 170ms lower prefetch latency than.The comparison is reported against.Source: §1 · p. 1
- The paper pairs latency improvements with Conformer encoder layers and a second-pass beam search to address expected quality degradation.The Voice Search experiments compare the resulting latency and quality ideas with.Source: §1 · p. 1
2. MODEL IMPROVEMENTS
The model changes adapt Conformer encoders for streaming, multi-domain ASR and add a two-pass RNN-T beam-search path that shares its decoder.
2.1. Conformer
- The streaming Conformer uses causal context, local attention, and group normalization, while swapping convolution and self-attention removes relative positional encoding.Group normalization addresses batch-normalization statistics that can be biased by domain-skewed batches; the improved block is shown in Fig. 1.Source: §2.1 · p. 1, Figure 1 · p. 2
2.2. Two-pass with Cascaded Encoders
- Cascaded Encoders add non-causal layers above the causal encoder for a second-pass RNN-T beam search, with both passes sharing one decoder.The first pass uses only the causal encoder; the second-pass layers use left and right context from its outputs. The design targets robustness to short search utterances and long captions.Source: §2.2 · p. 2, Figure 2 · p. 2
3. LATENCY IMPROVEMENTS
This section defines latency measures for streaming applications and describes mechanisms aimed at earlier correct partial outputs and prefetches.
3.1. Endpointer Latency
- Endpointer latency measures the time from when the user finishes speaking until the system predicts end of query (EOQ).The section reports median EP50 and 90th-percentile EP90 latency.Source: §3.1 · p. 2
3.2. Prefetch Latency
- Prefetch latency measures the interval from the first correct prefetch to the end of the user's speech; absent a correct prefetch, it falls back to endpointer latency.A prefetch is correct when its partial recognition result matches the final result.Source: §3.2 · p. 2
3.2.1. E2E Prefetching
- E2E prefetching uses the RNN-T endpointer's EOQ predictions to estimate query completion for each partial result without changing decoding.A prefetch decision is made when the estimated probability exceeds a predefined threshold.Source: §3.2.1 · p. 2
3.3. Partial Latency
- Partial latency is the interval from the first correct partial hypothesis to the end of speech, and forms a lower bound for prefetch latency.It is inherent to the ASR model rather than application-pipeline trigger logic.Source: §3.3 · p. 2
3.3.1. Constrained Alignment
- Constrained alignment penalizes word beginnings and endings that fall outside buffers centered on alignment-derived ground truth.The cited work reports more accurate prediction timing and potential partial-latency improvement.Source: §3.3.1 · pp. 2–3
3.3.2. FastEmit
- FastEmit regularizes the RNN-T lattice to favor non-blank tokens over blanks across the sequence, targeting lower partial latency.It uses forward-backward probabilities and does not require prior alignment information or add training or serving cost.Source: §3.3.2 · p. 3
4. EXPERIMENTAL SETUP
The experiments use a Conformer RNN-T streaming architecture, train on diverse multi-domain audio, and evaluate on Voice Search. The setup also includes a two-pass Cascaded Encoders variant and language-model rescoring.
Model architecture
- The streaming encoder uses 17 Conformer layers, while two non-causal layers with 5.04s of additional right context form the Cascaded Encoders variant.The RNN-T decoder has a two-layer LSTM prediction network and a single-layer joint network; decoding is rescored with a 2nd-pass MaxEnt language model.Source: §4 · p. 3
- Table 1 compares RNN-T models with different encoders.The setup uses 512D Conformer layers and a 528D input representation.Source: Table 1 · p. 3, §4 · p. 3
Datasets
- Training data includes multi-domain search, farfield, telephony, and YouTube utterances, with noise, reverberation, and 8 kHz/16 kHz variants.The utterances are anonymized and hand-transcribed, with YouTube transcriptions produced semi-supervisedly.Source: §4 · p. 3
Test set
- The test set contains around 12K anonymized, hand-transcribed Voice Search utterances shorter than 5.5 seconds.It is described as representative of Google’s Voice Search traffic.Source: §4 · p. 3
5. RESULTS
The results improve streaming RNN-T quality with Conformer encoders and reduce partial or prefetch latency with FastEmit and E2E prefetching. Cascaded Encoders combine two-pass decoding with favorable quality and latency comparisons against prior E2E and conventional systems.
5.1. Quality Improvement with Conformer
- C2 yields 7% relative WER reduction and 35% speedup versus the LSTM-encoder baseline B1.Group normalization addresses the Conformer batch-normalization issue; swapping convolution and self-attention order yields the C2 gains and speedup.Source: §5.1 · p. 3
5.2. Latency Improvement
- E2E streaming systems delay partial hypotheses toward the utterance end, leaving less time for network-latency prefetching.This is the latency issue tracked by the B0 versus B1 comparison.Source: §5.2 · pp. 3–4
5.4. Comparison to Conventional Model
- T2 Two-pass reaches 5.4% WER while retaining faster prefetch than conventional B0.The authors report that RNN-T systems generally improve quality and endpointer latency over B0, while T2 Two-pass further lowers WER and remains faster in prefetch.Source: Table 2 · p. 4, §5.4 · p. 4
5.2.1. Reducing Partial Latency with Constrained Alignment
- Alignment-constrained B2 reduces PR50 latency by 210ms and PR90 latency by 230ms, while WER rises from 6.0% to 6.9%.Negative latency means the model can hypothesize the whole query before the speech ends.Source: §5.2.1 · p. 4
5.2.2. Reducing Partial Latency with FastEmit
- C3 (C2+FastEmit) reaches 5.8% WER with PR50 of -110ms and PR90 of 90ms, outperforming B1 in quality and latency.On B1, FastEmit reduces PR50 by 180ms and PR90 by 130ms, with WER moving from 6.0% to 6.2%; it requires no alignment during training.Source: §5.2.2 · p. 4
5.2.3. Reducing Prefetch Latency with E2E Prefetching
- Adding E2E prefetch to C3 gives C4 PF50 of -50ms and PF90 of 110ms, with a PFR of 1.86 and WER of 6.0%.The authors describe this WER as still comparable to B1.Source: §5.2.3 · p. 4
5.3. Two-Pass with Cascaded Encoders
- T2 Two-pass matches T1’s WER with better prefetch latency, while T2 Single-pass Noncausal reaches 4.8% WER for non-streaming use.T2 uses beam search and costs substantially more computation than T1.Source: §5.3 · p. 4
6. CONCLUSIONS
The paper combines Conformer encoders, FastEmit latency regularization, and Cascaded Encoders for streaming RNN-T. The resulting system is reported to outperform the previous best E2E system and surpass the conventional model in quality and all latency metrics.
6. CONCLUSIONS
- The system combines Conformer encoder layers for quality, FastEmit for latency regularization, and Cascaded Encoders with a shared RNN-T decoder for two-pass processing.FastEmit is described as generic to transducer-based models.Source: §6 · p. 4
- The Cascaded Encoders RNN-T system is reported better and faster than the previous best E2E system, and better than the conventional model in quality and all latency metrics.Source: §6 · p. 4