Source-linked AI summary
Towards Fast and Accurate Streaming End-to-End ASR
Bo Li, Shuo-yiin Chang, Tara N. Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, Yonghui Wu
TL;DR
Streaming ASR must balance recognition quality with low endpointing latency, and independent endpointing and recognition leave information unshared. The paper extends RNN-T EP with timing penalties, MWER training, and LAS rescoring; the combined system reports 18.7% relative WER reduction and 160ms 90-percentile latency reduction versus the original RNN-T EP.
Problem
Streaming ASR needs to balance recognition quality with latency, while separately optimized endpointer and ASR models do not share ASR information for endpointing decisions.
Source: §1 · p. 1Method
The paper adds early and late penalties to RNN-T EP, applies MWER training, and uses a second-pass LAS rescorer.
Source: §1 · p. 1Results
18.7% relative WER reduction and 160ms 90-percentile latency reduction are reported for RNN-T EP with LAS and MWER versus the original RNN-T EP on Voice Search.
Source: §1 · p. 1Takeaways & Limitations
LAS rescoring's WER gains may provide room to trade WER for latency, although the rescorer does not directly change first-pass latency.
Source: §1 · p. 1LAS rescoring cannot directly change the first-pass model's latency.
Source: §4.4 · p. 4
Abstract
from arXiv · showhide
End-to-end (E2E) models fold the acoustic, pronunciation and language models of a conventional speech recognition model into one neural network with a much smaller number of parameters than a conventional ASR system, thus making it suitable for on-device applications. For example, recurrent neural network transducer (RNN-T) as a streaming E2E model has shown promising potential for on-device ASR. For such applications, quality and latency are two critical factors. We propose to reduce E2E model's latency by extending the RNN-T endpointer (RNN-T EP) model with additional early and late penalties. By further applying the minimum word error rate (MWER) training technique, we achieved 8.0% relative word error rate (WER) reduction and 130ms 90-percentile latency reduction over on a Voice Search test set. We also experimented with a second-pass Listen, Attend and Spell (LAS) rescorer . Although it did not directly improve the first pass latency, the large WER reduction provides extra room to trade WER for latency. RNN-T EP+LAS, together with MWER training brings in 18.7% relative WER reduction and 160ms 90-percentile latency reductions compared to the original proposed RNN-T EP model.
1. INTRODUCTION
The paper targets a better balance between recognition accuracy and endpointing latency in streaming ASR. It extends RNN-T EP with timing penalties, MWER training, and LAS rescoring.
1. INTRODUCTION
- E2E models combine acoustic, pronunciation, and language modeling in one neural network, and RNN-T has shown potential for on-device streaming ASR.The paper frames low latency alongside recognition quality as critical for streaming ASR.Source: §1 · p. 1
- RNN-T EP jointly optimizes endpointing and recognition, addressing the separation between independently trained endpointer and ASR models.The paper notes that premature </s> predictions may cause both substitution errors and deletions.Source: §1 · p. 1
- The proposed approach penalizes </s> predictions that are too early or late, applies MWER training, and uses a non-streaming LAS rescorer for further WER gains.LAS does not directly change first-pass latency, but its WER improvement may allow a WER–latency trade-off.Source: §1 · p. 1
- 18.7% relative WER reduction, 40ms median latency reduction, and 160ms 90-percentile latency reduction versus the original RNN-T EP are reported for RNN-T EP with LAS and MWER.The reported comparison is on a Voice Search task.Source: §1 · p. 1
2. RNN TRANSDUCER AND ENDPOINTER
The paper describes a streaming RNN-T endpointer, then adds penalties to control when its end token is predicted, uses MWER training, and applies LAS as a second-pass rescorer.
2. RNN TRANSDUCER AND ENDPOINTER
- RNN-T encodes acoustic frames with unidirectional LSTM layers and decodes as input is encoded; its word-piece target ends with the special token </s>.The token marks the final label in the target sequence.Source: §2 · p. 2
2.1. Early and Late Penalties
- Early and late penalties modify the RNN-T log-probability for </s> to discourage predictions before or after its forced-alignment reference time.The late penalty begins only after a tbuffer grace period; the scales and buffer are tuned experimentally.Source: §2.1 · p. 2
2.2. MWER Training
- MWER training is investigated because premature </s> termination deletes the remaining reference sequence, creating sequence-level loss rather than only a token-level error.The method uses N-best hypotheses for RNN-T EP sequence training.Source: §2.2 · p. 2
2.3. Listen, Attend and Spell Rescoring
- LAS rescoring ranks the top-K RNN-T hypotheses using LAS scores, and RNN-T EP's explicit </s> score makes score combination with LAS more consistent.LAS cannot make RNN-T emit </s> faster, but its WER gains may permit a WER–latency trade-off.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUPS
Experiments use a Voice Search test set, multidomain acoustic modeling, and a defined endpointing evaluation that varies operating thresholds and penalties to examine WER–latency trade-offs.
3.1. Dataset
- The test set contains 14K anonymized, hand-transcribed Voice Search utterances shorter than 5.5 seconds, while training uses a multidomain dataset with multistyle noise augmentation.Training noise configurations are randomly sampled from three million pre-generated configurations.Source: §3.1 · pp. 2–3
3.2. Modeling
- The RNN-T uses globally normalized 128-dimensional log-mel features and unidirectional 2048-unit LSTMs with 640-unit projections; its encoder has eight LSTM layers, prediction network two, and joint layer 640 units.Inputs use a 32 ms window, 10 ms shift, and factor-three subsampling, yielding 33 Hz.Source: §3.2 · p. 3
3.3. Inference
- Evaluation reports Voice Search WER, median latency (EP50), 90-percentile latency (EP90), and endpointing coverage (EOU), with operating points varied through thresholds and the </s> penalty.EOU is the percentage of test data receiving an end-of-utterance signal; sweeping α_</s> and β gives WER–latency trade-off curves.Source: §3.3 · p. 3
4. RESULTS
The results progressively improve RNN-T endpointer quality and latency through early and late penalties, MWER training, and LAS rescoring. The strongest reported comparison combines LAS and MWER, improving both WER and latency over the original RNN-T EP.
4.1. Baseline
- B2 reduces EP50 latency by 130ms and EP90 latency by 200ms, with 5.4% absolute EOU coverage improvement but a 0.3% WER increase over B1.Table 1 reports quality and latency performance for the baseline models.Source: §4.1 · p. 3, Table 1 · p. 3
4.2. Early and Late Penalties
- The early penalty reduces WER from 7.5% to 7.2% but worsens latency; a 3-frame late-penalty buffer gives the best median latency, while 5 frames gives the best EP90, still worse than B2.The early penalty can over-delay </s>; the tested late-penalty buffers correspond to 180ms, 300ms, and 420ms grace periods.Source: §4.2 · p. 3, Table 2 · p. 3
4.3. MWER training
- 8.0% relative WER reduction, 30ms EP50 reduction, and 130ms EP90 reduction: E5 improves over original RNN-T EP (B2).E5 uses MWER training with only the late penalty; its EP50 latency matches E2. Table 3 reports models with and without MWER training.Source: Table 3 · p. 4, §4.3 · p. 4
4.4. LAS Rescoring
- 18.7% relative WER reduction, 40ms EP50 reduction, and 160ms EP90 reduction: RNN-T EP + LAS with MWER outperforms original RNN-T EP (B2).Updating both RNN-T and LAS during MWER yields a further 70ms EP90 reduction; Table 4 reports the LAS-rescoring comparisons.Source: Table 4 · p. 4, §4.4 · p. 4
4.5. Analysis
- Across WER–EP90 operating points, E5 and E8 offer better trade-offs than B1 and B2, whose baseline behaviors favor delay or latency at WER cost.Lower curves are better; Fig. 2 marks operation points reported in earlier sections.Source: Figure 2 · p. 4, §4.5 · p. 4