Source-linked AI summary
Two-Pass End-to-End Speech Recognition
Tara N. Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, Ian McGraw, Chung-Cheng Chiu
TL;DR
On-device speech recognition needs streaming, low-latency decoding, but streaming RNN-T remains lower quality than large conventional systems while LAS is non-streaming. The paper combines a streaming RNN-T first pass with a shared-encoder LAS second pass, achieving a 17%-22% relative WER reduction with less than 200 ms of added finalization latency.
Problem
Streaming RNN-T meets on-device latency requirements but trails large conventional models in quality, whereas LAS is competitive in quality but non-streaming.
Method
A shared encoder feeds an RNN-T decoder for streaming predictions and a LAS decoder that rescales first-pass hypotheses, with MWER training aligning rescoring optimization and decoding.
Results
17%-22% relative WER reduction is achieved over RNN-T only, with latency increased by less than 200 ms.
Takeaways & Limitations
The two-pass design combines streaming RNN-T output with LAS-based quality improvement while keeping the reported latency increase below 200 ms.
Takeaways & Limitations
LAS is non-streaming because it must attend to the entire audio segment, limiting its direct use for interactive recognition.
Abstract
from arXiv · showhide
The requirements for many applications of state-of-the-art speech recognition systems include not only low word error rate (WER) but also low latency. Specifically, for many use-cases, the system must be able to decode utterances in a streaming fashion and faster than real-time. Recently, a streaming recurrent neural network transducer (RNN-T) end-to-end (E2E) model has shown to be a good candidate for on-device speech recognition, with improved WER and latency metrics compared to conventional on-device models [1]. However, this model still lags behind a large state-of-the-art conventional model in quality [2]. On the other hand, a non-streaming E2E Listen, Attend and Spell (LAS) model has shown comparable quality to large conventional models [3]. This work aims to bring the quality of an E2E streaming model closer to that of a conventional system by incorporating a LAS network as a second-pass component, while still abiding by latency constraints. Our proposed two-pass model achieves a 17%-22% relative reduction in WER compared to RNN-T alone and increases latency by a small fraction over RNN-T.
1. Introduction
On-device ASR requires streaming, low-latency recognition, and context use, but streaming RNN-T quality trails large conventional systems. The proposed shared-encoder two-pass architecture uses streaming RNN-T followed by LAS processing to improve quality while limiting added latency.
- Motivation: On-device ASR must produce streaming results with low latency and support user context, while RNN-T quality still trails a large conventional model.
- Motivation: LAS achieves competitive performance with large conventional models but must attend to the entire audio segment, preventing straightforward streaming use.
- Two-pass decoding: Two-pass decoding improves first-pass outputs through lattice rescoring or n-best reranking, but preserving low user-perceived latency is the central challenge.
- Proposed approach: The proposed architecture shares an encoder between RNN-T and LAS, producing streaming RNN-T predictions before LAS finalizes them.The shared encoder reduces model size and computation compared with dedicated encoders, while the study compares LAS beam search with RNN-T hypothesis rescoring.
- Results: 17%-22% relative WER reduction is obtained over first-pass RNN-T, while second-pass LAS increases finalization latency by less than 200 ms.Experiments use a ∼30,000 hour voice search task and report tradeoffs between beam search and rescoring.
2. Two-Pass E2E ASR
The two-pass architecture shares an encoder between streaming RNN-T decoding and a second-pass LAS decoder, which either beam-searches or rescores first-pass hypotheses. Training progresses from an RNN-T model to LAS initialization and joint fine-tuning, then optionally refines LAS with MWER training aligned to the decoding mode.
- Architecture: A shared encoder feeds streaming RNN-T predictions in the first pass and full-utterance representations to the second-pass LAS decoder.The shared encoder reduces model size and computation compared with separate encoders.
- Inference: The LAS decoder operates either in 2nd beam search mode, ignoring RNN-T output, or rescoring mode, selecting the highest-scoring sequence among top-K RNN-T hypotheses.Rescoring combines LAS sequence log probability with an attention coverage penalty.
- Training: The combined training loss equally weights RNN-T and LAS losses with λ = 0.5, but direct training from scratch is unstable because their loss ranges differ substantially.The model is therefore trained through staged initialization and joint deep fine-tuning.
- Training: Training proceeds by first training RNN-T, freezing its encoder while training LAS, then jointly fine-tuning the shared encoder and both decoders.This staged procedure initializes the two-pass model before applying the combined loss.
- MWER training: The original loss leaves LAS optimized independently of RNN-T, creating a mismatch between training and decoding strategies.Additional MWER refinement is introduced to reduce this mismatch.
- MWER training: MWER training generates hypotheses with beam search from the decoder used at inference: LAS for beam search mode and RNN-T for rescoring mode.The loss uses relative word errors and LAS probabilities over the generated hypothesis set.
3. Experimental Details
Experiments use a large anonymized English speech corpus and evaluate a two-pass architecture with specified acoustic features, recurrent encoders and decoders, and latency assumptions. The second-pass LAS decoder is evaluated under a 200ms latency target.
- Data: The experiments use approximately 30,000 hours of anonymized, hand-transcribed English speech comprising 43 million utterances representative of U.S. voice-search traffic.Additional data include artificially corrupted multi-style training examples and synthesized, noisy contact-name utterances with shallow-fusion biasing.
- Features: Inputs are 80-dimensional log-Mel features computed with a 25ms window, shifted every 10ms, stacked with two left-context frames, and downsampled to a 30ms frame rate.
- Model architecture: The shared encoder has eight LSTM layers with 2,048 hidden units and 640-dimensional projections, plus a factor-2 time-reduction layer after the second layer.
- Model architecture: The RNN-T decoder combines prediction and joint networks, while the LAS decoder uses four-head multihead attention followed by two LSTM layers with 2,048 hidden units and 640-dimensional projections.
- Model architecture: The RNN-T contains 114M parameters, and the additional second-pass LAS decoder contains 33M parameters.
- Latency measurement: Latency estimates assume 10GB/second CPU bandwidth and independently computed beam hypotheses, with fixed-beam latency proportional to K · H · N · Mdecoder.For adaptive beams, H · N is replaced by the number of lattice arcs; evaluation uses 295 frames, 28 target tokens, and a 33MB quantized LAS decoder under a 200ms target.
4. Results
Experiments compare LAS beam search and rescoring strategies, then evaluate latency, MWER training, and performance against a large conventional model. Rescoring with adaptive beams and MWER training provides the strongest reported trade-off, while side-by-side performance differences are statistically insignificant.
- 2nd Beam Search: LAS beam search improves over RNN-T on short utterances but degrades on long utterances.The comparison uses separately trained RNN-T-only and LAS-only baselines with fixed beam size H = 8.
- Rescoring: LAS rescoring improves WER on both short and long utterances and offers the best trade-off between quality across the two sets.Compared with beam search, rescoring takes a small short-utterance WER hit but performs much better on long utterances.
- Adaptive Beam: Adaptive-beam rescoring achieves similar WER to a fixed large beam while reducing first-pass computation cost.Increasing fixed beam size improves WER but proportionally increases computation; adaptive pruning avoids that full cost.
- Adaptive Beam: Restricting rescoring to four lattice hypotheses does not degrade accuracy, and adaptive-beam rescoring fits within the 200ms latency budget.The latency reduction comes from limiting the maximum number of arcs rescored at each step.
- MWER: 17% and 22% relative WER reductions are obtained on short and long utterances, respectively, after MWER-trained two-pass rescoring.MWER training improves rescoring WER by 8% relative on both utterance-length test sets.
- Comparison To Large Conventional Model: The two-pass model changes about 13% of traffic, with 48 wins, 61 losses, and 391 neutral outcomes versus the conventional model.The reported performance difference between the two systems is statistically insignificant.
5. Conclusions
The paper presents a two-pass E2E system that uses a LAS decoder to rescore first-pass RNN-T hypotheses. This reduces WER relative to RNN-T alone while adding less than 200ms of latency.
- 17% to 22% relative reduction in WER is achieved compared to RNN-T only, with latency increasing by less than 200ms.