Source-linked AI summary
On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
Jinyu Li, Yu Wu, Yashesh Gaur, Chengyi Wang, Rui Zhao, Shujie Liu
TL;DR
The paper addresses the lack of a comprehensive comparison of popular E2E speech-recognition models using large-scale training data. It compares RNN-T, RNN-AED, and Transformer-AED in streaming and non-streaming settings, finding Transformer-AED strongest overall and two streaming E2E models ahead of a highly optimized hybrid.
Problem
The paper identifies a lack of comprehensive comparisons of RNN-T, RNN-AED, and Transformer-AED across streaming and non-streaming modes using large training data.
Source: §1 · p. 1Method
The authors compare RNN-T, RNN-AED, and Transformer-AED in both modes, training the models on 65 thousand hours of Microsoft anonymized data.
Source: §1 · p. 1Results
Transformer-AED achieved the best WERs in both modes, and streaming RNN-T and Transformer-AED outperformed a highly optimized hybrid model.
Source: §5 · p. 4Takeaways & Limitations
CE pretraining is helpful even on a large-scale task, while chunk-based future context outperforms lookahead for streaming Transformer-AED.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Recently, there has been a strong push to transition from hybrid models to end-to-end (E2E) models for automatic speech recognition. Currently, there are three promising E2E methods: recurrent neural network transducer (RNN-T), RNN attention-based encoder-decoder (AED), and Transformer-AED. In this study, we conduct an empirical comparison of RNN-T, RNN-AED, and Transformer-AED models, in both non-streaming and streaming modes. We use 65 thousand hours of Microsoft anonymized training data to train these models. As E2E models are more data hungry, it is better to compare their effectiveness with large amount of training data. To the best of our knowledge, no such comprehensive study has been conducted yet. We show that although AED models are stronger than RNN-T in the non-streaming mode, RNN-T is very competitive in streaming mode if its encoder can be properly initialized. Among all three E2E models, transformer-AED achieved the best accuracy in both streaming and non-streaming mode. We show that both streaming RNN-T and transformer-AED models can obtain better accuracy than a highly-optimized hybrid model.
1. Introduction
The paper compares RNN-T, RNN-AED, and Transformer-AED in streaming and non-streaming modes using 65 thousand hours of training data. It also examines ways to improve streaming models and reports that single streaming E2E models can surpass a state-of-the-art hybrid model.
1. Introduction
- E2E systems map speech directly to output tokens with one network, unlike hybrid systems that optimize separate components such as acoustic and language models.The introduction situates this comparison within a shift from hybrid to E2E ASR.Source: §1 · p. 1
- The study compares RNN-T, RNN-AED, and Transformer-AED in streaming and non-streaming modes, training all models on 65 thousand hours of Microsoft anonymized data.The authors describe this as a detailed comparison not previously conducted at this scale.Source: §1 · p. 1
- A single streaming RNN-T or Transformer-AED model can surpass a state-of-the-art hybrid model.The paper contrasts this with earlier work combining RNN-T and RNN-AED in a two-pass decoding configuration.Source: §1 · p. 1
- The study proposes multi-layer context modeling, finds CE initialization more effective than CTC initialization for RNN-T, and favors chunk-based future context over lookahead for streaming Transformer-AED.It also releases Transformer-related code with reproducible Librispeech results.Source: §1 · p. 1
2. Popular End-to-End Models
RNN-T and AED differ in how they condition and synchronize output generation, while RNN-AED and Transformer-AED use different encoder-decoder building blocks. The section outlines these architectures and when RNN-T can operate in streaming mode.
2. Popular End-to-End Models
- RNN-T conditions each next label on previous label outputs and generates frame-synchronously, whereas AED also conditions on acoustics and generates label-synchronously.Both model families use an acoustic encoder and an autoregressive decoder.Source: §2 · pp. 1–2
2.1. RNN transducer
- RNN-T’s prediction network consumes previous non-blank targets, and its joint network combines prediction and encoder representations to produce softmax outputs.With a unidirectional LSTM encoder, RNN-T works in streaming mode by default; a bidirectional encoder makes it non-streaming.Source: §2.1 · p. 2
2. Popular End-to-End Models
- RNN-AED and Transformer-AED use LSTM-RNN and Transformer components, respectively, in their encoder and decoder.RNN-T has attracted industrial interest for streaming, while AED’s attention structure has drawn academic research.Source: §2.2 · p. 2
2.2.1. RNN-AED
- RNN-AED uses an attention-enhanced decoder whose context vector is a weighted combination of encoder outputs carrying acoustic information for the next token.Its encoder can share the RNN-T structure.Source: §2.2.1 · p. 2
2.2.2. Transformer-AED
- Transformer-AED uses Transformer blocks with multi-head self-attention and feed-forward layers; its decoder also attends over encoder outputs.Residual connections and layer normalization are used to connect layers and blocks.Source: §2.2.2 · p. 2
3. Our Models
The models use distinct recurrent and Transformer building blocks, with streaming variants tailored to each architecture. The streaming designs include future-context modeling, monotonic attention, and chunk-based or lookahead processing.
3.1. Model building block
- The model blocks include cuDNN LSTMs with projection and layer normalization, custom LSTMs for RNN-T, and a Transformer-AED with convolutional preprocessing and Pre-LN.Custom RNN-T LSTMs place layer normalization and projection inside the LSTM; this slows training by 50%.Source: §3.1 · p. 2
3.2. Non-streaming models
- Non-streaming RNN-T and RNN-AED use bidirectional LSTM encoders; RNN-AED pairs its encoder with a unidirectional decoder and location-aware attention.The RNN-AED configuration does not use multi-task training or joint CTC decoding.Source: §3.2 · p. 2
3.3. Streaming models
- Streaming RNN-T uses a unidirectional encoder, and the study compares CTC and cross-entropy encoder initialization.The authors also investigate custom LSTM blocks for RNN-T.Source: §3.3 · p. 3
- The proposed context modeling adds future context to LSTM blocks with only a slight increase in model parameters and is applied at every block.Unlike the cited lookahead-convolution approach, which applies context modeling only at the top layer, this study applies it to every LSTM block.Source: §3.3 · p. 3
- Streaming RNN-AED uses MoChA, which triggers on encoder states from left to right and applies softmax attention over a lookback window.Because trigger sampling precludes standard backpropagation, training uses expected context vectors.Source: §3.3 · p. 3
- Streaming Transformer-AED uses CTC frame-synchronized decoding to select candidates for attention reranking, comparing chunk-based context with fixed-window lookahead.The chunk-based decoder can see the end of a chunk; lookahead uses a fixed window.Source: §3.3 · p. 3
4. Experiments
Across 13 test scenarios, the experiments compare E2E models trained on 65 thousand hours of anonymized Microsoft data. Streaming results include models that outperform a strong hybrid baseline, while non-streaming AED models outperform RNN-T.
4. Experiments
- The evaluation averages WER over 13 application-scenario test sets containing 1.8 million words; all models use 65 thousand hours of anonymized transcribed Microsoft data.The data have personally identifiable information removed.Source: §4 · p. 3
4.1. Non-streaming E2E models
- Non-streaming AED models have a clear advantage over non-streaming RNN-T, and Transformer-AED improves on RNN-AED by 2.7% relative WER reduction.The comparison uses models with about 87 million parameters and a shared 4,000-word-piece output vocabulary.Source: §4.1 · p. 3, §4 · p. 3
4.2. Surpassing hybrid model with streaming E2E models
- The hybrid comparator is a highly optimized cltLSTM with 65 million parameters and 480 ms encoder latency; 24 frames of future context yield an 18.7% relative WER reduction over its CE baseline.The model uses a three-stage optimization process and a 5 GB 5-gram decoding graph.Source: §4.2 · p. 4
- 12.4% relative WER reduction: distributing four lookahead frames across each of six RNN-T encoder blocks yields 10.65% WER, versus 12.16% without lookahead.The distributed context modeling also outperforms placing 24 lookahead frames only on the top RNN block, which gives 11.19% WER.Source: §4.2 · p. 4
- 8.0% relative WER reduction: CE initialization lowers streaming RNN-T WER to 9.80% compared with random initialization.The authors report that CE initialization enables the encoder to learn time alignment; it requires alignments, approximated for word pieces by evenly segmenting each word’s duration.Source: §4.2 · p. 4
- 9.27% WER: the streaming RNN-T with a Custom LSTM and context modeling is slightly better than the best hybrid model.The Custom LSTM places projection and layer normalization inside the cell, allowing more memory cells at similar model size to the cuDNN setup.Source: §4.2 · p. 4
- 9.61% versus 9.80% WER: MoChA-based streaming RNN-AED slightly outperforms an encoder-matched RNN-T without requiring initialization.The authors describe this as the first streaming RNN-AED to outperform RNN-T on a large-scale task.Source: §4.2 · p. 4
- 9.16% WER: streaming Transformer-AED with chunk-based future context outperforms lookahead at 720 ms average latency, without extra decoder latency.The authors attribute the difference mainly to the lookahead approach’s bottom Transformer blocks not receiving the full advantages of right context.Source: §4.2 · p. 4, Table 2 · p. 4
5. Conclusions
Across the compared models, Transformer-AED achieved the best WERs in streaming and non-streaming modes. AED outperformed RNN-T with the same encoder structure, while customized LSTM and cross-entropy initialization made RNN-T better than RNN-AED; both streaming RNN-T and Transformer-AED outperformed a highly optimized hybrid model.
5. Conclusions
- Transformer-AED achieved the best WERs in both streaming and non-streaming modes; with the same encoder structure, AED beat RNN-T, while customized LSTM and CE initialization made RNN-T better than RNN-AED.The comparison used 65K hours of Microsoft internal anonymized data.Source: §5 · p. 4
- Both streaming RNN-T and Transformer-AED outperformed a highly optimized hybrid model.Source: §5 · p. 4
- Streaming RNN-T context modeling cut WER 12.4% relative versus no lookahead; CE initialization cut it 8.0% relative versus random initialization; chunk-based Transformer-AED future context beat lookahead by 10.7% relative.The text describes the CE initialization gain as evidence that pretraining helps even on a large-scale task.Source: §5 · p. 4