Source-linked AI summary
Improving RNN Transducer Modeling for End-to-End Speech Recognition
Jinyu Li, Rui Zhao, Hu Hu, Yifan Gong
TL;DR
RNN-T offers streaming and avoids CTC’s frame-independence assumption, but its training memory demands and model footprint remain challenges. The paper reduces training memory and introduces more efficient model structures, yielding a 216 Megabyte model with lower WER than baseline and similarly sized hybrid systems. The results support compact, accurate RNN-T deployment, while reported latency depends on encoder lookahead and data-order choices affect accuracy.
Problem
RNN-T training consumes substantial memory, restricting minibatch size and slowing training, while compact deployment requires accurate models.
Method
The paper reduces memory through efficient network-output combination and reformulated gradients, and proposes layer-trajectory model structures that separate classification from temporal modeling.
Results
The 216 Megabyte best model achieves 6.6%, 7.5%, and 11.8% relative WER reduction on Cortana, Conversation, and DMA versus the baseline RNN-T, and similar WERs to a 5120 Megabyte server hybrid.
Takeaways & Limitations
The best RNN-T model is smaller than the baseline, outperforms a similarly sized hybrid model, and is suitable for device deployment.
Takeaways & Limitations
The ecltGRU encoder relies on 24 frames of lookahead, producing an average latency of (24+2)*30ms.
Abstract
from arXiv · showhide
In the last few years, an emerging trend in automatic speech recognition research is the study of end-to-end (E2E) systems. Connectionist Temporal Classification (CTC), Attention Encoder-Decoder (AED), and RNN Transducer (RNN-T) are the most popular three methods. Among these three methods, RNN-T has the advantages to do online streaming which is challenging to AED and it doesn't have CTC's frame-independence assumption. In this paper, we improve the RNN-T training in two aspects. First, we optimize the training algorithm of RNN-T to reduce the memory consumption so that we can have larger training minibatch for faster training speed. Second, we propose better model structures so that we obtain RNN-T models with the very good accuracy but small footprint. Trained with 30 thousand hours anonymized and transcribed Microsoft production data, the best RNN-T model with even smaller model size (216 Megabytes) achieves up-to 11.8% relative word error rate (WER) reduction from the baseline RNN-T model. This best RNN-T model is significantly better than the device hybrid model with similar size by achieving up-to 15.0% relative WER reduction, and obtains similar WERs as the server hybrid model of 5120 Megabytes in size.
1. INTRODUCTION
End-to-end ASR avoids the multiple intermediate models of hybrid systems, but existing approaches trade off streaming, modeling assumptions, and training complexity. The paper improves RNN-T training efficiency and model structures to achieve accurate, compact models.
- E2E ASR directly maps acoustic features to output tokens, avoiding separately trained intermediate acoustic, language, lexicon, and decision-tree models.
- AED is challenging to use for online streaming, while CTC is criticized for its frame-independence assumption.
- RNN-T combines acoustic, language-model, and decoding functions, removing CTC’s frame-independence assumption while naturally supporting online streaming.
- The paper reduces RNN-T training memory consumption for larger minibatches and proposes structures with better accuracy and smaller footprint than the baseline.
- Experiments use 30 thousand hours of anonymized, transcribed production data and evaluate the proposed models on several ASR tasks.
2. RNN-T
RNN-T uses encoder, prediction, and joint networks to transform acoustic features and previous target context into output-token posteriors. Its loss is the negative log posterior of the target sequence, computed with forward-backward training.
- The encoder converts acoustic features into high-level representations analogous to an acoustic model.
- The prediction network acts like an RNN language model, conditioning on the previous non-blank target to produce a high-level representation.
- The joint network combines encoder and prediction-network outputs through a feed-forward transformation.
- A linear output transform followed by softmax produces the posterior for each output token.
- RNN-T minimizes the negative log posterior of the output sequence given the acoustic input using a forward-backward algorithm.
- The forward and backward variables represent probabilities of producing target prefixes and suffixes across acoustic time and label positions, with blank transitions handled separately.
3. TRAINING IMPROVEMENT
RNN-T training uses substantial memory, especially when variable-length encoder and prediction outputs are combined through broadcasted tensors. The paper reduces this cost through sequence-wise combination and function merging, enabling larger minibatches.
- Memory challenge: RNN-T training requires large memory because intermediate tensors scale with label length, acoustic length, and output dimension.Broadcasting produces a four-dimensional tensor with size N × max(U_n) × max(T_n) × D.
- Efficient output combination: Sequence-wise combination reduces each utterance’s tensor to T_n × U_n × D before concatenating the results.The resulting two-dimensional representation has total size (Σ_n T_nU_n) × D.
- Efficient output combination: The sequence-wise implementation preserves training speed because no recurrent operation follows the encoder–prediction combination.Later operations receive sequence-length information to process concatenated tensors correctly.
- Training trade-off: Sorting utterances by acoustic and label lengths reduces memory waste but produces worse accuracy than randomized training utterances.The authors therefore emphasize data shuffling rather than fully sorted training sequences.
- Function merging: Merging function computations avoids storing unnecessary derivative and probability tensors during backpropagation.In-place softmax and storage reuse reduce this part from three large tensors to one.
- Training impact: For a 4,000-label model, the V100 minibatch size increases from 2000 to 4000; for a 36,000-label model, it increases from 500 to 2000.These increases follow the reported reduction in memory cost from the two optimization methods.
4. MODEL STRUCTURE EXPLORATION
The paper explores recurrent structures that separate temporal modeling from target classification and incorporate future context. It replaces the baseline projected layer-normalized LSTM with lighter GRU-based and layer-trajectory designs aimed at accuracy and footprint.
- Baseline and design goals: The baseline uses projected layer-normalized LSTMs in both the encoder and prediction networks.The proposed structures seek greater modeling power through task decoupling and future-context modeling.
- Baseline and design goals: Layer normalization defines recurrent transformations using adaptive gain and bias, with element-wise products applied to normalized vectors.The LSTM formulation includes input, output, and forget gates together with memory-cell updates.
- Layer trajectory LSTM: Layer trajectory LSTM uses time-LSTMs for temporal modeling and depth-LSTMs across layers for target classification.The depth-LSTMs scan outputs across layers at the current time without time recurrence.
- Contextual layer trajectory: Contextual layer trajectory LSTM replaces the preceding depth-LSTM output with a lookahead embedding to incorporate future context.Each layer contributes τ lookahead frames, yielding L × τ frames of total lookahead.
- Layer trajectory GRU: The proposed layer trajectory GRU performs temporal modeling with time-GRU and target classification across layers with depth-GRU.This separates the two functions within a lighter GRU-based recurrent structure.
- Contextual layer trajectory GRU: Elementwise contextual layer trajectory GRU uses lookahead embeddings while further reducing model footprint.The encoder’s contextual output is used as the RNN-T encoder representation.
5. EXPERIMENTS
Experiments compare RNN-T structures and hybrid systems across three ASR test sets, showing that future-context GRU models improve WER while retaining a small footprint. The best 216 Mb RNN-T matches server-hybrid WERs and outperforms a similarly sized device hybrid.
- Experimental setup: 30 thousand hours of anonymized, transcribed Microsoft production data support evaluation on Cortana, Conversation, and DMA test sets.The test sets contain mixed close-talk and far-field recordings.
- RNN-T models with greedy search: 17.5%, 10.5%, and 14.0% relative WER reductions result from cltLSTM with τ = 4 on Cortana, Conversation, and DMA, respectively.This encoder uses four future context frames at each layer but increases model size to 469 Mb.
- RNN-T models with greedy search: 11.2%, 10.9%, and 13.3% relative WER reductions result from 216 Mb ecltGRU versus the baseline on Cortana, Conversation, and DMA, respectively.The ecltGRU uses τ = 4 and is smaller than the 255 Mb baseline.
- RNN-T models with greedy search: The ecltGRU's WER decreases monotonically as future context τ increases, while τ = 3 or 4 yields similar WER and smaller τ lowers latency.At τ = 0, ecltGRU reduces to ltGRU.
- Comparison with hybrid models: 6.6%, 7.5%, and 11.8% relative WER reductions distinguish the best 216 Mb RNN-T from the baseline on Cortana, Conversation, and DMA with beam search.It matches the server hybrid's WERs and improves over the similarly sized 218 Mb device hybrid by 15.0%, 9.9%, and 11.3%, respectively.
- Comparison with hybrid models: The ecltGRU has about 2 input frames of average alignment delay versus about 10 for the baseline, but its average latency is 780 ms versus 300 ms.The paper attributes part of its accuracy advantage to this latency trade-off.
6. CONCLUSIONS
The paper improves RNN-T training efficiency and model structure to achieve better accuracy with a smaller footprint. The best model reduces WER across three test sets while using 216 MB.
- Efficient gradient computation reduces RNN-T training memory consumption and enables significantly larger minibatches.The approach combines encoder and prediction-network outputs efficiently and avoids storing multiple large tensors.
- Layer trajectory separates classification from temporal modeling through depth and time LSTM or GRU units.The best size–accuracy tradeoff uses ecltGRU in the encoder and GRU in the prediction network.
- 6.6%, 7.5%, and 11.8% relative WER reductions were achieved on Cortana, Conversation, and DMA, respectively, versus the baseline RNN-T.The best model has a 216 MB footprint.
- 15.0%, 9.9%, and 11.3% relative WER reductions were achieved versus the similarly sized hybrid model on Cortana, Conversation, and DMA, respectively.The model also obtains similar WERs to a 5120 MB server-size hybrid model.