Source-linked AI summary
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, Shankar Kumar
TL;DR
Streaming speech recognition needs models that handle temporal dependencies without the growing computation and label-synchronous decoding challenges of standard Transformer systems. The paper introduces Transformer Transducer, which replaces RNN-T’s audio and label encoders with independently self-attending Transformer encoders trained with RNN-T loss. Full attention achieves state-of-the-art LibriSpeech accuracy, while restricted context enables streaming and limited future context narrows the accuracy gap.
Problem
Standard Transformer sequence-to-sequence decoding is label-synchronous and self-attention can require increasing computation, challenging streaming speech recognition.
Method
Transformer Transducer replaces RNN-T’s RNN-based audio and label encoders with Transformer self-attention and uses RNN-T loss to marginalize alignments for timesynchronous decoding.
Results
Full-attention Transformer Transducer achieves new state-of-the-art LibriSpeech accuracy, while limited context supports streaming with a flexible accuracy-latency trade-off.
Takeaways & Limitations
Limiting audio and label context makes Transformer Transducer suitable for streaming, and limited future-frame attention can bridge full- and limited-attention performance.
Takeaways & Limitations
Experiments use the RNN-T loss, which performed similarly to the monotonic RNN-T loss in the authors’ experiments.
Abstract
from arXiv · showhide
In this paper we present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks based on self-attention are used to encode both audio and label sequences independently. The activations from both audio and label encoders are combined with a feed-forward layer to compute a probability distribution over the label space for every combination of acoustic frame position and label history. This is similar to the Recurrent Neural Network Transducer (RNN-T) model, which uses RNNs for information encoding instead of Transformer encoders. The model is trained with the RNN-T loss well-suited to streaming decoding. We present results on the LibriSpeech dataset showing that limiting the left context for self-attention in the Transformer layers makes decoding computationally tractable for streaming, with only a slight degradation in accuracy. We also show that the full attention version of our model beats the-state-of-the art accuracy on the LibriSpeech benchmarks. Our results also show that we can bridge the gap between full attention and limited attention versions of our model by attending to a limited number of future frames.
1. INTRODUCTION
Transformer-based sequence-to-sequence models create streaming challenges because decoder attention is label-synchronous and self-attention can require growing computation. The paper addresses these challenges by replacing RNN-T encoders with Transformers while restricting context to preserve streamability.
- Motivation: Transformer sequence-to-sequence models attend over encoder features using decoder features, requiring label-synchronous decoding that challenges streaming speech recognition.Self-attention computation also becomes increasingly costly as the processed sequence grows.
- Motivation: RNNs have been the de facto streaming choice because they model temporal dependencies while maintaining constant computation for each audio frame.RNN-T, RNA, and Neural Transducer architectures use RNN encoders for audio and labels and support frame-synchronous decoding.
- Paper approach: The paper replaces RNN-based audio and label encoders with Transformer encoders using fixed past-frame and previous-label context to retain streamability.This restriction causes some performance degradation but gives constant computation per frame, while self-attention improves training time and resource utilization relative to RNN-T models with RNN encoders.
- RNN-T background: The RNN-T architecture predicts a distribution over labels, including a null label, at every time step while conditioning on previous label history.Unlike typical attention-based sequence-to-sequence models, RNN-T aligns acoustic features and labels through frame-synchronous predictions.
- RNN-T background: RNN-T training marginalizes the conditional probability over all valid alignments between input frames and output labels.The alignment sequence contains label-time pairs; removing blank labels yields the target output sequence.
2. TRANSFORMER TRANSDUCER
The Transformer Transducer replaces the conventional RNN-T encoders with Transformer encoders while retaining the RNN-T alignment and loss framework. Its attention design supports streaming by restricting context, while relative positional encoding and moving windows address inference cost and latency.
- Architecture: The Transformer Transducer replaces the conventional RNN-T audio and label encoders with Transformer encoders.The joint network combines encoder outputs and applies a feed-forward network with a softmax layer to produce label probabilities.
- RNN-T loss: RNN-T training marginalizes probabilities over valid alignments and computes the sequence likelihood with a forward algorithm.The forward variable α(t, u) sums probabilities of paths ending at a time-frame and label position, with α(T, U) corresponding to P(y|x).
- Transformer encoder: The Transformer encoder uses stacked layers containing multi-headed self-attention and feed-forward sub-layers with normalization, residual connections, and dropout.Attention masks can restrict conditioning to previous frames or labels.
- Streaming design: AudioEncoder and LabelEncoder operate independently, because label-encoder states do not attend to audio-encoder states.This preserves the streaming-oriented separation of audio and label processing used in the RNN-T framework.
- Streaming design: With relative positional encoding, one-step inference costs O(t), but attending to a moving window W makes the complexity constant.Absolute positional encoding prevents state reuse beyond the training length and yields O(t^2) complexity.
3. EXPERIMENTS AND RESULTS
Experiments on LibriSpeech evaluate Transformer Transducer accuracy and streaming trade-offs under limited audio and label attention. Full attention outperforms the LSTM-based RNN-T baseline, while restricted context enables tractable streaming with controllable latency and accuracy costs.
- 3.1. Data: The evaluation uses the LibriSpeech ASR corpus, with paired audio/transcript data for Transformer Transducer and LSTM-based baseline training.The corpus contains 970 hours of audio and transcripts, plus an 800M-word text-only dataset for language-model training.
- 3.3. Results: Full-attention Transformer Transducer models significantly outperform the bidirectional-LSTM RNN-T baseline and are competitive with similarly sized wordpiece end-to-end systems.The comparison is reported on LibriSpeech test sets, including systems with and without shallow-fusion language models.
- 3.3. Results: Limiting audio self-attention to a fixed left context makes one-step streaming inference computationally tractable, with more audio history yielding lower WER.Experiments use up to 10 previous frames per layer for streamable models.
- 3.3. Results: Adding limited future audio context narrows the gap between streamable and full-attention models, but aggregates latency across Transformer layers.Six right-context frames per layer are around 16% worse than full attention, while two frames provide around 30% improvement over the streamable model at around 1 second latency.
- 3.3. Results: Using only three previous label states per layer achieves similar accuracy to using 20 states, indicating that the label encoder needs little left context.The same trend appears when label context is limited alongside a full-attention audio encoder.
- 3.3. Results: The streaming configuration combines limited left audio context with look-ahead to trade recognition accuracy against latency.A 10-frame left context reduces one-step inference time complexity to a constant.
4. CONCLUSIONS
The paper presents Transformer Transducer as an RNN-T model with Transformer-based audio and label encoders. It reports state-of-the-art LibriSpeech accuracy, streaming suitability through limited context, faster training, and flexible accuracy–latency trade-offs.
- 4. CONCLUSIONS: Transformer Transducer embeds Transformer self-attention for audio and label encoding within the RNN-T architecture.It retains an end-to-end objective that efficiently marginalizes over alignments and supports time-synchronous decoding.
- 4. CONCLUSIONS: The model achieves new state-of-the-art accuracy on the LibriSpeech benchmark and can support streaming recognition by limiting audio and label context.The conclusion presents limited self-attention context as the mechanism for adapting the model to streaming use.
- 4. CONCLUSIONS: Transformer Transducer models train significantly faster than LSTM-based RNN-T models and allow flexible recognition accuracy–latency trade-offs.The reported trade-off follows from varying the context used in self-attention.