Source-linked AI summary
Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition
Anshuman Tripathi, Jaeyoung Kim, Qian Zhang, Han Lu, Hasim Sak
TL;DR
Long-form rescoring can add unbounded latency while providing limited accuracy gains. The paper trains a variable-context Transformer Transducer for parallel low- and high-latency decoding; it reports a 20% relative word error rate improvement for voice-search and very similar accuracy to full audio context with 2.4 seconds of lookahead.
Problem
For long-form audio, n-best or lattice rescoring adds unbounded latency and provides limited accuracy gains because of low diversity among hypotheses.
Source: §1 · p. 1Method
A single Transformer Transducer trained with variable right context runs parallel low- and high-latency decoding branches.
Source: §1 · p. 1, §2.3 · p. 2Results
Takeaways & Limitations
The model can provide streaming partial results and replace them with higher-quality final results with small additional end-of-utterance latency.
Source: §2.3 · p. 2, §5 · p. 5
Abstract
from arXiv · showhide
In this paper we present a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. The model is composed of a stack of transformer layers for audio encoding with no lookahead or right context and an additional stack of transformer layers on top trained with variable right context. In inference time, the context length for the variable context layers can be changed to trade off the latency and the accuracy of the model. We also show that we can run this model in a Y-model architecture with the top layers running in parallel in low latency and high latency modes. This allows us to have streaming speech recognition results with limited latency and delayed speech recognition results with large improvements in accuracy (20% relative improvement for voice-search task). We show that with limited right context (1-2 seconds of audio) and small additional latency (50-100 milliseconds) at the end of decoding, we can achieve similar accuracy with models using unlimited audio right context. We also present optimizations for audio and label encoders to speed up the inference in streaming and non-streaming speech decoding.
1. INTRODUCTION
The introduction describes the latency and accuracy limits of rescoring long-form speech, then proposes parallel decoding with different right contexts and a single variable-context model. The approach aims to provide low-latency streaming results and more accurate final results with bounded additional latency.
1. INTRODUCTION
- For long-form audio, n-best or lattice rescoring adds unbounded latency and yields limited accuracy gains because hypotheses lack diversity.Source: §1 · p. 1
- 20% relative word error rate improvement over the low-latency results is reported for the voice-search task.The larger-right-context decoding adds bounded latency because its right context is limited.Source: §1 · p. 1
- Parallel transformer decoding uses smaller right context for low-latency streaming results and larger right context for final results, working for short and long utterances.Source: §1 · p. 1
- A single model trained with variable right context can decode in either low-latency or high-latency mode.The paper describes efficient parallel decoding in both modes using the same model.Source: §1 · p. 1
2. TRANSFORMER TRANSDUCERS
This section describes the Transformer Transducer architecture and variable-right-context training, then explains parallel decoding and constrained alignment training.
2.1. Architecture
- The Transformer Transducer combines transformer audio and label encoders with a joint network to predict label probabilities at each time step.The model is trained with end-to-end RNN-T loss.Source: §2.1 · pp. 1–2
2.2. Variable Context Training
- Variable-context training samples among configured right-context patterns, allowing inference with any right-context configuration used during training.The paper reports that independently sampling each layer’s context led to unstable training, so it limits the available configurations.Source: §2.2 · p. 2
2.3. Y-model
- The Y-model uses zero-right-context initial layers and variable-context final layers, with parallel branches using smaller and larger right contexts.Only the variable-context final layers need recomputation for the two branches.Source: §2.3 · p. 2
2.3.1. Inference and Recognition Latency
- During inference, low-latency partial results are replaced at utterance end by high-latency results computed using the remaining lookahead audio.The remaining context can be batched rather than processed while waiting for streaming audio.Source: §2.3.1 · p. 2
2.4. Constrained Alignment Training
- Constrained alignment training masks high-delay paths from the RNN-T loss using word-boundary windows based on reference alignments.The full-attention non-streaming Transformer Transducer supplies the reference alignments.Source: §2.4 · p. 2
3. INFERENCE OPTIMIZATIONS
The inference optimizations target streaming and non-streaming audio encoding and label decoding to make parallel decoding feasible for the Y-model.
3.1. Streaming Transformer Audio Encoder
- Batch step inference encodes multiple time steps in parallel because Transformer audio encoding has no time dependency.Streaming encoding takes audio frames and previous states as input and outputs encoded features and next states.Source: §3.1 · p. 3
3.2. Non-streaming Transformer Audio Encoder
- Query slicing processes a few query blocks at a time, each attending to limited keys, keeping memory consumption constant.It addresses the Transformer's O(T^2) attention-matrix memory cost in non-streaming inference.Source: §3.2 · p. 3
3.3. Decoder Optimizations
- The label encoder represents predicted label history; the paper tests limited-context Transformers and bigram embeddings, whose lookup runtime is very fast.The label encoder runs for every hypothesis, so its computational complexity strongly affects decoding speed.Source: §3.3 · p. 3
- A cache reuses label-encoder outputs for label contexts encountered repeatedly across decoding hypotheses.The same label history may be encoded multiple times during decoding.Source: §3.3 · p. 3
4. EXPERIMENTS AND RESULTS
On voice-search experiments, Y-Model2’s 2.4-second-lookahead mode approached full-attention performance, while constrained alignment reduced timing delay with a low-latency quality trade-off. The section also reports label-decoder and inference-speed benchmarks, using 30K hours of training speech and a 14K-utterance test set.
4.1. Data
- Experiments use 30K hours of voice-search training speech and a 14K-utterance test set of utterances shorter than 5.5 seconds.The speech data are anonymized and hand-transcribed.Source: §4.1 · p. 3
4.2. Limited Context Decoding
- Reducing label context to 3 graphemes greatly speeds decoding over 40 labels without much WER difference; context 2 is faster still because it is a lookup table.The reported speed gain for 3 graphemes is attributed to caching label-encoder outputs.Source: §4.2 · p. 3
4.3.1. Y-Model1
- The Y-model experiments use 20-layer audio encoders with variable right-context configurations; Y-Model2 keeps its first 15 layers at zero right context and varies the final five.Y-Model1 instead trains all 20 layers with variable context; both model descriptions specify a four-frame output delay.Source: §4.3.2 · p. 3, §4.3.1 · p. 3, §4.3 · p. 3
4.3.2. Y-Model2
- With 2.4-second lookahead, Y-Model2 is very close to full-attention performance, while its low-latency mode is much more accurate than the best no-right-context streaming model.The reported comparison is for baselines without constrained alignment training.Source: Table 2 · p. 4, §4.3.2 · p. 4
- Constrained alignment cuts delay from 767 to 119 msec in low-latency mode and from 742 to 74 msec in high-latency mode, with low-latency WER degradation but no high-latency WER loss.The authors attribute the low-latency degradation to reduced lookahead frames.Source: Table 3 · p. 4, §4.3.2 · p. 4
4.4. Inference Benchmarks
- For 100 seconds of audio in high-latency TPU encoding, training, query-slicing, and batch-step modes take 0.3, 0.6, and 1.8 seconds, respectively.Encoding more time steps per inference run is faster; batch-step mode is slowest because of query-block and state-handling overhead.Source: Figure 3 · p. 4, §4.4 · p. 4
- A 36-minute audio file is recognized in under 3 minutes on 8 desktop CPU cores, with less than 8% real-time factor; bigrapheme lookup can further speed decoding with slight WER regression.The CPU benchmark uses query-slicing inference.Source: Figure 4 · p. 4, §4.4 · pp. 4–5
5. CONCLUSIONS
The paper presents a Transformer-Transducer approach for one model to support low- and high-latency speech decoding. With limited future context and parallel decoding optimizations, it reports accuracy similar to full-context models with very small additional latency.
5. CONCLUSIONS
- With 2.4 sec of future audio context, the model achieves very similar accuracy to a model with full audio context.The Y-model uses variable right context at penultimate layers.Source: §5 · p. 5
- A Transformer-Transducer training method allows one model to decode speech in both low- and high-latency modes.The low-latency mode provides streaming results, while the high-latency mode provides final results.Source: §5 · p. 5
- Parallel audio-frame encoding and decoder optimizations can make the high-latency mode's additional end-of-decoding latency very small.Source: §5 · p. 5