Source-linked AI summary
Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, Jinyu Li
TL;DR
Transformer inference is computationally expensive for low-latency streaming speech recognition. The paper combines Transformer-XL with chunk-wise processing to build streamable Transformer and Conformer Transducers, which outperform the compared streaming models while achieving competitive runtime with limited look-ahead.
Problem
Transformer Transducers have high inference cost that grows with input sequence length, limiting their practical use for real-time streaming recognition.
Method
The paper combines Transformer-XL and chunk-wise streaming processing, using attention masks to truncate history and allow limited future information.
Results
Transformer and Conformer Transducers outperform hybrid, RNN-T, and streamable Transformer encoder-decoder models in streaming accuracy, with competitive RTF given small latency.
Takeaways & Limitations
With limited history and small look-ahead, the proposed models retain performance while reducing runtime cost for real-time streaming recognition.
Takeaways & Limitations
Training uses non-overlapping chunks to guarantee training efficiency.
Abstract
from arXiv · showhide
Recently, Transformer based end-to-end models have achieved great success in many areas including speech recognition. However, compared to LSTM models, the heavy computational cost of the Transformer during inference is a key issue to prevent their applications. In this work, we explored the potential of Transformer Transducer (T-T) models for the fist pass decoding with low latency and fast speed on a large-scale dataset. We combine the idea of Transformer-XL and chunk-wise streaming processing to design a streamable Transformer Transducer model. We demonstrate that T-T outperforms the hybrid model, RNN Transducer (RNN-T), and streamable Transformer attention-based encoder-decoder model in the streaming scenario. Furthermore, the runtime cost and latency can be optimized with a relatively small look-ahead.
1. INTRODUCTION
The paper targets real-time Transformer Transducer speech recognition, addressing Transformer inference cost and streaming latency while balancing accuracy, training cost, and runtime cost. It combines Transformer-XL with chunk-wise processing and reports strong streaming accuracy and efficiency.
- Motivation: Transformer inference cost grows with input sequence length, obstructing practical use of Transformer Transducers for streaming recognition.Streaming recognition and runtime computational cost are identified as crucial deployment factors.
- Motivation: Existing approaches trade off latency, accuracy, or Transformer parallelism through restricted context, chunking, or memory-based processing.Time restriction can introduce latency, chunking can reduce accuracy, and memory-based methods can break parallel computation.
- Approach: The proposed streaming T-T and C-T models combine Transformer-XL with chunk-wise processing without overlapping training chunks.The design aims to balance training cost, runtime cost, and accuracy.
- Evaluation and Results: Over 65 thousand hours of anonymized data were trained in 2 days using mixed precision on 32 V100 GPUs.This demonstrates the reported training setup and scale.
- Evaluation and Results: Over 10% relative word error rate improvement was reported against hybrid, RNN-T, and streaming Transformer attention-based encoder-decoder models.The comparison was conducted on streaming speech recognition evaluation.
- Evaluation and Results: With 360ms look-ahead, T-T reaches 0.25 real-time factor on CPU while limited history maintains performance with 1% WERR degradation.The reported runtime result satisfies the stated industry requirement for a real application.
2. MODEL STRUCTURE
The model uses a transducer architecture with an encoder, predictor, and joint network, replacing the usual recurrent encoder with Transformer or Conformer components. Its attention mechanisms model sequence order and context, while causal convolution supports streaming.
- Transducer Architecture: A transducer combines an acoustic encoder, label predictor, and joint network to produce sentence-piece probability distributions.Encoder and predictor outputs are combined before nonlinear transformation and softmax.
- Transducer Architecture: This paper uses Transformer as the encoder and LSTM as the predictor to address speed and memory considerations.The resulting architecture is a Transformer Transducer rather than an all-LSTM RNN-T.
- Transformer and Conformer: Transformer self-attention computes sequence representations through query, key, and value projections, with masks controlling the usable input range.Multi-head attention applies multiple parallel self-attention operations.
- Transformer and Conformer: Relative position embedding represents frame offsets in attention and is implemented through a lookup table for efficiency and memory savings.The paper reports this implementation as more efficient and memory friendly than the cited alternative.
- Transformer and Conformer: Conformer inserts a CNN-based structure into each Transformer block to improve local information modeling.The implementation changes the depth-wise CNN to a causal version to avoid extra latency.
3. STREAMING TRANSFORMER-TRANSDUCER IN REAL-TIME
The paper addresses Transformer Transducer streaming by truncating history and limiting future context through chunk-wise attention masking. Inference caching and chunk-wise computation further reduce repeated work and can trade modest latency for substantial speedup.
- Streaming challenges: Full-history attention makes Transformer Transducer inference computationally quadratic for long utterances, motivating truncated history and limited future context.The cost of computing each frame’s attention grows linearly with its position in the utterance.
- Attention mask design: The mask segments acoustic features into chunks where frames within a chunk attend to one another, while earlier chunks cannot attend to later chunks.This design permits bounded future context while preserving access to historical information across layers.
- Attention mask design: The strategy’s average look-ahead is half the chunk size, so frames near a chunk boundary use less future context than earlier frames in the same chunk.For chunk size 3, the example includes up to two future frames for the first frame and none for the last.
- Attention mask design: The mask allows left context to increase linearly with transformer depth while keeping the right context bounded across layers.In the example, the left receptive field grows 3 frames per layer while the right context remains restricted to x12.
- Inference optimizations: Caching stores intermediate key and value variables to avoid recomputing them for later transformer encoder outputs.The cached variables are used in subsequent attention calculations.
- Inference optimizations: Grouping frames into chunks enables simultaneous encoder computation, and several frames of introduced latency can produce significant speedup even with zero look-ahead.Chunk-wise computation applies efficient matrix operations instead of repeated matrix-vector operations.
4. EXPERIMENT
Experiments evaluate streaming transducer models on 65K hours of anonymized Microsoft data across 13 application scenarios, focusing on accuracy, runtime, latency, and INT8 optimization. T-T and C-T improve streaming accuracy, while chunking, truncated history, lookahead, and quantization expose accuracy–latency–speed trade-offs.
- Experimental Setup: The evaluation uses 65K hours of transcribed training data and 1.8M test words spanning 13 application scenarios.WER is averaged across all test scenarios; the data are anonymized and use a 4000-piece vocabulary.
- Zero Look-Ahead: In zero-lookahead decoding, T-T and C-T significantly outperform RNN-T in accuracy, although full-context transformer inference has RTF above 1.Truncated history preserves nearly the same performance while substantially reducing RTF and memory consumption.
- Zero Look-Ahead: Batching improves decoding speed: T-T reaches RTF below 1 with two frames and RTF 0.2 with batch size 15, but the latter adds 450ms latency.Two frames correspond to 60ms latency; transformer batching is more suitable than RNN-T batching.
- Small Look-Ahead: With small lookahead, T-T and C-T provide strong accuracy at acceptable runtime cost and outperform corresponding streamable S2S models.T-T with small lookahead is reported to perform very close to the full-utterance T-T model.
- 8-bit Optimization: INT8 quantization yields 3.6 times speedup for RNN-T without WER loss, while T-T and C-T achieve about 2 times speedup with slight WER degradation.The comparison uses one CPU thread; the paper suggests float-precision transformer softmax as one possible explanation for the difference.
5. CONCLUSION
The paper develops streaming T-T and C-T models by combining Transformer-XL with chunk-wise processing for real-time speech recognition. In streaming scenarios, they outperform the compared hybrid, RNN-T, and streamable Transformer AED models in accuracy while achieving comparable or better RTF than RNN-T with small latency.
- Conclusion: Streaming T-T and C-T combine Transformer-XL with chunk-wise processing to support real-time speech recognition.The design aims to avoid latency growing linearly with the number of transformer layers.
- Conclusion: T-T and C-T outperform hybrid, RNN-T, and streamable Transformer AED models in streaming accuracy.With small latency, they can achieve comparable or better RTF than RNN-T.