Source-linked AI summary
Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kalgaonkar, Yongqiang Wang, Duc Le, Mahaveer Jain, Kjell Schubert, Christian Fuegen, Michael L. Seltzer
TL;DR
The paper addresses how to improve neural-transducer speech recognition while retaining compactness and streaming capability. It combines a causal VGG-Transformer encoder with truncated self-attention, achieving strong LibriSpeech results with linear complexity and a small system footprint.
Problem
The paper seeks improved neural-transducer speech recognition using Transformer networks while retaining practical footprint, accuracy, and efficiency.
Method
It uses causal VGG convolution for positional information and frame-rate reduction, then truncated self-attention to support streaming and reduce computational complexity.
Results
The Transformer-Transducer achieved word error rates of 6.37 % on test-clean and 15.30 % on test-other with 45.7 M parameters for the entire system.
Takeaways & Limitations
The proposed Transformer-Transducer is reported as accurate, streamable, compact, and efficient for resource-limited scenarios such as on-device speech recognition.
Abstract
from arXiv · showhide
We explore options to use Transformer networks in neural transducer for end-to-end speech recognition. Transformer networks use self-attention for sequence modeling and comes with advantages in parallel computation and capturing contexts. We propose 1) using VGGNet with causal convolution to incorporate positional information and reduce frame rate for efficient inference 2) using truncated self-attention to enable streaming for Transformer and reduce computational complexity. All experiments are conducted on the public LibriSpeech corpus. The proposed Transformer-Transducer outperforms neural transducer with LSTM/BLSTM networks and achieved word error rates of 6.37 % on the test-clean set and 15.30 % on the test-other set, while remaining streamable, compact with 45.7M parameters for the entire system, and computationally efficient with complexity of O(T), where T is input sequence length.
1. INTRODUCTION
End-to-end speech recognition simplifies the traditional hybrid pipeline, but the paper seeks further improvements to RNN-T using Transformer networks while preserving practical deployment advantages.
- Motivation: Hybrid systems require separate components and substantial human intervention, including acoustic units, vocabulary, and pronunciation modeling.They can also require higher computational complexity and memory consumption, complicating deployment in resource-limited scenarios.
- Motivation: End-to-end approaches directly model the transformation from audio to word tokens, simplifying training and achieving promising recognition accuracy.
- Motivation: RNN-T offers promising potential in model footprint, accuracy, and efficiency among end-to-end speech-recognition approaches.
- Motivation: RNNs are difficult to compute in parallel because each state depends on the previous state and fixed-length states condense historical information.
- Contribution: The paper applies Transformer networks to neural transducers, using causal VGG convolution for contextual information and frame-rate reduction, plus truncated self-attention for streaming and lower complexity.
2. NEURAL TRANSDUCER (RNN-T)
Neural transducers model variable-length audio-to-text sequences by combining an encoder with a predictor, while Transformer encoders provide a non-recurrent alternative to recurrent architectures.
- Sequence-to-sequence modeling: Speech recognition is a sequence-to-sequence task in which input and output sequence lengths can vary.
- CTC and RNN-T: CTC introduces a blank label and represents each target sequence through possible paths whose blanks and repeated symbols collapse to the target.
- CTC and RNN-T: Unlike CTC, RNN-T conditions each output distribution on the input sequence and previous non-blank output symbols.
- Neural transducer: An RNN-T encoder maps the acoustic sequence to encoded representations, while its predictor encodes previous non-blank output history.
- Neural transducer: The paper calls the architecture a neural transducer because Transformer encoders are not recurrent, unlike the recurrent encoders commonly used in RNN-T.
3. TRANSFORMER
Transformer encoders use attention and feed-forward processing instead of recurrence, enabling parallel sequence encoding and access to broader input context through self-attention.
- Attention: Attention models relationships among contextual signals using queries, keys, and values to produce higher-level sequence representations.
- Self-attention: Self-attention sets Q = K = V and encodes an input sequence without recurrent connections between time steps.
- Self-attention: Self-attention can generate encodings efficiently in parallel and attend to all available contexts rather than conditioning on fixed recurrent states.
- Multi-head attention: Multi-head attention splits input dimensions into multiple projected chunks, processes them independently, and concatenates their encodings into the output.
- Transformer encoder: A Transformer encoder contains attention, feed-forward, and layer-normalization blocks, with the feed-forward block projecting features to another dimension and back.
4. TRANSFORMER-TRANSDUCER
The paper applies Transformer networks to neural transducers using causal VGG convolutions for positional information and frame-rate reduction, plus truncated self-attention for streaming and lower complexity.
- 4. TRANSFORMER-TRANSDUCER: The proposed design combines causal convolution and truncated self-attention to improve Transformer-based neural transducer efficiency.The stated goals are to incorporate context and positional information, enable streaming, and reduce computational complexity.
- 4.1. Context Modeling with Causal Convolution: Causal convolution prevents future information from entering the current computation, preserving causal processing.The causal VGGNet stacks convolutional layers and max-pooling before Transformer encoder layers.
- 4.1. Context Modeling with Causal Convolution: The VGGNet front end incorporates positional information and reduces the frame rate for efficient inference.A linear layer follows the VGGNet before multiple Transformer encoder layers generate higher-level representations.
- 4.2. Truncated Self-Attention: Unlimited self-attention prevents streaming and has O(T^2) computational complexity because each output attends to the entire input sequence.Its output at time t depends on all input frames, so inference waits until the final input length is known.
- 4.2. Truncated Self-Attention: Truncated self-attention restricts each output to contexts from x_(t−L) through x_(t+R), making the model streamable and reducing complexity to O(T).The right context R determines look-ahead and latency, while the approach may degrade recognition performance.
5. EXPERIMENTS
Experiments on LibriSpeech compare LSTM, BLSTM, and Transformer neural-transducer components, then evaluate truncated-attention contexts for streaming recognition.
- 5. EXPERIMENTS: Experiments use the 960-hour LibriSpeech corpus, with dev-clean/dev-other for beam-search tuning and test-clean/test-other for evaluation.The corpus includes clean and acoustically more challenging other subsets.
- 5.3. Results on Transformer/LSTM Combinations: The encoder comparison includes streamable LSTM, non-streamable BLSTM, and a VGG-Transformer, while predictors use LSTM 2x700 or Transformer 6x.Architectures are compared with roughly the same total parameter count.
- 5.3. Results on Transformer/LSTM Combinations: The VGG-Transformer with unlimited self-attention outperforms BLSTM as encoder but remains non-streamable, whereas LSTM remains the stronger and smaller predictor.The experiments therefore retain LSTM 2x700 as the predictor.
- 5.4. Results on Truncated Self-Attention: Truncated self-attention performance depends strongly on both left and right context, especially when right context R = 0.The evaluation searches right context first with unlimited left context, then reduces left context.
- 5.4. Results on Truncated Self-Attention: With (L, R) = (16, 4), the VGG-Transformer achieves better WER than both LSTM and BLSTM baselines.With (L, R) = (32, 4), it loses 4.7 % on test-clean and 10.1 % on test-other relatively compared with the reference configuration.
6. CONCLUSION
The proposed Transformer-Transducer combines accuracy with streamability, compactness, and linear computational complexity for resource-limited speech recognition.
- 6. CONCLUSION: 6.37 % test-clean and 15.30 % test-other are achieved with a 45.7 M-parameter streamable system.The system uses causal convolution and truncated self-attention and has O(T) computational complexity.