Source-linked AI summary
Streaming automatic speech recognition with the transformer model
Niko Moritz, Takaaki Hori, Jonathan Le Roux
TL;DR
Although transformer encoder-decoder ASR achieves strong accuracy, its requirement for complete speech utterances limits practical use to offline recognition. This paper combines time-restricted encoder self-attention with triggered decoder attention to enable streaming, achieving 2.8% and 7.2% WER on LibriSpeech test-clean and test-other.
Problem
Transformer encoder-decoder ASR is mainly limited to offline recognition because it typically requires an entire speech utterance as input.
Method
The system applies time-restricted self-attention in the encoder and triggered attention in encoder-decoder attention for streaming recognition.
Results
2.8% and 7.2% WER were achieved on LibriSpeech test-clean and test-other, respectively.
Takeaways & Limitations
The proposed transformer-based streaming ASR system reports the best published LibriSpeech streaming end-to-end ASR result to the authors’ knowledge.
Abstract
from arXiv · showhide
Encoder-decoder based sequence-to-sequence models have demonstrated state-of-the-art results in end-to-end automatic speech recognition (ASR). Recently, the transformer architecture, which uses self-attention to model temporal context information, has been shown to achieve significantly lower word error rates (WERs) compared to recurrent neural network (RNN) based system architectures. Despite its success, the practical usage is limited to offline ASR tasks, since encoder-decoder architectures typically require an entire speech utterance as input. In this work, we propose a transformer based end-to-end ASR system for streaming ASR, where an output must be generated shortly after each spoken word. To achieve this, we apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech, which to our knowledge is the best published streaming end-to-end ASR result for this task.
1. INTRODUCTION
End-to-end ASR systems use sequence-to-sequence models, with CTC, RNN-T, and attention-based encoder-decoder approaches among the leading methods. Transformer architectures use self-attention but, like other encoder-decoder systems, have primarily been suited to offline recognition because they require complete utterances.
- End-to-end ASR converts speech to text with a single sequence-to-sequence model.
- CTC, RNN-T, and attention-based encoder-decoder architectures are the most popular successful end-to-end ASR approaches.
- Transformer models use self-attention and have achieved lower WERs than RNN-based architectures.
- Bidirectional LSTMs perform strongly but are unsuitable for streaming, requiring unidirectional or latency-controlled alternatives.
2. STREAMING TRANSFORMER
The streaming transformer uses time-restricted encoder self-attention and triggered encoder-decoder attention to limit future context while decoding speech incrementally. Joint CTC-triggered attention decoding combines frame-synchronous CTC prefix search with transformer scores.
- Architecture: The transformer uses encoder self-attention and encoder-decoder attention, with the latter controlling attention from decoder queries to encoder states.Both attention types use scaled dot-product attention; self-attention produces an output sequence with the same length as its input.
- 2.1. Encoder: Time-restricted self-attention: The encoder combines a two-layer CNN with a stack of self-attention layers to process acoustic feature sequences.The CNN uses two stride-2 layers, reducing the output frame rate by a factor of 4.
- 2.1. Encoder: Time-restricted self-attention: Time-restricted self-attention limits the encoder’s future context to a fixed number of look-ahead frames, εenc, to control latency.The restricted encoder processes input through position n+εenc, where εenc denotes the look-ahead frames.
- 2.2. Decoder: Triggered attention: Triggered attention restricts decoder attention to past encoder frames plus εdec look-ahead frames using alignments generated by an auxiliary CTC objective.The CTC and triggered-attention models share the encoder and are trained jointly with a weighted multi-objective loss.
- 2.4. Joint CTC-triggered attention decoding: Joint decoding performs frame-synchronous one-pass scoring by combining CTC prefix beam search with triggered-attention transformer scores.The procedure maintains joint and triggered-attention hypothesis sets while pruning prefixes and computing joint scores frame by frame.
3. EXPERIMENTS
Experiments benchmark transformer ASR systems on LibriSpeech, comparing decoding strategies and streaming encoder configurations under varied model and training settings.
- 3.1. Dataset: LibriSpeech provides about 960 hours of training data, 10.7 hours of development data, and 10.5 hours of test data split into “clean” and “other”.
- 3.2. Settings: Table 1 evaluates full-sequence CTC-transformer systems using joint CTC-attention, CTC prefix beam search, and attention beam search decoding.The table also varies RNN-LM inclusion, SpecAugment, and transformer size.
- 3.3. Results: Joint CTC-attention decoding provides significantly better ASR results than CTC-only or attention-only decoding.CTC prefix beam search generally attains lower WERs than attention beam search, with stated exceptions under some no-language-model conditions.
- 3.3. Results: Joint CTC-TA decoding consistently improves WERs over CTC prefix beam search, and larger look-ahead values approach full-sequence CTC-attention results.The comparison uses time-restricted encoder attention with varying εenc and decoder delay εdec settings.
- 3.3. Results: 2.8% and 7.2% WER are achieved on LibriSpeech test-clean and test-other with 2190 ms overall processing delay.With εenc = 1 and εdec = 18, WERs are 3.0% and 7.8% with 1230 ms total delay.
4. CONCLUSIONS
The paper proposes a fully streaming transformer-based end-to-end ASR system using restricted encoder attention and triggered decoder attention, achieving strong LibriSpeech results.
- 4. CONCLUSIONS: Time-restricted self-attention controls encoder latency, while triggered attention controls decoder output latency.
- 4. CONCLUSIONS: A frame-synchronous one-pass decoding algorithm supports streaming recognition and joint CTC-transformer model scoring.Increasing look-ahead frames produces results similar to full-sequence CTC-attention.
- 4. CONCLUSIONS: 2.8% and 7.2% WER are obtained on LibriSpeech test-clean and test-other, respectively, for the proposed fully streaming system.The authors identify this as the best published LibriSpeech result for a fully streaming end-to-end ASR system.