Source-linked AI summary
State-of-the-art Speech Recognition With Sequence-to-Sequence Models
Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Ekaterina Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, Michiel Bacchiani
TL;DR
The paper asks whether attention-based sequence-to-sequence ASR can outperform conventional systems on challenging voice-search tasks. It develops structural and optimization improvements to LAS, including word pieces, multi-head attention, and several training strategies. The resulting system reaches 5.6% WER on voice search and 4.1% on dictation, outperforming the reported conventional baselines.
Problem
Sequence-to-sequence ASR had not established that it could outperform state-of-the-art conventional systems on large-vocabulary continuous speech recognition, especially for voice search.
Method
The paper improves LAS with word-piece models, multi-head attention, MWER training, scheduled sampling, label smoothing, synchronous training, and language-model rescoring.
Results
5.6% WER on voice search versus 6.7% for a hybrid HMM-LSTM baseline, and 4.1% versus 5% on dictation.
Takeaways & Limitations
The proposed structural and optimization changes allow LAS to outperform the reported conventional systems on both voice-search and dictation tasks.
Abstract
from arXiv · showhide
Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic speech recognition (ASR) system into a single neural network. In previous work, we have shown that such architectures are comparable to state-of-theart ASR systems on dictation tasks, but it was not clear if such architectures would be practical for more challenging tasks such as voice search. In this work, we explore a variety of structural and optimization improvements to our LAS model which significantly improve performance. On the structural side, we show that word piece models can be used instead of graphemes. We also introduce a multi-head attention architecture, which offers improvements over the commonly-used single-head attention. On the optimization side, we explore synchronous training, scheduled sampling, label smoothing, and minimum word error rate optimization, which are all shown to improve accuracy. We present results with a unidirectional LSTM encoder for streaming recognition. On a 12, 500 hour voice search task, we find that the proposed changes improve the WER from 9.2% to 5.6%, while the best conventional system achieves 6.7%; on a dictation task our model achieves a WER of 4.1% compared to 5% for the conventional system.
1. INTRODUCTION
The paper investigates whether sequence-to-sequence ASR can outperform conventional systems on voice search, improving LAS through structural and optimization changes. These changes, plus second-pass language-model rescoring, substantially reduce WER and also improve dictation performance.
- Sequence-to-sequence ASR had not yet outperformed state-of-the-art conventional systems on large-vocabulary continuous speech recognition tasks.
- The paper improves LAS using word pieces, multi-head attention, MWER training, scheduled sampling, label smoothing, and synchronous training.The work also uses a unidirectional encoder for low-latency streaming decoding.
- 9.2% to 5.6% WER on voice search after structural, optimization, and rescoring improvements, versus 6.7% for a strong conventional baseline.The reduction relative to the conventional baseline is 16%.
2. SYSTEM OVERVIEW
The paper frames this section as a detailed examination of structural and optimization improvements to the basic LAS model.
- The paper details structural and optimization improvements to the basic LAS model.
2.1. Basic LAS Model
The basic LAS model is an attention-based encoder-decoder with listener, attender, and speller modules. Together, these modules map acoustic features to output symbols.
- The basic LAS model consists of three modules: a listener encoder, an attender, and a speller decoder.
- The listener encoder maps input features x to a higher-level representation h_enc, analogous to a conventional acoustic model.
- The attender selects encoder features relevant to predicting the next output symbol y_i, functioning similarly to a dynamic time warping alignment module.
- The speller decoder receives the attention output to generate the model’s output sequence.
2.2. Structure Improvements
The paper improves LAS structure with word-piece output units and multi-head attention. Word pieces strengthen decoding flexibility, while multiple attention heads provide distinct views of encoder outputs.
- Wordpiece models: Word pieces can strengthen the decoder language model, increase effective decoder memory, memorize frequent pronunciations, and reduce decoding steps.
- Wordpiece models: Word pieces range from graphemes to entire words and avoid out-of-vocabulary words.They are position-dependent subword units trained to maximize language-model likelihood.
- Multi-headed attention: Multi-head attention assigns multiple attention heads distinct distributions over encoder outputs, allowing each head to attend to a different role or location.
- Multi-headed attention: The authors hypothesize that multi-head attention can better distinguish speech from background noise when encoded representations are less ideal.
2.3. Optimization improvements
The paper improves LAS optimization by aligning training more closely with word error rate, reducing training–inference mismatch, and stabilizing distributed training and predictions.
- Minimum Word Error Rate (MWER) Training: MWER training minimizes expected word errors rather than relying only on a sequence loss not closely related to WER.The MWER objective is interpolated with cross-entropy to stabilize training.
- Minimum Word Error Rate (MWER) Training: N-best beam-search hypotheses approximate the MWER expectation, weighting normalized word errors by re-normalized hypothesis probabilities.Subtracting the average word error provides variance reduction without changing the gradient.
- Scheduled Sampling: Scheduled sampling feeds sampled previous predictions during training to reduce the mismatch between teacher-forced training and inference.
- Synchronous Training: Synchronous training yields a better converged optimum at similar wall-clock time, although distributed training requires variance-reduction techniques.Asynchronous training uses gradual replica ramp-up to reduce early gradient variance.
- Label Smoothing: Label smoothing regularizes LAS by preventing over-confident predictions and making the model more adaptable.The ground-truth distribution is smoothed with a uniform distribution over labels.
2.4. Second-Pass Rescoring
The second-pass rescoring stage supplements LAS with an external language model trained on text from multiple domains, then reranks beam-search hypotheses.
- External Language Model: An external language model compensates for the decoder’s limited exposure to text-only training data during inference.
- External Language Model: The external model is a large 5-gram language model whose domain-specific components are combined using Bayesian interpolation.
- Second-Pass Rescoring: Log-linear interpolation rescoring selects the final transcript from the N-best hypotheses generated by LAS beam search.
- Second-Pass Rescoring: The rescoring criterion demotes hypotheses with low language-model probability and includes a term addressing increased deletions after language-model integration.
3. EXPERIMENTAL DETAILS
Experiments use large-scale anonymized voice-search data with simulated acoustic conditions and evaluate both voice-search and longer-sentence dictation utterances.
- Data: The training set contains approximately 12,500 hours and 15 million anonymized, hand-transcribed English utterances representative of voice-search traffic.
- Data: Clean utterances are artificially corrupted with simulated noise and reverberation spanning 0dB to 30dB SNR, averaging 12dB.
- Evaluation: Evaluation uses approximately 14.8K Google-traffic voice-search utterances and 15.7K dictation utterances with longer sentences.
- Features and Models: Inputs use 80-dimensional log-Mel features computed with 25ms windows, 10ms shifts, frame stacking, and downsampling to a 30ms frame rate.
- Features and Models: The encoder comprises five LSTM layers, with experiments covering both unidirectional and bidirectional LSTM architectures.
- Training: Neural networks are initialized and trained using the cross-entropy criterion, with TensorFlow used for training.
4. RESULTS
Structural, optimization, and rescoring changes substantially improved LAS performance, with benefits for both unidirectional and bidirectional encoders. The improved LAS also outperformed the conventional system on voice search and dictation while using a smaller first-pass model.
- Structure Improvements: WPM slightly outperformed graphemes, yielding roughly a 2% relative improvement in WER.The result is consistent with WPM providing a stronger decoder language model.
- Structure Improvements: 11.1% improvement came from multi-head attention combined with WPM.The model uses multiple attention distributions over the encoder output.
- Optimization Improvements: 27.5% overall optimization gains reduced WER from 8.0% to 5.8%.Synchronous training, scheduled sampling, label smoothing, and MWER contributed relative improvements of 3.8%, 7.8%, 5.6%, and 13.4%, respectively.
- Rescoring: 3.4% improvement from second-pass rescoring reduced WER from 5.8% to 5.6%.A larger language model in the log-linear combination produced a further 0.2% WER improvement.
- Encoder Directionality: 37.8% relative WER reduction occurred for the unidirectional system, versus 28.4% for the bidirectional system.The comparison indicates that most proposed methods improved both model topologies.
- Comparison with the Conventional System: 16% and 18% relative WER improvements over the production system occurred on voice search and dictation, respectively.The LAS first-pass model was around 18 times smaller, although the 80 GB second-pass model dominated total model size.
5. CONCLUSION
The paper presents an attention-based sequence-to-sequence ASR model that integrates acoustic, pronunciation, and language modeling in one neural network. On voice search and dictation, the improved LAS achieved lower WER than the hybrid baseline, while unidirectional encoding retained a latency-related limitation.
- The attention-based model integrates acoustic, pronunciation, and language models into a single neural network without requiring a lexicon or separate text normalization.
- 11% structural gains, 27.5% optimization gains, and 3.4% rescoring gains cumulatively improved the model.
- 5.6% WER on voice search and 4.1% on dictation compared with 6.7% and 5% for the hybrid system, respectively.
- The unidirectional LAS encoder must see the entire utterance before decoding labels, despite encoding the utterance in a streaming fashion.The paper identifies streaming attention-based models such as Neural Transducer as a next step.