Source-linked AI summary
Improved training of end-to-end attention models for speech recognition
Albert Zeyer, Kazuki Irie, Ralf Schlüter, Hermann Ney
TL;DR
The paper addresses how to train simple attention-based end-to-end speech recognizers on subword units effectively. It combines encoder pretraining, BPE decoding, and subword LSTM language-model fusion, achieving competitive Switchboard and LibriSpeech results, including 3.54% dev-clean and 3.82% test-clean WER on LibriSpeech.
Problem
The paper investigates whether simple attention-based end-to-end speech recognition on subword units can achieve competitive results across standard speech-recognition tasks.
Method
The authors use BPE subword targets, encoder pretraining with initially high then reduced time reduction, and shallow fusion with an LSTM language model.
Results
3.54% dev-clean WER and 3.82% test-clean WER are reported on LibriSpeech, while shallow fusion yields 17% to 27% relative WER improvements on different subsets.
Takeaways & Limitations
The reported results show that subword attention models can provide competitive end-to-end recognition on Switchboard and LibriSpeech.
Abstract
from arXiv · showhide
Sequence-to-sequence attention-based models on subword units allow simple open-vocabulary end-to-end speech recognition. In this work, we show that such models can achieve competitive results on the Switchboard 300h and LibriSpeech 1000h tasks. In particular, we report the state-of-the-art word error rates (WER) of 3.54% on the dev-clean and 3.82% on the test-clean evaluation subsets of LibriSpeech. We introduce a new pretraining scheme by starting with a high time reduction factor and lowering it during training, which is crucial both for convergence and final performance. In some experiments, we also use an auxiliary CTC loss function to help the convergence. In addition, we train long short-term memory (LSTM) language models on subword units. By shallow fusion, we report up to 27% relative improvements in WER over the attention baseline without a language model.
1. Introduction
The paper investigates recurrent encoder-attention-decoder models for simple end-to-end speech recognition on subword units. It introduces training and decoding techniques and reports competitive results on Switchboard and LibriSpeech.
- Motivation: End-to-end speech recognition can use a single model operating directly on words, subwords, or characters, removing pronunciation-lexicon and explicit phone modeling requirements.This simplifies decoding compared with conventional hybrid HMM/NN systems.
- Approach: The authors use LSTM networks for both encoder and decoder in a recurrent encoder-attention-decoder architecture without monotonic-attention modifications.The model follows a machine-translation architecture, with encoder time reduction.
- Approach: BPE subword units provide the model’s output representation, while encoder pretraining, beam-search decoding, and subword language-model fusion improve the system.The paper also provides open-source code and experiment setups.
2. Pretraining
The pretraining strategy makes attention-model training easier by initially using substantial encoder time reduction and then reducing it for final performance. Layer-wise encoder growth and temporarily disabling regularization further support convergence.
- Time reduction: The encoder applies max-pooling in the time dimension, including at multiple stages, to compress long speech inputs.This addresses the much longer input sequences typical of speech recognition.
- Time reduction: High encoder time-reduction factors of 16 or 32 usually make training converge, whereas a lower factor such as 8 can perform better after high-factor pretraining.The authors report that high time reduction simplifies training and that lowering it can improve final performance.
- Pretraining scheme: The pretraining scheme grows encoder depth while decreasing the initially high encoder time-reduction factor.The paper presents this as a new pretraining method for encoder-attention-decoder models.
- Regularization: Label smoothing is disabled during pretraining and enabled afterward, while encoder dropout is disabled during the first few pretraining epochs.These settings are intended to improve convergence further.
3. Model
The model is a deep bidirectional LSTM encoder with optional time pooling and an LSTM attention decoder. Its MLP attention uses encoder states, decoder state, and attention-feedback information to generate output probabilities.
- Encoder: A deep bidirectional LSTM encoder optionally applies max-pooling after each layer to reduce encoder sequence length.The resulting encoder length is T′ = red·T, with at least two encoder layers.
- Attention: The encoder-decoder model uses MLP attention and an inverse attention-weight feedback variant for improved numerical stability.The architecture follows earlier machine-translation attention models.
- Attention: Attention energies are computed from the decoder state, encoder state, and attention-weight feedback through trainable parameters and a tanh transformation.The feedback term is defined separately and influences the attention-energy computation.
- Decoder: Attention weights produce a context vector, which is used with the recurrent decoder state to compute the next output distribution.The decoder state is recurrent, and the final readout uses a linear-maxout-linear composition.
4. Sub-word units
The recognizer uses BPE subword units as decoder targets, preserving open-vocabulary recognition while avoiding the computational and coverage problems of word-level outputs. Decoding merges BPE units back into words and can represent special speech events.
- Choice of units: Subword units are used because they can outperform characters while remaining generic, whereas word units cannot recognize out-of-vocabulary words.Word-level outputs also require a large softmax and may have difficult length and label-frequency distributions.
- BPE decoding: BPE units are the decoder’s output targets, and beam search selects hypotheses directly over these units.The BPE sequence is merged into words after decoding for word-level evaluation.
- BPE decoding: Because BPE decoding is open-vocabulary, the system can recognize words beyond a fixed word list.The vocabulary also includes transcription tokens for noise, vocalized noise, and laughter.
5. Language model combination
The models combine subword-level external language models with attention-based recognition through shallow fusion, improving recognition accuracy. Both LSTM and n-gram language models are trained using the same BPE vocabulary.
- Language model training: LSTM language models are trained on the same BPE vocabulary as the end-to-end speech recognizer.Training data includes 27M words for Switchboard and an official 800M-word dataset for LibriSpeech.
- Language model combination: Shallow fusion integrates the external language model into beam search, with optimized weights of 0.23 for Switchboard and 0.36 for LibriSpeech.The attention-model weight remains 1.
- Language model comparison: LibriSpeech also uses Kneser-Ney-smoothed n-gram language models for perplexity and WER comparisons against the LSTM language model.The 4-gram count model is combined by shallow fusion with weight 0.01.
- Evaluation: Perplexities are evaluated on the concatenated LibriSpeech dev-clean and dev-other sets using a shared 10K BPE vocabulary.
6. Experiments
Experiments evaluate subword attention models on Switchboard and LibriSpeech, including encoder pretraining, time reduction, language-model combinations, and beam-search behavior. The reported results show competitive recognition performance, while the beam analysis attributes most errors to the model rather than search pruning.
- Training: Layer-wise encoder pretraining starts with two layers and total time reduction factor 32, then adds layers while reducing pooling to reach a final factor such as 8.The pretraining procedure grows encoder depth and lowers time reduction during training.
- Training: A final time reduction factor of 8 achieves 20.4 WER% on Hub5’00, compared with 21.0 for factor 16 and 21.9 for factor 32.Lower reduction factors require more memory and computation and were constrained by available hardware.
- Switchboard 300h: On Switchboard 300h, the attention model performs best on the easier Hub5’00 Switchboard subset and remains competitive on the harder Callhome subset.The experiments use only the 300-hour Switchboard training data.
- LibriSpeech 1000h: On LibriSpeech 1000h, an auxiliary CTC loss is applied during training to help convergence but is not used during decoding.
- LibriSpeech 1000h: Shallow fusion with an LSTM language model yields 17% to 27% relative WER improvements across LibriSpeech subsets.The best reported WERs are 3.54% on dev-clean and 3.82% on test-clean.
- Beam search: Beam search for attention models operates directly on BPE labels with a fixed low beam size, unlike the dynamic large beams used in hybrid systems.The analysis investigates whether pruning causes recognition errors.
- Beam search: With standard beam size 12, reference-related search errors remain below 1%, and increasing beam size does not noticeably improve WER.The results suggest a model problem rather than a search problem, although only errors involving the real target sequence were counted.
7. Conclusions
The paper presents a BPE-based encoder-decoder-attention speech recognizer with encoder pretraining and LSTM language-model fusion. It achieves competitive results across Switchboard and LibriSpeech, including the best reported LibriSpeech WERs using only official training data.
- The model operates on BPE subword units and uses an encoder-decoder-attention architecture for speech recognition.
- The new encoder pretraining method is crucial for convergence and WER performance.
- Joint beam search with an LSTM language model further improves recognition accuracy.
- The system achieves competitive results on both the 300h Switchboard and 1000h LibriSpeech tasks.
- 3.54% dev-clean WER and 3.82% test-clean WER are reported as the best results on LibriSpeech when only official training data is used.