Source-linked AI summary
End-to-End Attention-based Large Vocabulary Speech Recognition
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, Yoshua Bengio
TL;DR
Hybrid LVCSR systems use separate components and indirect acoustic-model training, motivating a more direct end-to-end approach. The paper uses a character-level attention-based recurrent sequence generator with windowing, temporal pooling, and WFST language-model integration; the resulting system achieves decent, though not yet state-of-the-art, performance with a trigram language model.
Problem
Hybrid LVCSR systems use separate components and forced-alignment-based training, so the acoustic model is not directly optimized for the final objective.
Method
The paper uses an end-to-end character-level ARSG whose attention learns input-output alignment, with windowing, temporal pooling, and WFST language-model integration.
Results
Combined with a trigram language model, the system shows decent, although not yet state-of-the-art, performance.
Takeaways & Limitations
Attention-based encoder-decoder models can be applied to large-scale speech recognition with fewer training stages, less auxiliary data, and less domain expertise than the dominating HMM-DNN approach.
Takeaways & Limitations
The internal language model was trained on fewer than 4 million WSJ transcript characters, and determinization becomes impractical for moderately large language-model FSTs.
Abstract
from arXiv · showhide
Many of the current state-of-the-art Large Vocabulary Continuous Speech Recognition Systems (LVCSR) are hybrids of neural networks and Hidden Markov Models (HMMs). Most of these systems contain separate components that deal with the acoustic modelling, language modelling and sequence decoding. We investigate a more direct approach in which the HMM is replaced with a Recurrent Neural Network (RNN) that performs sequence prediction directly at the character level. Alignment between the input features and the desired character sequence is learned automatically by an attention mechanism built into the RNN. For each predicted character, the attention mechanism scans the input sequence and chooses relevant frames. We propose two methods to speed up this operation: limiting the scan to a subset of most promising frames and pooling over time the information contained in neighboring frames, thereby reducing source sequence length. Integrating an n-gram language model into the decoding process yields recognition accuracies similar to other HMM-free RNN-based approaches.
1 CIFAR Fellow
The paper concerns neural-network approaches to large-vocabulary continuous speech recognition, with emphasis on attention.
- The work addresses large-vocabulary continuous speech recognition (LVCSR).
- Neural networks are a central modeling approach in the paper.
- Attention is identified as a key technique for speech recognition.
1. INTRODUCTION
The paper targets the complexity and indirect training of hybrid LVCSR systems by investigating end-to-end attention-based recurrent sequence generation. It introduces windowing, temporal pooling, and language-model integration to make character-level recognition more practical.
- Motivation: Hybrid LVCSR systems retain separate neural, HMM, and language-model components and use forced alignment for two-stage training.
- Motivation: The authors replace HMMs with Attention-based Recurrent Sequence Generators trained end-to-end for direct sequence prediction.
- Contributions: Windowing limits attention to promising locations, reducing training complexity from quadratic to linear for long sequences.
- Contributions: Temporal pooling between recurrent layers successively reduces source sequence length by combining neighboring frames.
- Contributions: A character-level ARSG is combined with a word-level n-gram language model through the WFST framework.
2. ATTENTION-BASED RECURRENT SEQUENCE GENERATORS FOR SPEECH
The proposed attention-based recurrent sequence generator maps speech frames directly to characters while learning input-output alignment. Its encoder-decoder architecture uses recurrent networks, attention, pooling, and windowing to handle variable-length sequences and reduce computational cost.
- System architecture: The system maps sequences of speech frames to character sequences using recurrent neural networks for both encoding and decoding.The decoder combines an RNN with attention to learn alignment between input frames and output characters.
- Encoder architecture: The encoder uses recurrent layers to transform variable-length speech inputs into a sequence of representations for the decoder.A deep bidirectional RNN provides the encoder representation, and stacked recurrent layers can incorporate information from both past and future frames.
- Encoder architecture: Pooling over time reduces source sequence length by combining neighboring hidden states in successive recurrent layers.The architecture can average or subsample lower-layer states, producing a shorter representation for attention and decoding.
- Attention-based decoding: Attention selects relevant temporal locations in the encoded input while generating output characters sequentially.The selected context is a weighted sum of encoder states, with nonnegative attention weights summing to one.
- Efficient attention: The attention mechanism is an improved convolutional hybrid-attention design used within the Attention-based Recurrent Sequence Generator.The paper introduces this mechanism after describing the ARSG architecture and its alignment role.
- Efficient attention: Windowing restricts attention to promising input positions, reducing training complexity from O(LT) to linear complexity and making training faster.The method can also constrain early alignments using estimated speech timing and speaker-speed ranges to help initialize training.
3. INTEGRATION WITH A LANGUAGE MODEL
The paper addresses the mismatch between word-level language models and character-level ARSG outputs by converting the language model into a character-level representation and combining it with ARSG decoding.
- Word-level language models must be reconciled with the ARSG’s character-sequence probability model.The paper identifies this representation mismatch as the central integration challenge.
- WFST composition converts a word-level language model into a character-level model for decoding.The construction composes a language-model FST with a lexicon FST and determinizes and minimizes the result.
- Decoding minimizes a combined encoder-decoder and language-model cost with tunable length and language-model weights.The length penalty prevents the language-model term from favoring excessively short sequences.
- The resulting FST and ARSG can be integrated using left-to-right beam search because both depend on the previous state and input symbol.For nondeterministic FSTs, path probabilities for character prefixes are recomputed in the logarithmic semiring.
4. RELATED WORK
Related work established end-to-end character-based speech recognition and sequence-transduction alternatives, while ARSG explicitly computes alignments rather than marginalizing over them.
- CTC trains recurrent networks for shorter output sequences by summing over possible alignments with dynamic programming.Its conditionally independent outputs lack an internal language model, making external language-model integration important for speech recognition.
- ARSG and RNN Transducer have roughly equivalent capabilities, but ARSG explicitly computes alignment while the Transducer models an alignment distribution.The paper hypothesizes that this distinction may affect future development.
- Concurrent work also proposed character-based encoder-decoder recognition with pooling and FST construction from n-gram language models.The authors state that their research was conducted independently of those works.
5. EXPERIMENTS
Experiments on WSJ evaluated an attention-based encoder-decoder with pooled recurrent representations, beam-search decoding, and optional language-model integration. The model outperformed CTC without an external language model, but its relative gains from language-model addition were smaller.
- 81 hours of WSJ SI-284 speech, about 37K sentences, were used for training, with 123-dimensional frame features and eval92 evaluation.Hyperparameters were selected on dev93.
- The encoder used four bidirectional GRU layers and reduced utterance length by a factor of 4 through temporal pooling.A centered width-200 convolution extracted an attention feature from the previous alignment.
- Beam search minimized a combined cost until k terminated sequences were cheaper than every non-terminated beam sequence.Termination required generation of the special end-of-sequence token.
- Beam size 200 provided only approximately 10% relative improvement over beam size 10.Decoding used different language-model weights with and without an external language model, and asymmetric attention was needed for large γ.
- 13.0% versus 14.1% was better than Hannun et al., while 11.3% versus 9.0% was worse than Miao et al. with the same language models.Without an external language model, the model outperformed CTC systems, but CTC benefited more from adding one.
6. DISCUSSION
The ARSG approach offers intrinsic and potentially jointly trained language modeling, while two computational methods make attention-based LVCSR more scalable. Combined with a trigram language model, it achieved decent but not state-of-the-art performance.
- Discussion: An implicitly learned language model distinguishes ARSG from CTC, but training on fewer than 4 million transcript characters may have caused overfitting.The authors hypothesize that this overfitting contributed to performance below a reported CTC system.
- Discussion: ARSG supports joint training of a language model with the rest of the network, potentially reducing overfitting when more speech or text data are available.The authors also suggest integrating a pretrained RNN language model as an additional input from the beginning of training.
- Discussion: The encoder-decoder attention system is simpler than the dominant HMM-DNN approach, requiring fewer training stages, less auxiliary data, and less domain expertise.The paper reports that its trigram-language-model combination has decent, although not yet state-of-the-art, performance.
- Discussion: Pooling over time and windowing reduce attention computation, with windowing keeping decoder operations constant for each output character.Together, these methods facilitate applying attention-based models to large-scale speech recognition.
- Discussion: Unlike CTC networks, the model has intrinsic language-modeling capability and may support joint training with an external language model.The authors identify investigation of joint external-language-model training as future work.