PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Attention-based Large Vocabulary Speech Recognition

Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, Yoshua Bengio

arXiv:1508.04395v2cs.CLcs.AIcs.LGcs.NE

TL;DR

Most state-of-the-art LVCSR systems remain hybrids with separate modeling components. This paper replaces the HMM with a character-level recurrent sequence generator whose attention learns alignments, and reports that limiting attention to promising locations reduces training complexity from quadratic to linear.

  • Problem

    Most state-of-the-art LVCSR systems remain hybrids, combining neural acoustic models with HMMs and other statistical components.

    Source: §1 · p. 1

  • Method

    The paper uses a character-level encoder-decoder with an attention-equipped RNN to learn input-output alignment, plus windowing and temporal pooling to reduce computation.

    Source: §2 · p. 2, §2.3 · p. 3, §6.1 · pp. 6–7

  • Results

    Limiting attention to promising locations reduces total training complexity from quadratic to linear.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Together, the two efficiency methods facilitate applying attention-based models to large-scale speech recognition.

    Source: §6.1 · pp. 6–7

    The internal language model was trained on fewer than 4 million WSJ transcript characters, and the authors report having to combat overfitting.

    Source: §6 · p. 6

Abstract

from arXiv · show

Many of the current state-of-the-art Large Vocabulary Continuous Speech Recognition Systems (LVCSR) are hybrids of neural networks and Hidden Markov Models (HMMs). Most of these systems contain separate components that deal with the acoustic modelling, language modelling and sequence decoding. We investigate a more direct approach in which the HMM is replaced with a Recurrent Neural Network (RNN) that performs sequence prediction directly at the character level. Alignment between the input features and the desired character sequence is learned automatically by an attention mechanism built into the RNN. For each predicted character, the attention mechanism scans the input sequence and chooses relevant frames. We propose two methods to speed up this operation: limiting the scan to a subset of most promising frames and pooling over time the information contained in neighboring frames, thereby reducing source sequence length. Integrating an n-gram language model into the decoding process yields recognition accuracies similar to other HMM-free RNN-based approaches.

1. INTRODUCTION

The introduction motivates replacing hybrid HMM-based LVCSR with an end-to-end Attention-based Recurrent Sequence Generator (ARSG), avoiding acoustic-model training that is not directly optimized for the final objective. It presents training-time attention windowing, recurrent frame pooling, and WFST integration of a character-level ARSG with an n-gram language model as contributions.

1. INTRODUCTION

  • ARSG replaces HMMs to train neural LVCSR directly for sequence prediction, addressing hybrid systems’ reliance on GMM-HMM forced alignments and acoustic models not optimized for the final objective.The hybrid approach requires two-stage training, using GMM-HMM-derived state labels to train the neural acoustic model.Source: §1 · p. 1
  • Prior CTC-based systems predicted character sequences and combined them with word-level language models, with a CTC-specific WFST yielding accuracy competitive with DNN-HMM hybrids (Miao et al., 2015).The cited CTC work reported promising results on Wall Street Journal and Switchboard tasks (Graves and Jaitly, 2014; Hannun et al., 2014a, 2014b).Source: §1 · p. 1
  • Attention windowing limits training to promising input locations, reducing total training complexity from quadratic to linear; prior windowing was used only during decoding (Chorowski et al., 2015).Source: §1 · pp. 1–2
  • Following the Clockwork RNN (Koutnik et al., 2014) and hierarchical gating RNN (Chung et al., 2015), the authors introduce recurrent pooling of neighboring frames to shorten the source sequence.Source: §1 · p. 2
  • The authors combine a character-level ARSG with an n-gram word-level language model in a complete system using the Weighted Finite State Transducers framework.Source: §1 · p. 2

2. ATTENTION-BASED RECURRENT SEQUENCE GENERATORS FOR SPEECH

The system maps speech-frame sequences to character sequences with a differentiable encoder-decoder built from RNNs. Its attention-equipped decoder learns alignment by selecting relevant encoded frames as it generates each character.

2.1. Recurrent Neural Networks

  • RNNs model sequences with recurrent hidden states; this system uses GRU transitions, and bidirectional RNNs and stacked layers let representations use past and future frames at multiple levels.GRUs were chosen for their simpler architecture and easier efficient implementation.Source: §2.1 · p. 2

2.2. Encoder-Decoder Architecture

  • The deep BiRNN encoder represents speech with a state vector for every 10 ms, but pooling between layers reduces this overly precise, redundant sequence for the decoder.The pooling-over-time figure shows an upper layer running twice as slowly as the lower layer, by averaging or subsampling its hidden states.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3. Attention-equipped RNNs

  • An ARSG generates outputs one at a time while aligning each to the encoded input, using attention-weighted input states to update its RNN and predict the next output.The attention mechanism combines decoding history, candidate-location content, and previous-step focus; location awareness is crucial for reliable behavior on long input sequences.Source: §2.3 · pp. 3–4
  • Windowing restricts attention to positions around the previous alignment median, reducing training complexity from O(LT) to O(L + T) and significantly speeding training.For initialization, the authors instead use rough training-set estimates of silence duration and speaker speed to constrain each character’s window, helping prevent all characters from aligning to one audio location.Source: §2.3 · p. 4

3. INTEGRATION WITH A LANGUAGE MODEL

Because training transcripts are typically inadequate for a high-quality language model and the ARSG predicts characters while speech-recognition LMs are word-based, the section describes integrating them through WFSTs. It combines ARSG and LM costs in beam-search decoding and addresses large, nondeterministic FSTs.

3. INTEGRATION WITH A LANGUAGE MODEL

  • The integration addresses the mismatch between word-based speech-recognition language models and the character-sequence distributions modeled by ARSGs, whose training transcripts are typically insufficient for a high-quality LM.The section motivates supplementing the ARSG’s implicit dependence on prior output symbols with an external language model.Source: §3 · p. 4
  • A word-level LM is converted to character-sequence log-probabilities by composing it with a spelling lexicon and determinizing and minimizing the resulting WFST.The method constructs T = min(det(L ◦ G)) and pushes its weights toward the starting state to help hypothesis pruning.Source: §3 · p. 4
  • The decoding cost combines encoder-decoder and LM outputs with tunable parameters; its γT term prevents the LM from favoring excessively short sequences, a criterion also proposed for CTC (Hannun et al., 2014b).The supplied passage identifies β and γ as tunable parameters.Source: §3 · p. 4
  • Left-to-right beam search can jointly integrate an FST and ARSG because each system’s current state depends only on the previous state and input symbol.The search approximates the transcript minimizing the combined decoding cost.Source: §3 · p. 4
  • Determinization becomes impractical for moderately large FSTs, including the WSJ trigram model, so nondeterministic FST paths are handled by summing their log-probabilities for each character prefix.The method assumes logarithmic-semiring weights and can recompute the probability when a character is appended.Source: §3 · pp. 4–5

4. RELATED WORK

The related work contrasts CTC, RNN Transducers, and attention-based recurrent sequence generation in how they model alignments and language. It also notes closely overlapping work on character-level recognition and pooling, and on n-gram language models built with FSTs.

4. RELATED WORK

  • CTC trains sequence predictors by summing over possible input-output alignments, but its conditionally independent labels leave speech systems without a language model, whose addition can greatly boost performance.The cited applications include phoneme recognition and character-based LVCSR; the RNN Transducer extends CTC by combining an input-time-scale network with an RNN conditioned on previous output labels.Source: §4 · p. 5
  • RNN Transducers can produce output sequences longer than their inputs and achieved state-of-the-art TIMIT phoneme recognition, later matched by an ASRG network.Unlike CTC, Transducer inference accounts for constraints from both RNNs using dynamic programming similar to HMM forward-backward inference.Source: §4 · p. 5
  • RNN Transducers and ARSG learn an implicit language model jointly, but ARSG explicitly computes alignment while Transducers model a distribution over alignments.The authors hypothesize that this alignment difference may substantially affect further development of the methods.Source: §4 · p. 5
  • Chan et al. (2015) independently proposed a similar character-recognition encoder-decoder with pooling between BiRNN layers, while Miao et al. (2015) advocated FSTs for character-level n-gram models.The authors state their research proceeded independently and without communication with either group.Source: §4 · p. 5

5. EXPERIMENTS

On WSJ eval92, the model outperformed CTC without an external language model, but its final performance fell between the results reported by (Hannun et al., 2014b) and (Miao et al., 2015) with the same language models.

5.1. Data

  • The WSJ experiments trained on 81 hours of SI-284 and evaluated on eval92, using 123-dimensional acoustic features and 32 output labels.Hyperparameter selection used dev93; decoding used the dataset’s 20K closed-vocabulary bigram and trigram language models.Source: §5.1 · p. 5

5.2. Training

  • The encoder used four bidirectional GRU layers and reduced utterance length by a factor of four, while attention used a width-200 convolution filter.Training began with rough alignment for one epoch, then used windowing; AdaDelta with gradient clipping and column-norm constraints was applied, and the best CER model was annealed.Source: §5.2 · pp. 5–6

5.3. Decoding and Evaluation

  • Beam search minimized the combined cost and stopped after finding k terminated sequences cheaper than any non-terminated beam sequence.Beam size 200 gave only about 10% relative improvement over size 10; with large language-model weight, decoding used a left attention window of 10 to prevent looping across the input.Source: §5.3 · p. 6

5.4. Results

  • With the same language models, the model’s 13.0% vs 14.1% was better than (Hannun et al., 2014b), but its 11.3% vs 9.0% was worse than (Miao et al., 2015).Without an external language model, the model outperformed CTC systems; adding a language model improved CTC-based systems more.Source: §5.4 · p. 6

6. DISCUSSION

The attention-based encoder-decoder offers a simpler alternative to HMM-DNN systems, with decent but not state-of-the-art performance using a trigram language model. The discussion highlights its implicit language model, possible overfitting, and methods for making attention more efficient and integrating external language models.

6. DISCUSSION

  • The authors attribute ARSG’s stronger performance without an external LM to its implicit RNN language model, while hypothesizing that overfitting explains its gap to CTC (Miao et al., 2015).The implicit model was trained on fewer than 4 million WSJ transcript characters; weight clipping consistently improved performance but did not change the overall picture, and RNN overfitting on corpora of this size has been reported (Graves, 2013).Source: §6 · p. 6
  • An RNN language model trained on larger text could supply states to ARSG from training’s start, potentially reducing memorization and eliminating an extra n-gram model (Gulcehre et al., 2015).The authors suggest overfitting may be less of an issue beyond WSJ’s approximately 80 hours of training data.Source: §6 · p. 6

6.1. Conclusion

  • The attention-based encoder-decoder is simpler than HMM-DNN systems, but with a trigram language model it achieves decent rather than state-of-the-art performance.The authors cite fewer training stages, less auxiliary data, and less domain expertise as advantages.Source: §6.1 · p. 6
  • Pooling between BiRNN layers shortens the encoded sequence, while training-time windowing keeps decoder operations per output character constant, enabling large-scale speech recognition.Together, these methods reduce the model’s computational complexity.Source: §6.1 · pp. 6–7
  • The model has intrinsic language-modeling capability and potential for joint training with an external language model; trainable n-gram integration is another avenue for future work.The authors identify investigation of joint external-LM training as future work.Source: §6 · p. 6, §6.1 · p. 7

Related papers

Loading 1508.04395v2…