Source-linked AI summary
Speech Recognition with Deep Recurrent Neural Networks
Alex Graves, Abdel-rahman Mohamed, Geoffrey Hinton
TL;DR
Speech recognition had lagged behind deep feedforward networks despite RNNs’ ability to model long-range context. This paper introduces deep bidirectional LSTM RNNs trained end-to-end and reports state-of-the-art phoneme recognition on TIMIT.
Problem
RNN-based speech recognition had not matched deep feedforward networks despite speech’s inherently dynamic nature and RNNs’ potential for modeling it.
Method
The paper stacks recurrent hidden layers to build deep bidirectional LSTM RNNs and trains them end-to-end for speech recognition.
Results
Deep bidirectional LSTM RNNs with end-to-end training and weight noise achieve state-of-the-art phoneme recognition on the TIMIT database.
Takeaways & Limitations
The results establish deep bidirectional LSTM RNNs as an effective approach for TIMIT phoneme recognition.
Abstract
from arXiv · showhide
Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connectionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output alignment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cursive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper investigates \emph{deep recurrent neural networks}, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suitable regularisation, we find that deep Long Short-term Memory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score.
1. INTRODUCTION
The introduction motivates end-to-end recurrent speech recognition as an alternative to HMM-based systems and asks whether RNNs can benefit from stacked recurrent layers. It therefore introduces deep Long Short-term Memory RNNs for speech recognition.
- Motivation: Deep feedforward networks have recently produced dramatic improvements in speech acoustic modelling, while HMM-RNN systems do not currently perform as well.Speech is inherently dynamic, motivating recurrent alternatives despite the success of deep feedforward models.
- Motivation: End-to-end RNN training exploits richer dynamics and larger state spaces than HMMs while avoiding potentially incorrect alignment targets.Combining end-to-end training with Long Short-term Memory has proved especially effective for cursive handwriting recognition.
- Contribution: The paper tests whether RNNs benefit from spatial depth by stacking multiple recurrent hidden layers, introducing deep Long Short-term Memory RNNs for speech recognition.RNNs are already deep in time because each hidden state depends on previous hidden states.
2. RECURRENT NEURAL NETWORKS
This section introduces recurrent neural networks, emphasizing LSTM memory cells for exploiting long-range context and bidirectional processing for using both past and future context. It then defines deep bidirectional LSTM by stacking recurrent layers, identifying it as the paper’s main architecture for speech recognition.
- Long Short-Term Memory: LSTM uses purpose-built memory cells to store information and better exploit long-range context than conventional sigmoid-based RNNs.The cell includes input, forget, and output gates alongside cell activation vectors.
- Bidirectional RNNs: Bidirectional RNNs process sequences in both directions with separate hidden layers that feed a shared output layer, enabling future-context use.This is especially applicable to speech recognition, where whole utterances are transcribed at once.
- Bidirectional RNNs: Bidirectional LSTM combines bidirectional processing with LSTM, providing access to long-range context in both input directions.The architecture extends the contextual capabilities of LSTM beyond a single processing direction.
- Deep RNNs: Deep RNNs stack multiple recurrent hidden layers, with each layer’s output sequence serving as the next layer’s input.Deep architectures build progressively higher-level acoustic representations, motivating deep recurrent models for speech recognition.
- Deep RNNs: Deep bidirectional LSTM replaces each stacked hidden sequence with forward and backward sequences, forming the paper’s main architecture.Every hidden layer receives input from both directions of the layer below; the authors describe this as the first application of deep LSTM to speech recognition.
3. NETWORK TRAINING
The network-training approach uses end-to-end sequence modelling, with CTC summing over phoneme-label decisions and an RNN transducer jointly modelling acoustics and prior outputs. Training is strengthened by a feedforward transducer output network, pretrained initialization, beam-search decoding, and regularisation.
- End-to-end training: End-to-end training maps acoustic sequences directly to phonetic sequences without requiring a predefined alignment for training targets.The network parameterises a differentiable distribution over possible phonetic output sequences and optimises the target sequence probability.
- Connectionist Temporal Classification: CTC assigns a softmax distribution over K phonemes plus a blank symbol at each input timestep, then sums over possible alignments with a forward-backward algorithm.CTC ignores segmentation and instead integrates over single-timestep label decisions.
- RNN transducers: The RNN transducer conditions each distribution on both input timestep t and output timestep u, combining acoustic modelling with predictions from previously emitted phonemes.Unlike CTC’s acoustic-only formulation, the transducer jointly represents acoustic and language-model information.
- RNN transducers: The paper replaces the original transducer’s multiplied acoustic and linguistic distributions with a feedforward network that combines both hidden activations before softmax normalisation.This change allows a richer set of possibilities for the output distribution.
- Decoding: RNN transducers are decoded with beam search, while CTC uses the same search with output probabilities made independent of previous outputs.The approach produces an n-best list of candidate transcriptions.
- Regularisation: Regularisation is essential because RNN flexibility makes overfitting likely; the paper uses early stopping and Gaussian weight noise during training.Weight noise is added once per training sequence rather than at every timestep and is intended to improve generalisation.
4. EXPERIMENTS
Experiments on TIMIT evaluated RNNs across training methods, depth, and layer configurations. Deeper bidirectional LSTM networks substantially improved CTC phoneme recognition, while LSTM and pretraining provided additional advantages.
- 4. EXPERIMENTS: Nine TIMIT RNNs varied training method, hidden-layer depth, and LSTM-cell count, with bidirectional LSTM used almost universally.The evaluated methods were CTC, Transducer, and pretrained Transducer; exceptions used tanh or unidirectional LSTM units.
- 4. EXPERIMENTS: At similar parameter counts, LSTM outperformed tanh, bidirectional LSTM slightly outperformed unidirectional LSTM, and depth mattered more than layer size.These conclusions came from comparing CTC-3l-500h-tanh, CTC-1l-622h, CTC-3l-421h-uni, and CTC-3l-250h.
- 4. EXPERIMENTS: Pretraining made the Transducer’s advantage more substantial than with random initialization.The advantage was slight when weights were randomly initialized but increased with pretraining.
5. CONCLUSIONS AND FUTURE WORK
The combination of deep bidirectional LSTM RNNs, end-to-end training, and weight noise achieves state-of-the-art TIMIT phoneme-recognition results. Future work includes extending the system to large-vocabulary speech recognition and combining frequency-domain CNNs with deep LSTM.
- Conclusions: Deep bidirectional LSTM RNNs trained end-to-end with weight noise achieve state-of-the-art phoneme-recognition results on TIMIT.The paper identifies this combination as the source of its strongest result.
- Future Work: The system could next be extended to large-vocabulary speech recognition.The paper presents this as an obvious next step.
- Future Work: Another direction is combining frequency-domain convolutional neural networks with deep LSTM.This is proposed as an additional research direction.