Source-linked AI summary
Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
Haşim Sak, Andrew Senior, Françoise Beaufays
TL;DR
Conventional RNN speech recognition is limited by gradient instability and has seen less large-vocabulary use than DNNs. This paper develops parameter-efficient LSTM architectures and compares them with RNN and DNN models, finding state-of-the-art large-vocabulary performance and improved accuracy across several output sizes.
Problem
Conventional RNN training suffers from vanishing and exploding gradients, while LSTM is designed to address these problems for sequence modeling.
Method
The paper proposes LSTM architectures with recurrent and optional non-recurrent projection layers to use parameters more effectively and improve computational efficiency.
Results
The proposed architectures achieve state-of-the-art performance in large-vocabulary speech recognition with thousands of context-dependent states.
Takeaways & Limitations
The results establish LSTM networks as effective large-vocabulary speech-recognition acoustic models, including for relatively small models.
Abstract
from arXiv · showhide
Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully used for sequence labeling and sequence prediction tasks, such as handwriting recognition, language modeling, phonetic labeling of acoustic frames. However, in contrast to the deep neural networks, the use of RNNs in speech recognition has been limited to phone recognition in small scale tasks. In this paper, we present novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition. We train and compare LSTM, RNN and DNN models at various numbers of parameters and configurations. We show that LSTM models converge quickly and give state of the art speech recognition performance for relatively small sized models.
1. INTRODUCTION
RNNs provide dynamic sequence context but conventional training struggles with gradient instability and long-range dependencies. The paper motivates LSTM architectures for extending recurrent speech recognition from small phone-recognition tasks to large-vocabulary systems.
- RNNs use recurrent activations from previous time steps to model sequence history rather than fixed feedforward context windows.
- Conventional RNN training with BPTT suffers from vanishing and exploding gradients, limiting modeled dependencies to roughly 5–10 time steps.
- LSTM addresses these training problems with memory cells and gates that store temporal state and control information flow.
- LSTM and RNN models have succeeded in sequence prediction, labeling, handwriting recognition, language learning, and phonetic labeling tasks.
- Before this work, LSTM speech-recognition applications were limited mainly to TIMIT phone recognition and often required additional models or techniques.
- The paper proposes modified LSTM RNN architectures for state-of-the-art large-vocabulary recognition while improving parameter usage and computational efficiency.
2. LSTM ARCHITECTURES
The paper describes standard LSTM computation and introduces recurrent and optional non-recurrent projection layers. These architectures target the parameter and computational costs that grow with memory-cell and output-layer sizes.
- A standard LSTM connects input, recurrent LSTM, and output layers, with cell outputs recurrently connected to gates and cell inputs.
- W = nc × nc × 4 + ni × nc × 4 + nc × no + nc × 3 counts standard-LSTM parameters using memory cells, inputs, and outputs.
- Learning complexity is O(W) per time step, becoming expensive when many memory cells and output units are required.
- The proposed recurrent projection layer connects cell outputs to recurrent units, gates, and output units while reducing recurrent parameter growth.
- An optional non-recurrent projection layer increases projection capacity without increasing the nc × nr × 4 recurrent-connection term.
- The network maps input sequence x = (x1, ..., xT) to output sequence y = (y1, ..., yT) by iteratively computing activations over time.
3. EXPERIMENTS
The experiments compare DNN, RNN, and LSTM architectures across speech-recognition models with 126, 2000, and 8000 output states. Frame-accuracy and WER results show advantages for LSTMs, including proposed architectures designed for larger output spaces.
- Experimental setup: The study compares DNN, RNN, and LSTM architectures on the Google English Voice Search task.Models are evaluated with multiple output-state inventories and parameter configurations.
- Experimental setup: Models are trained on about 1900 hours of anonymized, hand-transcribed voice-search and dictation traffic.The data contain 3 million utterances represented by 25ms frames of 40-dimensional log-filterbank features computed every 10ms.
- Output-state configurations: The experiments use 126, 2000, and 8000 phone HMM output states, spanning context-independent and context-dependent configurations.The 126-state inventory is context independent, while the larger inventories are context dependent.
- Evaluation: Evaluation reports frame accuracy on held-out frames and WER on 23,000 hand-transcribed test utterances.The decoding language model has a vocabulary of 2.6 million words.
- Frame-accuracy results: LSTMs achieve much better frame accuracy than RNNs and DNNs, while RNNs are unstable early in training because of exploding gradients.RNNs were evaluated only for the 126-state configuration because they performed significantly worse than DNNs and LSTMs.
- WER results: LSTMs improve speech-recognition accuracy for 126, 2000, and 8000 output-state models, with proposed architectures essential in the 2000-state comparison against DNNs.The 2000-state comparison contrasts LSTM c1024 r256 with LSTM c512; DNN depth is also shown to matter.
4. CONCLUSION
The paper applies LSTMs to large-vocabulary speech recognition and introduces projection-based architectures to use parameters more effectively. These architectures improve LSTM performance over standard LSTMs and outperform DNNs with many output states, while larger training remains a scalability challenge.
- The paper presents what the authors describe as the first LSTM application to a large-vocabulary speech-recognition task.
- Two projection-based architectures address scalability to large networks with many output units by using model parameters more effectively.One adds a recurrent projection layer; the other adds a non-recurrent projection layer that decouples projection size from recurrent connections.
- The proposed architectures significantly improve LSTM performance over the standard LSTM architecture.
- The proposed LSTM architectures outperform DNNs on large-vocabulary speech recognition with many output states.
- Training larger LSTM networks on a single multi-core machine does not scale well.The authors identify GPU and distributed CPU implementations as future directions.