Source-linked AI summary
Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition
Xiangang Li, Xihong Wu
TL;DR
The paper asks whether adding spatial depth can improve LSTM acoustic modeling beyond shallow networks. It develops and evaluates alternative deep LSTM architectures on large-vocabulary conversational telephone speech recognition, finding that deep designs outperform shallow LSTMs and DNNs while leaving further architectural exploration open.
Problem
The paper investigates whether LSTM acoustic models can gain further performance from spatial depth, motivated by the representational efficiency of deep hierarchical models.
Method
The paper constructs alternative deep LSTM architectures by deepening input-to-hidden, hidden-to-hidden, and hidden-to-output functions, and by stacking LSTM layers.
Results
Deep LSTM architectures outperformed standard shallow LSTM networks and DNNs on the evaluated large-vocabulary speech recognition task.
Takeaways & Limitations
Deep LSTM networks benefit from depth, with an LSTM-OP followed by three feed-forward intermediate layers outperforming stacked LSTM-OPs.
Takeaways & Limitations
The study is preliminary, and many additional LSTM architectures remain to be explored and evaluated.
Abstract
from arXiv · showhide
Long short-term memory (LSTM) based acoustic modeling methods have recently been shown to give state-of-the-art performance on some speech recognition tasks. To achieve a further performance improvement, in this research, deep extensions on LSTM are investigated considering that deep hierarchical model has turned out to be more efficient than a shallow one. Motivated by previous research on constructing deep recurrent neural networks (RNNs), alternative deep LSTM architectures are proposed and empirically evaluated on a large vocabulary conversational telephone speech recognition task. Meanwhile, regarding to multi-GPU devices, the training process for LSTM networks is introduced and discussed. Experimental results demonstrate that the deep LSTM networks benefit from the depth and yield the state-of-the-art performance on this task.
1. INTRODUCTION
The paper motivates LSTM acoustic modeling by speech’s temporal structure and conventional RNN training difficulties, then investigates spatially deep LSTM architectures for large-vocabulary recognition.
- CD-DNN-HMM systems had become dominant for acoustic modeling in speech recognition.
- RNNs model temporal context through cyclic connections, making them suitable for sequence modeling and dynamic speech.
- Conventional RNNs are difficult to train because of vanishing-gradient and exploding-gradient problems.
- LSTM was proposed as an RNN architecture addressing these training problems and has achieved strong results across sequence and speech tasks.
- This paper explores spatial depth in LSTMs, evaluates alternative deep architectures on large-vocabulary Mandarin conversational telephone speech, and compares their performance.
2. CONSTRUCTING LSTM BASED DEEP RNNS
The paper constructs deep LSTMs by increasing depth in recurrent transitions, input-to-hidden mappings, hidden-to-output mappings, or stacked layers, using alternative architectural designs.
- Conventional recurrent architecture: A conventional RNN maps an input sequence to hidden and output sequences recurrently through weight matrices, biases, and a hidden-layer function.
- Conventional LSTM architecture: An LSTM recurrent layer uses memory blocks containing self-connected cells and multiplicative gates controlling information flow.
- Conventional LSTM architecture: The LSTM equations define input, forget, and output gates, cell activation, cell state, and hidden state, with peephole connections and nonlinear activations.
- Deep LSTM strategies: The paper deepens LSTMs along input-to-hidden, hidden-to-hidden, and hidden-to-output functions, alongside stacked LSTM layers.
- Deep hidden-to-hidden transition: LSTM-IP inserts a multilayer transformation before cell input activation, while LSTM-OP adds a separate hidden layer after the memory blocks.
- Other deep constructions: Other strategies use DNN-derived higher-level inputs, intermediate layers before softmax, or vertically stacked LSTM layers.
3. GPU IMPLEMENTATION
LSTM training is implemented across multiple GPUs using truncated back-propagation through time and parallel processing of subsequences. The implementation addresses the substantial computational cost of LSTM layers.
- GPU training procedure: Training splits each sentence into equal-length subsequences with overlapping frames for truncated back-propagation through time.Gradients are computed for each subsequence and back-propagated to its start.
- GPU training procedure: Each GPU processes multiple subsequences from different utterances in parallel to improve computational efficiency.The example uses 20 subsequences per GPU at a time.
- Training cost: A shallow conventional LSTM with 750 cells required about two days of training on four GPUs using a 150-hour speech corpus.The reported training time illustrates the computational demands of LSTM acoustic modeling.
- Training cost: Training an LSTM layer took around two to five times as long as training a fully connected feed-forward hidden layer.
4. EXPERIMENTS
Experiments compare baseline acoustic models with several deep LSTM constructions on HKUST Mandarin conversational telephone speech recognition. The results show that depth and combinations of LSTM architectural strategies improve recognition performance, with a deep LSTM-OP variant offering a strong accuracy–computation trade-off.
- Experimental setup: The experiments use 25ms Melscale log-filterbank frames with energy and temporal derivatives; DNNs use context windows, whereas LSTMs use only current features.DNN inputs concatenate the current frame with five frames on each side.
- Experimental setup: All systems use a trigram language model and hybrid neural-network/HMM acoustic modeling with 3304 tied context-dependent HMM states.Network training uses alignments generated by a well-trained GMM-HMM system.
- Deep LSTM strategies: Deep LSTM architectures extend depth through hidden-to-hidden transitions, input-to-hidden functions, or hidden-to-output functions.The evaluated models include feed-forward layers combined with LSTM layers and stacked LSTM designs.
- Results: A 1-layer conventional LSTM performs worse than the feed-forward DNN baselines, while deeper hidden-to-hidden, input-to-hidden, and hidden-to-output designs improve performance.The LSTM-OP achieves performance comparable to stacked LSTMs.
- Results: The best performance comes from combining LSTM-OP with a deep hidden-to-output function, while stacking LSTM-IPs and LSTM-OPs also further improves performance.Three fully connected hidden layers above LSTM-OP require fewer computations than stacked LSTM-OPs during training and testing.
- Overall results: 13.98% relative CER reduction is obtained over the shallow LSTM network, while CER falls from 38.01% to 34.65% versus feed-forward DNNs, an 8.87% relative reduction.
5. DISCUSSION AND CONCLUSIONS
The paper develops and evaluates deep LSTM architectures for large-vocabulary speech recognition, including alternatives across input-to-hidden, hidden-to-hidden, and hidden-to-output functions. Deep LSTMs outperformed shallow LSTMs and DNNs, while the authors characterize the study as preliminary.
- Architecture construction: The study constructs deep LSTM RNNs by varying input-to-hidden, hidden-to-hidden, and hidden-to-output functions.Deeper variants also combine alternatives from these three construction points.
- Training: Multi-GPU training used truncated BPTT, and experiments found that LSTM RNNs could be trained quickly on GPUs.
- Experimental findings: Deep LSTM architectures outperformed standard shallow LSTM networks and DNNs on large-vocabulary Mandarin conversational telephone speech recognition.
- Experimental findings: LSTM-OP with three feed-forward intermediate layers outperformed stacked LSTM-OPs.
- Limitations and future work: The authors describe the work as a preliminary study and identify additional deep LSTM architectures for future evaluation.Examples include LSTM-IP, stacked LSTMs with feed-forward layers, input/output projection variants, and maxout-based designs.