Source-linked AI summary
A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
Albert Zeyer, Patrick Doetsch, Paul Voigtlaender, Ralf Schlüter, Hermann Ney
TL;DR
The paper addresses the lack of an overview of how training choices affect deep BLSTM acoustic models for ASR. It systematically studies architectures and training procedures across speech corpora, finding that deep BLSTMs can outperform the FFNN baseline and benefit from layer-wise pretraining.
Problem
The study addresses the missing overview of how deep BLSTM training choices and their interdependencies affect ASR recognition performance.
Method
The authors systematically vary BLSTM architecture and training procedures across corpora using RASR and RETURNN, including depth, optimization, regularization, batching, and gradient clipping.
Results
Over 15% relative WER improvement over the best FFNN baseline is reported, with layer-wise pretraining improving deeper LSTMs and good results reproduced across several corpora.
Takeaways & Limitations
The reported experience favors Adam or (N)Adam, Newbob scheduling, moderate dropout with L2 regularization, gradient noise, and pretraining for deeper models.
Takeaways & Limitations
The study does not investigate projections, although other groups report layer sizes up to 2000 with projection size about 500.
Abstract
from arXiv · showhide
We present a comprehensive study of deep bidirectional long short-term memory (LSTM) recurrent neural network (RNN) based acoustic models for automatic speech recognition (ASR). We study the effect of size and depth and train models of up to 8 layers. We investigate the training aspect and study different variants of optimization methods, batching, truncated backpropagation, different regularization techniques such as dropout and $L_2$ regularization, and different gradient clipping variants. The major part of the experimental analysis was performed on the Quaero corpus. Additional experiments also were performed on the Switchboard corpus. Our best LSTM model has a relative improvement in word error rate of over 14\% compared to our best feed-forward neural network (FFNN) baseline on the Quaero task. On this task, we get our best result with an 8 layer bidirectional LSTM and we show that a pretraining scheme with layer-wise construction helps for deep LSTMs. Finally we compare the training calculation time of many of the presented experiments in relation with recognition performance. All the experiments were done with RETURNN, the RWTH extensible training framework for universal recurrent neural networks in combination with RASR, the RWTH ASR toolkit.
1. INTRODUCTION AND RELATED WORK
The paper provides a comprehensive study of training deep BLSTM acoustic models, addressing the missing overview of how training choices and their interdependencies affect recognition performance. It reports improvements over FFNN baselines and introduces layer-wise pretraining for deeper LSTMs.
- LSTM acoustic models had recently been shown to outperform FFNNs on LVCSR, motivating systematic study of their training.
- The study examines topology, sequence chunking, batch sizes, optimization, and regularization because these choices produce substantial recognition-performance variance.
- Over 15% relative WER improvement was obtained against the best FFNN baseline.
- The experiments train BLSTM acoustic models with up to 10 layers and find that layer-wise pretraining can improve deeper LSTM performance.
- The work uses standard LSTMs without peephole connections while surveying prior bidirectional, unidirectional, projected, convolutional, and feed-forward LSTM acoustic models.
2. LSTM MODEL AND IMPLEMENTATION
The implementation combines RASR for speech-processing and decoding with RETURNN for acoustic-model training and recognition-time forwarding. RETURNN supports the LSTM configurations studied, including a custom CUDA implementation for speed.
- RASR handles feature extraction and decoding, while RETURNN performs acoustic-model training and forwarding during recognition.
- RETURNN supports multiple LSTM implementations and the training aspects examined in the study.
- A custom CUDA kernel provides substantial speed improvements for one RETURNN LSTM implementation.
3. COMPARISONS AND EXPERIMENTS
The experiments use Quaero speech recognition data with a hybrid NN-HMM setup, frame-wise cross-entropy training, fixed alignments, and chunked minibatches. Evaluation tracks WER across selected training epochs while varying optimization and scheduling choices.
- The Quaero experiments use 50 hours of training speech, about 3.5 hours each for development and evaluation, and a 4-gram language model.
- The acoustic models are hybrid NN-HMM systems trained frame-wise with cross-entropy using fixed Viterbi alignments.
- The LSTM inputs are 50-dimensional VTLN-normalized Gammatone features without added context windows or delta frames.
- The baseline uses 9x2000 ReLU layers with a 17-frame context window and achieves 15.3% WER on eval10 and 20.3% on eval11.
- Minibatches commonly contain 40 chunks of up to 50 frames sampled every 25 frames, totaling 2000 frames.
- Training runs for 30 epochs, evaluates selected checkpoints using WER, FER, and CE, and applies Newbob learning-rate scheduling when CV CE improvement falls below 0.01.
3.2. Number of Layers
The layer-depth experiments show that the best depth depends on other hyperparameters and that deeper models do not necessarily overfit more. With pretraining, the overall best result is obtained at six layers.
- 3.2. Number of Layers: Optimization becomes harder as depth increases, motivating experiments with initialization, skip connections, highway structures, and residual learning.
- 3.2. Number of Layers: The optimum is between 4 and 6 layers in this experiment, compared with about 3 to 4 layers in earlier experiments.
- 3.2. Number of Layers: With pretraining, the overall best result is obtained with 6 layers.
- 3.2. Number of Layers: Deeper networks do not show greater overfitting in the reported comparison, likely because their optimization problem is harder.
- 3.2. Number of Layers: Table 1 compares layer counts at fixed 500-unit directional layers using dropout, L2 regularization, Adam, and 40 chunks per minibatch.
3.3. Layer Size
The study finds that hidden-layer size trades off recognition performance against model size, while bidirectionality substantially improves WER relative to unidirectional LSTMs.
- Hidden-layer size: The best hidden-layer size is about 600–700, but size 500 is much smaller with only a modest performance disadvantage.For three layers with Adadelta, the optimum is about 700.
- Scope: Projections were not investigated in this work, although other groups report layer sizes up to 2000 with projection sizes of about 500.This limits direct comparison with projected LSTM architectures.
- Bidirectionality: 19.6% WER for unidirectional LSTM versus 15.6% WER for bidirectional LSTM represents over 20% relative degradation for the unidirectional model.The authors note that the unidirectional network was not tuned as extensively.
- Bidirectionality: Bidirectional RNNs/LSTMs can support online recognition on continuous input with some recognition delay while reaching the original WER.This result motivated further research on online bidirectional recognition.
3.5. Batching
The experiments compare batching and optimization choices for deep BLSTM training, finding useful performance from model averaging, Adam, Newbob scheduling, and gradient clipping.
- Batching: Approximately 40 chunks per minibatch gave the best results among tested values, with some experiments showing a notable advantage over 20 chunks.The authors suggest that more stable minibatch gradients may explain the improvement.
- Optimization: Averaging several independently updated model copies after k minibatch updates yielded the best result among the optimization experiments.The authors postpone further investigation of this method.
- Optimization: Adam was consistently a good choice, while standard SGD sometimes approached it but converged more slowly.Newbob learning-rate scheduling was also considered important.
- Gradient clipping: Clipping the total gradient at 10 stabilized training in some cases, although no clipping often produced the best performance.The comparison covered several gradient-clipping variants.
3.7. Regularization Methods
Regularization performance depended on dropout settings and hidden size, with the combination of dropout and L2 regularization producing the best reported result.
- Dropout: Dropout 0.1 most often produced the optimal WER, while dropout 0.2 was worse in most experiments.Higher dropout could be used when the hidden layer was larger.
- Dropout: The optimal dropout factor depended on hidden size and other training aspects.The authors describe dropout as dropping a fraction of activations and rescaling them.
- Dropout and L2: Combining L2 regularization with dropout gave a large improvement and the best result.The comparison is reported in Table 4.
- Pretraining: The pretraining scheme added one layer after each epoch before the output layer, with full-network training usually outperforming greedy training of only the new layer.The scheme was evaluated against no pretraining for different final depths.
- Pretraining: Pretraining helped more for deeper networks, and the overall best result came from a pretrained six-layer bidirectional LSTM.A nine-layer BLSTM without pretraining diverged after two epochs, while early pretrained epochs also required less calculation time.
3.9. Calculation Time vs. WER
The study relates training calculation time to WER across more than 300 experiments, using standardized epoch-time measurements and reporting trade-offs between speed and recognition performance.
- Experimental scope: More than 300 training experiments supplied statistics relating calculation time to WER.The analysis collected timing and recognition-performance information across many configurations.
- Timing measure: Training times are reported as pure train-epoch calculation time on a GeForce GTX 980, excluding cross-validation testing and other epoch preparation.This measurement choice isolates the training computation itself.
- Time–WER trade-off: Different hyperparameter and method combinations usually produced the best results, while time downsampling reduced calculation time with a performance trade-off.Table 6 reports cumulative train-epoch time and the best WER reached by each specified time.
4. EXPERIMENTS ON OTHER CORPORA
Experiments on Switchboard and Babel Javanese show that LSTM-based acoustic models improve recognition results, while training-time comparisons expose performance–calculation-time trade-offs.
- Switchboard: Table 7 reports Switchboard results for BLSTM models trained with Nadam, gradient noise, dropout, and L2 regularization, including an associative LSTM variant.The table compares BLSTM configurations and an additional associative LSTM layer.
- Babel Javanese: 52.8% WER was achieved by a 3-layer BLSTM on Babel Javanese with CE-training, versus 54.3% for the 6-layer FFNN CE baseline.The BLSTM had 19M parameters, while the FFNN had 34M parameters.
5. CONCLUSIONS
The study evaluates LSTM hyperparameters and deep acoustic-model training across several corpora. It finds that layer-wise pretraining supports deeper models and that the best Quaero BLSTM outperforms the best FFNN by over 15% relatively.
- Scope: The study examines various LSTM hyperparameters and trains deep LSTM acoustic models with up to 10 layers.The conclusions cover model depth, optimization, scheduling, regularization, and gradient noise.
- Pretraining: Layer-wise pretraining allows deeper LSTM acoustic models and is especially relevant for deeper networks.The authors identify pretraining as important for achieving the reported depth.
- Results: Over 15% relative WER improvement was obtained over the best FFNN on Quaero, with good results reproduced across several corpora.The conclusion also reports promising first experiments with associative LSTMs.
- Training choices: The authors recommend (N)Adam, learning-rate scheduling such as Newbob, moderate dropout with L2 regularization, and gradient noise based on their experience.They state that dropout and L2 work best together when not set too high.