Source-linked AI summary
Recurrent Batch Normalization
Tim Cooijmans, Nicolas Ballas, César Laurent, Çağlar Gülçehre, Aaron Courville
TL;DR
Training recurrent networks is difficult because exploding and vanishing gradients hinder learning long-term dependencies. The paper reparameterizes LSTM to batch-normalize both input-to-hidden and hidden-to-hidden transformations, finding faster convergence and better generalization across sequential tasks.
Problem
Exploding and vanishing gradients make recurrent neural networks difficult to optimize and can prevent learning long-term dependencies.
Method
The paper reparameterizes LSTM by introducing batch normalization into both the input-to-hidden and hidden-to-hidden transformations.
Results
BN-LSTM consistently outperforms the LSTM baseline across tasks in time to convergence and performance, including language modeling and question answering.
Takeaways & Limitations
Batch-normalizing recurrent hidden states yields optimization and generalization benefits similar to batch normalization in feed-forward networks.
Takeaways & Limitations
On in-order MNIST, constant early inputs create zero-variance hidden states that can make back-propagated gradients explode, requiring Gaussian noise in the initial hidden states.
Abstract
from arXiv · showhide
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps. We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.
1 INTRODUCTION
Recurrent networks are powerful but difficult and costly to optimize, while batch normalization's recurrent use has largely been limited to vertical normalization. This paper proposes hidden-to-hidden normalization in LSTM and reports faster convergence and better generalization across tasks.
- High-capacity recurrent networks achieve strong results on speech recognition, machine translation, and image and video captioning but are computationally intensive and costly to train.
- Internal covariate shift changes layer-input distributions during training and may be especially important in recurrent networks because they resemble very deep feed-forward networks.
- Batch normalization standardizes activations, reduces internal covariate shift, decouples layer parameters, and improves optimization conditioning, convergence, and generalization.
- Prior recurrent applications mainly normalized inputs to RNNs vertically rather than hidden-to-hidden transitions horizontally across timesteps.
- The paper reparameterizes LSTM with batch normalization and finds it easier to optimize and better generalizing.
- Across sequential tasks, the LSTM reparameterization consistently outperforms the LSTM baseline in time to convergence and performance.
2 PREREQUISITES
RNNs process variable-length sequences naturally, but first-order SGD is hindered by exploding and vanishing gradients. LSTM addresses this with a memory cell and gates, while batch normalization standardizes activations using minibatch or training-set statistics.
- An RNN computes each hidden state from the previous hidden state and current input through a nonlinear activation.
- RNNs process variable-length sequences naturally, but SGD training is difficult because gradients can explode or vanish, limiting learning of long-term dependencies.
- The LSTM recurrent transition uses separate cell and hidden-state updates, with sigmoid gates and a Hadamard product.
- LSTM adds a nearly linear memory cell and gates that regulate information carryover, current-input flow, and reading from the cell.
- Batch normalization standardizes activations using empirical means and variances, while retaining correlations because decorrelation would require costly matrix inversion.
- Training estimates normalization statistics from the current minibatch, whereas inference typically uses statistics estimated from the entire training set.
3 BATCH-NORMALIZED LSTM
The proposed BN-LSTM normalizes both input-to-hidden and hidden-to-hidden transformations, separately controlling their contributions while preserving LSTM cell dynamics. Its normalization statistics are handled independently across timesteps because early activations differ from steady-state activations.
- BN-LSTM applies batch normalization to both input-to-hidden and hidden-to-hidden transformations within an LSTM reparameterization.
- The recurrent and input terms are normalized separately, allowing γh and γx to control their relative contributions.
- The formulation avoids redundant biases and omits batch normalization in the cell-state transition to preserve LSTM dynamics and gradient flow through ct.
- Sharing normalization statistics across timesteps severely degrades performance because initial activation statistics differ significantly from later ones.
- Population statistics are estimated separately for each training timestep, with the final timestep's statistic reused beyond Tmax at test time.
- During training, statistics are estimated independently across the minibatch for each timestep; inference uses estimates averaged over the training set.
4 INITIALIZING γ FOR GRADIENT FLOW
The section shows that γ controls gradient propagation: large values cause gradients to vanish, while small values preserve them, motivating careful initialization.
- Large γ values make the gradient norm quickly approach zero during backward propagation through time.The experiment uses a simple RNN on sequential MNIST, and the plot is read from right to left.
- Small γ values keep the gradient norm nearly constant across time steps.
- Increasing pre-activation variance lowers the expected tanh derivative by pushing inputs toward saturation.At unit standard deviation, the expected derivative is much smaller than 1.
- The authors recommend initializing γ to 0.1, while values of 0.01 or lower caused training instabilities.
5 EXPERIMENTS
Across sequential classification, language modeling, and question answering, batch-normalized LSTMs were evaluated for convergence and generalization against vanilla LSTM baselines. The experiments report faster convergence and improved performance, including stronger results on tasks requiring long-term dependencies.
- Experimental setup: The evaluation covers sequential MNIST, Penn Treebank, text8, and CNN question answering, with batch-normalization parameters initialized consistently across experiments.The MNIST models process images one pixel at a time; language models predict the next character.
- Sequential MNIST: On sequential MNIST, BN-LSTM converges faster than LSTM on both pixel orderings and generalizes significantly better on permuted MNIST.Permuted MNIST requires preserving longer-term dependencies across pixels.
- Sequential MNIST: Recurrent batch normalization improves sequential MNIST test accuracy, especially on permuted MNIST, where models must leverage long-term temporal dependencies.The reported table also states that BN-LSTM achieves state-of-the-art accuracy on both MNIST variants.
- Character-level Penn Treebank: On Penn Treebank, BN-LSTM converges faster and generalizes better than the LSTM baseline, with population statistics improving generalization to longer sequences.The best models were early-stopped on validation performance before test evaluation.
- Teaching machines to read and comprehend: On CNN question answering, BN-LSTM trains dramatically faster than LSTM, while BN-everywhere further improves over BN-LSTM and both show a generalization benefit.Validation error minima are 50.3% for the baseline, 49.5% for BN-LSTM, and 50.0% for BN-everywhere; BN-e* and BN-e** reach 47.1% and 43.9%.
6 CONCLUSION
The paper finds that batch-normalizing recurrent hidden states improves optimization and generalization across several sequential tasks, including language modeling and question answering.
- 6 CONCLUSION: Batch-normalizing recurrent hidden states improves optimization.The authors report this result as contrary to previous findings.
- 6 CONCLUSION: BN-LSTM trains faster and generalizes better on language modeling and question-answering tasks.
- 6 CONCLUSION: The model applies to variable-length data, bidirectionality, and highly nonlinear attention mechanisms.
A CONVERGENCE OF POPULATION STATISTICS
Figure 5 examines how population statistics converge to stationary distributions over recurrent time on Penn Treebank.
- A CONVERGENCE OF POPULATION STATISTICS: Figure 5 tracks convergence of population statistics to stationary distributions.
- A CONVERGENCE OF POPULATION STATISTICS: The horizontal axis represents RNN time.
- A CONVERGENCE OF POPULATION STATISTICS: Each curve represents one hidden unit, with only a random subset of units displayed.
B SENSITIVITY TO INITIALIZATION OF γ
Initialization sensitivity differs between pMNIST and Penn Treebank: higher initial γ harms pMNIST optimization, whereas the effect disappears on Penn Treebank.
- B SENSITIVITY TO INITIALIZATION OF γ: Higher initial γ values are detrimental to pMNIST optimization.
- B SENSITIVITY TO INITIALIZATION OF γ: The detrimental effect of higher initial γ disappears for Penn Treebank.
- B SENSITIVITY TO INITIALIZATION OF γ: pMNIST requires gradients to propagate through the entire sequence before the final prediction receives feedback.
- B SENSITIVITY TO INITIALIZATION OF γ: Penn Treebank supplies a fresh learning signal at every timestep, allowing short-term dependencies to support learning.
C TEACHING MACHINES TO READ AND COMPREHEND: TASK SETUP
The question-answering setup uses the CNN corpus with entity placeholders, a reduced four-sentence context heuristic, and specified training configurations for evaluated model variants.
- C TEACHING MACHINES TO READ AND COMPREHEND: TASK SETUP: The evaluation uses the CNN corpus with named-entity placeholders and a vocabulary of 65829 words.Examples are sampled with replacement, and placeholder order is shuffled within minibatches.
- C TEACHING MACHINES TO READ AND COMPREHEND: TASK SETUP: A string-matching heuristic retains only the 4 most relevant description sentences to reduce computation.The same preprocessing is applied to training and validation sets.
- C TEACHING MACHINES TO READ AND COMPREHEND: TASK SETUP: The heuristic can remove answers from passages, imposing a 57% upper bound on validation accuracy.
- C TEACHING MACHINES TO READ AND COMPREHEND: TASK SETUP: LSTM, BN-LSTM, and BN-everywhere use identical hyperparameters selected for the baseline.The hidden state has 240 units, minibatches contain 64 examples, and gradient clipping is set to 10.
- C TEACHING MACHINES TO READ AND COMPREHEND: TASK SETUP: BN-e* and BN-e** use a learning rate of 8 × 10^-4 and minibatches of 40.
D HYPERPARAMETER SEARCHES
The experiments explored hyperparameter values across tasks using independent variation or grid searches, with baseline and BN-LSTM models evaluated under the same tried values. Reported results used the model achieving the best validation performance.
- Table 5 lists the hyperparameter values explored in the experiments.The table reports the values tried rather than a single fixed configuration.
- MNIST and pMNIST varied hyperparameters independently.
- Penn Treebank used a full grid search over learning rate and hidden state size, followed by sensitivity analysis of batch size and initial γ.
- Text8 and Attentive Reader experiments used grid searches over learning rate and hidden state size.
- The baseline and BN-LSTM were tested with the same values, and reported results came from the model with the best validation performance.