Source-linked AI summary
Batch Normalized Recurrent Neural Networks
César Laurent, Gabriel Pereyra, Philémon Brakel, Ying Zhang, Yoshua Bengio
TL;DR
RNNs can model sequential data but are expensive to train, motivating normalization-based approaches. This paper applies batch normalization to RNN transitions and evaluates its effects on training and generalization. Input-to-hidden normalization speeds training, while hidden-to-hidden normalization does not help and generalization does not appear to improve.
Problem
RNNs can learn long-term dependencies but are computationally expensive to train and difficult to parallelize.
Method
The paper evaluates batch normalization for RNNs, including normalization of input-to-hidden transitions and frame-wise normalization.
Results
Input-to-hidden batch normalization produces faster training in both speech and language experiments, while the normalized networks also overfit more.
Takeaways & Limitations
Batch normalization for RNNs is more challenging than for feedforward networks, but input-to-hidden normalization can still improve optimization conditioning.
Takeaways & Limitations
Because normalization covers only parts of the network, higher learning rates did not work well when they affected unnormalized parts.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly improve convergence rates in feedforward neural networks . In particular, batch normalization, which uses mini-batch statistics to standardize features, was shown to significantly reduce training time. In this paper, we show that applying batch normalization to the hidden-to-hidden transitions of our RNNs doesn't help the training procedure. We also show that when applied to the input-to-hidden transitions, batch normalization can lead to a faster convergence of the training criterion but doesn't seem to improve the generalization performance on both our language modelling and speech recognition tasks. All in all, applying batch normalization to RNNs turns out to be more challenging than applying it to feedforward networks, but certain variants of it can still be beneficial.
1 Introduction
RNNs, especially stacked LSTMs, are effective for sequential tasks but can be prohibitively expensive to train. The paper examines normalization as a way to reduce RNN training time.
- RNNs have succeeded in speech recognition, machine translation, and language modelling.
- Stacking multiple RNNs can improve deep architectures but substantially increases training cost.The paper states that training large datasets with stacked RNNs can cost an order of magnitude more than simpler n-gram models.
- Parallelizing RNN training across multiple GPUs has been explored to reduce computation time.
- Batch normalization standardizes intermediate representations using mini-batch statistics and has reduced CNN training time by an order of magnitude.It also provided a regularization effect and state-of-the-art ImageNet object-recognition results in prior work.
2 Batch Normalization
Batch normalization approximates whitening by standardizing intermediate features with mini-batch statistics. Learnable scale and shift parameters preserve representational flexibility, while test-time statistics must be estimated separately.
- Batch normalization addresses internal covariate shift by standardizing intermediate representations with current mini-batch statistics.The method approximates computationally expensive layer-wise whitening.
- For each feature, the method computes a sample mean and variance along the mini-batch axis before standardization.The mini-batch size is denoted by m.
- A small positive ε improves numerical stability during normalization.
- Learnable γ and β parameters respectively scale and shift standardized activations, compensating for reduced representational power.The original representation can be recovered by setting γ_k to σ_k and β_k to x̄_k.
- At test time, batch normalization uses estimated population statistics rather than the current mini-batch statistics.These estimates can come from forwarded training batches or a running average.
3 Recurrent Neural Networks
RNNs transform input sequences into hidden-state sequences and can use recurrent, bidirectional, or stacked architectures. LSTMs address vanishing gradients through gated state dynamics, but recurrent training remains difficult.
- An RNN maps an input sequence of vectors to a sequence of hidden states computed across time steps.The recurrent weight matrix, input-to-hidden weight matrix, and activation function define the computation.
- Bidirectional RNNs incorporate information from both past and future time steps when the full input sequence is available.
- Stacking RNNs feeds one recurrent layer’s hidden states into another, creating deeper architectures.
- Vanilla RNNs are difficult to train because sigmoid-like activations can produce vanishing and exploding gradients.
- 3.1 Long Short-Term Memory: LSTMs mitigate the vanishing-gradient problem using gating functions in their state dynamics.They maintain hidden and cell vectors that control state updates and outputs.
4 Batch Normalization for RNNs
The proposed RNN normalization applies batch normalization only to input-to-hidden transitions, while frame-wise and sequence-wise schemes accommodate different sequence-access settings.
- Applying batch normalization to the hidden-to-hidden transition did not help the training procedure.
- Batch normalization is applied on vertical connections between layers, not horizontal connections within recurrent layers.
- 4.1 Frame-wise and Sequence-wise Normalization: Frame-wise normalization computes statistics at each time step when future frames are unavailable, such as in language modeling.
- 4.1 Frame-wise and Sequence-wise Normalization: Sequence-wise normalization computes feature statistics across time and batch dimensions for variable-length speech sequences with padding.
5 Experiments
The experiments evaluate batch-normalized and baseline recurrent models on speech recognition and language modeling tasks using established datasets and model configurations.
- Experiments covered speech recognition and language modeling tasks implemented with Theano and Blocks.
- 5.1 Speech Alignment Prediction: The speech experiments used the WSJ corpus, training on si284 and evaluating on dev93 with 40-dimensional log mel filter-bank features plus derivatives.
- 5.1 Speech Alignment Prediction: The speech baseline was a stack of 5 bidirectional LSTM layers with 250 hidden units each, followed by a 3546-class softmax.
- 5.1 Speech Alignment Prediction: Figure 1 compares baseline and batch-normalized WSJ frame-wise cross-entropy training and validation curves.
- Language modeling used the standard Penn Treebank split and 10k-word vocabulary with small, medium, and large LSTM models.
- All language models used two stacked LSTM layers, stochastic gradient descent with learning rate 1, and mini-batches of 32.
6 Results and Discussion
Across speech recognition and language modeling, batch normalization speeds training but increases overfitting, leaving best generalization results comparable rather than clearly improved. The experiments suggest input-to-hidden normalization can improve optimization conditioning, although partially normalized networks cannot always exploit higher learning rates.
- Speech recognition: About twice as fast at some points, batch-normalized speech networks train faster than the baseline but overfit more.The best speech results are reported as comparable to prior results.
- Language modeling: Batch normalization makes the large Penn Treebank LSTM train faster but also overfit more than the baseline.Figure 2 compares training and validation perplexity for baseline and batch-normalized networks.
- Cross-task pattern: Both experiments show faster training and greater overfitting with this batch-normalization variant.The weaker overfitting effect in speech may reflect dataset size, normalization granularity, or differences in prediction setup.
- Optimization trade-off: Higher learning rates did not work well because normalization covered only part of the network, leaving unnormalized components affected by the larger updates.This contrasts with the higher-learning-rate behavior reported for batch normalization in feedforward networks.
- Optimization trade-off: Input-to-hidden batch normalization can improve the conditioning of the optimization problem.The paper presents this as an experiment-based suggestion rather than a claim of improved generalization.
A Experimentations with Normalization Inside the Recurrence
The recurrence experiments applied batch normalization to hidden-to-hidden transitions in Penn Treebank LSTMs and found no consistent training benefit. Across the search, normalized networks were usually slightly worse or equivalent to baselines, motivating a different input-to-hidden design.
- Experimental setup: The PennTreebank experiment predicted the next characters of fixed-length sequences containing 100 symbols.It tested whether feedforward-style batch normalization could be applied inside recurrent computation.
- Experimental setup: Hidden-to-hidden normalization computed one mean and variance for each of 250 features at every time step, while sharing γ and β across time.The model used three recurrent layers with 250 hidden units each.
- Results: Across the grid search, batch-normalized networks were slightly worse than or at best equivalent to baselines, except when high learning rates caused baseline divergence.The best baseline and normalized networks nevertheless had similar performances under their respective settings.
- Discussion: The tested recurrent normalization may hurt training because time-step-specific statistics or repeated γ and β applications could produce exploding or vanishing gradients.These mechanisms are proposed explanations that the authors planned to investigate through back-propagation analysis.
- Results: Figure 3 shows typical baseline and batch-normalized training curves from the grid search using learning rate 7.8e-4, momentum 0.5, and batch size 64.The baseline is blue and the batch-normalized network is red.