Source-linked AI summary
Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
TL;DR
Batch normalization speeds training but depends on mini-batch statistics and is difficult to apply to recurrent networks. This paper introduces layer normalization, which normalizes summed inputs within each layer and improves training speed and generalization, especially for recurrent networks.
Problem
Batch normalization depends on mini-batch statistics and is not straightforward to apply to recurrent neural networks.
Method
Layer normalization computes normalization statistics from all summed inputs within a layer for each training case, with adaptive neuron-specific gain and bias.
Results
Recurrent neural networks benefit most from layer normalization, especially on long sequences and with small mini-batches, while experiments show faster training and improved validation or generalization.
Takeaways & Limitations
Layer normalization is a practical alternative to batch normalization for recurrent networks, stabilizing hidden-state dynamics while reducing training time.
Takeaways & Limitations
In preliminary convolutional-network experiments, batch normalization outperformed layer normalization because convolutional units have differing statistics across receptive-field locations.
Abstract
from arXiv · showhide
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques.
1 Introduction
State-of-the-art deep networks can take many days to train, while batch normalization has limitations for recurrent, online, and small-minibatch settings. The paper introduces layer normalization to improve training speed without creating dependencies between training cases, including in RNNs.
- Motivation: State-of-the-art deep neural networks often require many days of training.The introduction motivates normalization as a way to speed learning.
- Limitations of batch normalization: Batch normalization appears difficult to apply to RNNs because recurrent summed-input statistics vary with sequence length and time step.Fixed-depth feed-forward networks can store statistics per hidden layer, but RNNs may require separate statistics for different time steps.
- Limitations of batch normalization: Batch normalization cannot be applied to online learning or extremely large distributed models that require small minibatches.
- Layer normalization: The paper introduces layer normalization, which estimates statistics from neurons’ summed inputs within a hidden layer without adding dependencies between training cases.The method is presented as a simple normalization approach for improving training speed across neural network models.
- Layer normalization: Layer normalization works well for recurrent neural networks and improves training time and generalization performance.
2 Background
This section introduces feed-forward neural networks as nonlinear mappings whose hidden-layer summed inputs are formed by linear projections and transformed element-wise. It also motivates batch normalization as a response to correlated layer outputs and describes estimating normalization statistics from empirical samples rather than the full training distribution.
- A feed-forward neural network maps an input pattern x nonlinearly to an output vector y.
- Each hidden layer forms summed inputs from the previous layer through a weight-matrix projection and applies an element-wise nonlinear function.
- Network parameters are learned with gradient-based optimization, using gradients computed by back-propagation.
- Batch normalization was proposed to reduce undesirable covariate shift caused by strong dependence of weight gradients on correlated outputs from the previous layer.
- Normalization statistics are typically estimated from empirical samples because exact expectations over the whole training distribution require impractical full-dataset forward passes.
3 Layer normalization
Layer normalization normalizes summed inputs using statistics computed across all hidden units in a layer for each training case, avoiding batch-size constraints. Applied to recurrent networks, it stabilizes hidden-to-hidden dynamics by making them invariant to rescaling of summed inputs.
- Layer normalization: Layer normalization computes shared mean and variance across all hidden units in a layer, while different training cases use different normalization terms.H denotes the number of hidden units in the layer.
- Layer normalization: Unlike batch normalization, layer normalization is independent of mini-batch size and supports pure online training with batch size 1.The same normalization terms are shared across units within a layer but vary across training cases.
- Layer normalization in recurrent networks: In recurrent networks, layer normalization re-centers and re-scales activations computed from the current input and previous hidden state.The recurrent computation uses Whhht−1 + Wxhxt, with separate bias and gain parameters matching the hidden-state dimension.
- Layer normalization in recurrent networks: Standard recurrent networks can develop exploding or vanishing gradients as summed-input magnitudes grow or shrink across time-steps.This instability arises from changes in the average magnitude of summed inputs to recurrent units.
- Layer normalization in recurrent networks: Layer-normalized recurrent networks are invariant to rescaling summed inputs, producing much more stable hidden-to-hidden dynamics.The normalization terms remove sensitivity to the scale of all summed inputs within a layer.
4 Related work
Prior work extended batch normalization to recurrent neural networks and found that time-step-specific statistics improve recurrent batch-normalization performance. It also reported that initializing the recurrent normalization gain to 0.1 significantly affects final performance.
- Batch normalization has been extended to recurrent neural networks in several previous studies.
- The best recurrent batch-normalization performance was reported when normalization statistics were kept independent for each time step.
- Initializing the recurrent batch-normalization gain parameter to 0.1 significantly affected final performance.
5 Analysis
The analysis compares normalization methods through invariance properties and parameter-space geometry. It shows that normalization can preserve model predictions under specific rescalings while stabilizing learning through curvature-dependent effects.
- Invariance properties: Layer, batch, and weight normalization all normalize summed inputs with scalars µ and σ and learn neuron-specific bias and gain parameters.For layer and batch normalization, µ and σ follow Eq. 2 and 3; for weight normalization, µ = 0 and σ = ∥w∥2.
- Invariance properties: Batch and weight normalization are invariant to rescaling a neuron’s incoming weights because µ and σ scale by the same factor.The normalized summed inputs remain unchanged before and after scaling the weight vector by δ.
- Invariance properties: All normalization methods are invariant to dataset rescaling, while layer normalization also preserves predictions when individual training cases are rescaled.Layer normalization’s statistics depend only on the current input data, and batch normalization is additionally invariant to dataset recentering.
- Learning geometry: Normalization scalars σ implicitly reduce learning rates and stabilize training by changing the Fisher-information curvature along weight directions.If a weight vector’s norm doubles, σ also doubles, changing curvature along that direction by a factor of 1
- Learning geometry: Normalization makes large-norm weight vectors harder to reorient, creating an implicit early-stopping effect that stabilizes convergence.The analysis focuses on generalized linear models and extends to deep networks under a block-diagonal Fisher-information approximation.
6 Experimental results
Across six tasks, layer normalization generally accelerated training and often improved final performance, including in recurrent, generative, and sequence-generation models. Its benefits were strongest in recurrent settings, while batch normalization remained superior in preliminary convolutional experiments.
- Image-sentence ranking: Layer normalization accelerated image-sentence ranking training across all reported retrieval metrics and reached its best validation model in 60% of the baseline’s time.It also improved generalization on the test sets.
- Question answering: In question answering, layer normalization trained faster and converged to a better validation result than both the baseline and recurrent batch-normalization variants.Scale initialization at 1.0 performed significantly better than initialization at 0.1.
- Generative modelling: Layer-normalized DRAW converged almost twice as fast as the baseline and achieved 82.09 nats versus 82.36 nats for the baseline after 200 epochs.The comparison evaluated test variational bounds during the first 100 epochs and final test variational log likelihood after 200 epochs.
- Handwriting sequence generation: For handwriting sequence generation, layer normalization converged much faster while reaching a comparable log likelihood to the baseline under small batches and very long sequences.These conditions make stable hidden dynamics especially important.
- MNIST classification and convolutional networks: Layer normalization was robust across batch sizes and converged faster than batch normalization applied to all layers, but batch normalization outperformed other methods in preliminary convolutional experiments.The convolutional result was attributed to hidden units making less similar contributions in convolutional layers than in fully connected layers.
7 Conclusion
The paper introduced layer normalization to speed neural-network training and analyzed its invariance relative to batch and weight normalization. Experiments found the greatest benefits for recurrent networks, particularly with long sequences and small mini-batches.
- 7 Conclusion: Layer normalization was introduced to speed up neural-network training and shown to be invariant to per-training-case feature shifting and scaling.The paper also theoretically compared its invariance properties with batch normalization and weight normalization.
- 7 Conclusion: Recurrent neural networks benefited most from layer normalization, especially on long sequences and with small mini-batches.
Supplementary Material
The supplementary material specifies how layer normalization is implemented across the experiments, including recurrent architectures and parameter initialization. It also compares normalized and standard parameterizations, concluding that batch- and layer-normalized models are more robust to input and parameter scaling.
- Layer-normalization definition: Layer normalization is defined as a mapping LN: R^D→R^D with adaptive gains α and biases β.The supplementary material introduces this function for consistent notation across experiments.
- LSTM implementation: The layer-normalized LSTM applies separate normalization terms to recurrent and input transformations and normalizes the cell state before computing the hidden state.The parameters α_i and β_i are initialized to zero and one vectors, respectively.
- GRU implementation: A gated recurrent unit variant also incorporates layer normalization, using the same zero-gain and one-bias initialization.The passage identifies the architecture as a gated recurrent unit variant and states that layer normalization is applied to it.
- Output normalization: In another experiment, layer normalization is applied only to the LSTM hidden-state output by normalizing the cell state before producing h_t.The additive parameter α starts at zero and the multiplicative parameter β starts at one.
- Parameterization comparison: Batch-normalized and layer-normalized models are more robust than the standard model to scaling of the input and its parameters.The comparison examines how gradient updates change the magnitude of equivalent weights under normalized and original parameterizations.