Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe; Christian Szegedy

Source-linked AI summary

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy

arXiv:1502.03167v3Feb 11, 2015cs.LG

TL;DR

Changing layer inputs during training complicates deep-network optimization. Batch Normalization normalizes activations per mini-batch within the architecture, reaching prior accuracy with 14 times fewer training steps and surpassing ImageNet results.

Problem
Deep-network training requires careful learning-rate tuning and initialization because preceding-layer updates alter later-layer inputs and can induce vanishing gradients.
Method
Batch Normalization normalizes each activation feature’s mean and variance per mini-batch within the network architecture while backpropagating through normalization parameters.
Results
14 times fewer steps reach 72.2% accuracy than Inception, while batch-normalized ensembles improve upon the best known ImageNet system.
Takeaways & Limitations
Batch Normalization substantially accelerates training and supports higher learning rates while preserving the network’s representation ability.
Takeaways & Limitations
Computing normalization parameters outside the gradient-descent step can leave outputs unchanged while parameters grow indefinitely, causing the model to blow up.

Abstract

from arXiv · show

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

1 Introduction

Training deep networks is hindered by changing layer-input distributions, which force continual adaptation and can cause vanishing gradients in saturating nonlinearities. Batch Normalization reduces this internal covariate shift by fixing layer-input means and variances, accelerating training and improving ImageNet results.

1 Introduction: Mini-batches improve gradient estimates as batch size increases and make computation more efficient through parallelism.These properties motivate the mini-batch training setting used by stochastic gradient methods.
1 Introduction: Changing layer-input distributions force layers to continually adapt, while preceding-parameter changes amplify as networks become deeper.The resulting training difficulty requires careful tuning of learning rates and parameter initialization.
1 Introduction: In sigmoid layers, parameter changes can move many pre-activation dimensions into saturation, causing gradients to vanish and slowing convergence.This effect is amplified by network depth because pre-activations depend on parameters throughout preceding layers.
1 Introduction: Batch Normalization reduces internal covariate shift by normalizing layer inputs to fix their means and variances.The method is designed to accelerate deep neural-network training by reducing changes in internal-node distributions during training.
1 Introduction: Using only 7% of the training steps, Batch Normalization matches the best-performing ImageNet network and further exceeds its accuracy by a substantial margin.An ensemble of batch-normalized networks improves upon the best known ImageNet top-5 error rate.

2 Towards Reducing Internal Covariate Shift

The section defines internal covariate shift as parameter-driven changes in activation distributions and motivates normalization to stabilize layer inputs and accelerate training. It argues that effective normalization must be differentiable, account for parameter dependence, avoid costly full whitening, and preserve activation information.

Motivation: Internal covariate shift is the change in network-activation distributions caused by changing parameters during training, and fixing layer-input distributions is expected to improve training speed.Whitened inputs are known to yield faster convergence.
Limitations of naive normalization: Naively interspersing normalization with optimization can make gradient descent ignore normalization’s parameter dependence, leaving layer outputs and loss unchanged while parameters drift indefinitely.The problem can worsen when normalization both centers and scales activations.
Differentiable normalization: The desired approach must make normalization part of the differentiable network so gradients account for how normalization depends on model parameters.This ensures the network produces activations with the desired distribution for any parameter values.
Whitening cost: Full whitening is impractical because it requires covariance estimation, an inverse square root, and derivatives of these transformations for backpropagation.This motivates an alternative normalization method that is differentiable without requiring the full whitening computation.
Preserving representation: Normalization based on individual examples or feature maps can discard absolute activation scale, so the method should normalize each example relative to statistics from the entire training data.This preserves the network’s representational information.

3 Normalization via Mini-Batch Statistics

Batch Normalization simplifies costly full whitening by independently standardizing each activation using mini-batch statistics, while learned scale and shift preserve the network’s representation power. The resulting differentiable transform introduces normalized activations that reduce internal covariate shift during training and can represent the identity transformation.

3 Normalization via Mini-Batch Statistics: Batch Normalization independently normalizes each scalar feature to zero mean and unit variance instead of jointly whitening layer inputs and outputs.This simplification avoids the cost and nondifferentiability of full whitening; such normalization can speed convergence even without decorrelating features.
3 Normalization via Mini-Batch Statistics: Learned parameters γ(k) and β(k) scale and shift normalized activations, preserving representation power by allowing the inserted transform to implement the identity.Simply normalizing inputs could constrain sigmoid inputs to its linear regime; the affine transform addresses this limitation.
3 Normalization via Mini-Batch Statistics: Mini-batches provide activation-specific mean and variance estimates that participate fully in gradient backpropagation during stochastic training.The normalization is applied independently to each activation across the examples in a mini-batch.
3 Normalization via Mini-Batch Statistics: The BN transform is differentiable and makes each example’s output depend on the other examples in its mini-batch, while passing scaled and shifted activations to subsequent layers.The normalized activations have fixed means and variances, although their joint distribution may change during training.
3 Normalization via Mini-Batch Statistics: Batch Normalization reduces internal covariate shift by introducing normalized activations that enable layers to learn from more stable input distributions without sacrificing network capacity.The learned affine transformation preserves the identity transformation while the normalization accelerates training.

4 Experiments

Batch Normalization stabilizes layer-input distributions, accelerating training and improving accuracy on MNIST and ImageNet. On Inception, it enables much larger learning rates, faster convergence, sigmoid training, and state-of-the-art ImageNet results.

MNIST: Batch Normalization stabilizes evolving sigmoid-input distributions and yields higher MNIST test accuracy than the original network during training.The original network’s activation distributions change substantially in mean and variance, whereas batch-normalized distributions remain much more stable.
Inception modifications: The optimized batch-normalized Inception configuration increases the learning rate, removes Dropout and local response normalization, reduces L2 regularization fivefold, and shuffles examples more thoroughly.Within-shard shuffling improves validation accuracy by about 1%, while the other changes target faster training or improved accuracy.
ImageNet training speed: BN-x5 reaches Inception’s 72.2% accuracy in 14 times fewer steps, while BN-x30 reaches 74.8% after 6·10^6 steps, five times fewer than Inception.Figure 2 compares single-crop validation accuracy across Inception and batch-normalized variants during training.
Sigmoid networks: BN-x5-Sigmoid achieves 69.8% accuracy, whereas Inception with sigmoid without Batch Normalization remains at no better than 1/1000 accuracy.This demonstrates that Batch Normalization permits training deep networks with sigmoid nonlinearities despite their known training difficulty.
ImageNet benchmark: 4.9% top-5 validation error and 4.82% test error improve upon the previous best ImageNet result and exceed estimated human-rater accuracy.Figure 4 reports new state-of-the-art performance on ImageNet classification benchmarks by a healthy margin.

5 Conclusion

Batch Normalization accelerates deep-network training by stabilizing activation distributions through architectural normalization. It reaches and surpasses prior image-classification performance, while motivating further study in recurrent networks and gradient propagation.

5 Conclusion: 4.82% top-5 test error was achieved by the BN-Inception ensemble on 100,000 ImageNet test images.The ensemble result was reported by the ImageNet test server.
5 Conclusion: Batch Normalization accelerates training by normalizing activations within the network architecture, with normalization compatible with stochastic optimization.The method targets covariate shift in sub-networks and layers and incorporates normalization directly into the architecture.
5 Conclusion: Adding Batch Normalization substantially speeds training, while higher learning rates, removing Dropout, and related modifications enable surpassing single-network state of the art with few training steps.These changes are described as benefits afforded by Batch Normalization.
5 Conclusion: Batch Normalization stabilizes activation distributions during training and differs from standardization layers through learned scale and shift, convolutional handling, deterministic inference, and layerwise convolutional normalization.The method is applied before nonlinearities to support stable first and second moments.
5 Conclusion: Future work will apply Batch Normalization to recurrent neural networks, where internal covariate shift and vanishing or exploding gradients may be especially severe.These studies would further test whether normalization improves gradient propagation.

Appendix · Variant of the Inception Model Used

The appendix describes the Inception variant relative to GoogLeNet, documenting architectural changes that trade greater depth and capacity for higher computational cost while introducing efficiency-oriented convolutions.

Variant of the Inception Model Used: Figure 5 documents the architecture changes made relative to GoogLeNet, with interpretation requiring Szegedy et al. (2014).
Variant of the Inception Model Used: The variant replaces each 5×5 convolution with two consecutive 3×3 convolutions, increasing maximum depth by 9 weight layers, parameters by 25%, and computational cost by about 30%.
Variant of the Inception Model Used: The architecture increases the number of 28×28 Inception modules from 2 to 3.
Variant of the Inception Model Used: Pooling within modules alternates between average pooling and maximum pooling, as indicated in the architecture table.
Variant of the Inception Model Used: Instead of pooling between every pair of Inception modules, the variant uses stride-2 convolution or pooling before filter concatenation in modules 3c and 4e.
Variant of the Inception Model Used: The first convolutional layer uses separable convolution with depth multiplier 8, reducing computational cost while increasing training-time memory consumption.