PaperlayerView exact revision on arXiv

Source-linked AI summary

How Does Batch Normalization Help Optimization?

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry

arXiv:1805.11604v5stat.MLcs.LGcs.NE

TL;DR

BatchNorm’s effectiveness is widely recognized, but the connection between its success and internal covariate shift remains poorly supported. The paper investigates this connection and instead identifies optimization-landscape smoothing as a key effect that makes gradients more predictive and training faster.

  • Problem

    The paper addresses the poorly understood basis of BatchNorm’s effectiveness, especially the limited evidence connecting its performance gains to internal covariate shift.

  • Method

    The authors compare BatchNorm with standard and deliberately distributionally unstable networks, analyze gradient-based internal covariate shift, and provide empirical and theoretical optimization-landscape analyses.

  • Results

    BatchNorm’s performance is tenuously related to internal covariate shift, while its reparametrization improves loss stability and effective β-smoothness, making gradients more predictive and optimization faster.

  • Takeaways & Limitations

    Optimization-landscape smoothing offers a more fundamental account of BatchNorm’s training benefits and is also observed with several other natural normalization strategies.

  • Takeaways & Limitations

    The paper focuses on training; whether its smoothing explanation accounts for BatchNorm’s tendency to improve generalization remains a possibility for future work.

Abstract

from arXiv · show

Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training.

1 Introduction

BatchNorm is widely used to improve neural-network training, but its effectiveness remains poorly understood. This paper challenges the prevailing internal-covariate-shift explanation and identifies optimization-landscape smoothness as a more fundamental mechanism.

  • BatchNorm is a prominent technique for improving neural-network training and has become widely adopted in research and real-world models.
  • The prevailing explanation attributes BatchNorm’s success to reducing internal covariate shift by stabilizing layer-input distributions.ICS is defined as changes in layer-input distributions caused by updates to preceding layers.
  • Concrete evidence linking internal covariate shift to training performance remains limited, motivating this paper’s investigation.
  • The authors find no clear link between BatchNorm’s performance gains and reduced internal covariate shift, which may not decrease under some measures.
  • BatchNorm makes the optimization landscape smoother, producing more predictive gradients, permitting larger learning-rate ranges, and accelerating convergence.The paper theoretically analyzes improved Lipschitzness of both loss and gradients, also called β-smoothness.
  • Other natural normalization techniques can produce similar or stronger smoothing effects and comparable training improvements.

2 Batch normalization and internal covariate shift

The paper tests whether BatchNorm’s benefits arise from stabilizing layer-input distributions or reducing internal covariate shift. Experiments instead show that performance can remain strong amid distributional instability, and gradient-based ICS may even increase.

  • BatchNorm normalizes layer inputs by controlling their mini-batch means and variances, then typically applies trainable scaling and shifting.
  • VGG networks with BatchNorm show faster training and better performance than standard networks, despite less pronounced differences in layer-input distribution evolution.
  • The study asks whether BatchNorm’s effectiveness is related to internal covariate shift and whether input-distribution stabilization actually reduces it.
  • 2.1 Does BatchNorm’s performance stem from controlling internal covariate shift?: Adding time-varying, non-zero-mean and non-unit-variance noise after BatchNorm creates deliberate distributional instability for each activation and time step.
  • 2.1 Does BatchNorm’s performance stem from controlling internal covariate shift?: The noisy BatchNorm model nearly matches standard BatchNorm and outperforms the standard network, undermining distributional stability as the source of BatchNorm’s gain.
  • 2.1 Does BatchNorm’s performance stem from controlling internal covariate shift?: The same noise prevents the standard non-BatchNorm network from training, highlighting that comparable distributional instability has different consequences across architectures.
  • 2.2 Is BatchNorm reducing internal covariate shift?: The broader optimization-based ICS measure compares a layer’s gradients before and after updates to preceding layers.
  • 2.2 Is BatchNorm reducing internal covariate shift?: BatchNorm networks often exhibit increased gradient-based ICS, even while achieving substantially better accuracy and loss than standard networks.

3 Why does BatchNorm work?

The paper argues that BatchNorm’s benefit is not explained by reduced internal covariate shift, but by smoothing the optimization landscape. Smoother landscapes make gradients more predictive, support larger learning rates, and accelerate training; related normalization schemes can produce similar effects.

  • Why does BatchNorm work?: BatchNorm’s optimization benefit is only tenuously linked to internal covariate shift, which can even increase despite improved training performance.In deep linear networks, BatchNorm showed substantial internal covariate shift while standard networks showed almost none, yet BatchNorm still achieved better accuracy and loss.
  • The smoothing effect of BatchNorm: BatchNorm reparametrizes the optimization problem, making the loss landscape smoother and its gradients more Lipschitz.The paper distinguishes smoother loss changes from improved effective β-smoothness, where gradient changes are also reduced.
  • The smoothing effect of BatchNorm: Smoother gradients are more reliable and predictive, allowing larger learning rates and faster, less hyperparameter-sensitive training.The authors connect improved gradient Lipschitzness to taking larger steps without sudden transitions into flat regions or sharp minima.
  • Exploration of the optimization landscape: Gradient-direction measurements show substantially greater gradient predictiveness for BatchNorm, with differences close to two orders of magnitude early in training.The study also finds consistent effective β-smoothness differences and observes similar behavior in random directions and deep linear networks.
  • Is BatchNorm the best (only?) way to smoothen the landscape: Normalization strategies based on ℓp norms achieve performance comparable to BatchNorm and similar landscape smoothness, despite producing larger distributional shifts.For deep linear networks, ℓ1-normalization performs even better than BatchNorm, motivating broader exploration of normalization design.

4 Theoretical Analysis

The theoretical analysis compares otherwise identical networks with and without a BatchNorm layer inserted after a fully connected layer. It shows that BatchNorm improves activation- and weight-space landscape properties, including Lipschitzness, gradient predictability, smoothness, and initialization.

  • Setup: The analysis inserts one BatchNorm layer after an arbitrary fully connected layer and compares the resulting landscape with the original network’s landscape.The comparison isolates effects of landscape reparametrization rather than merely input normalization.
  • Setup: The two compared networks use identical layer parameters, inputs, and arbitrary loss functions, while BatchNorm standardizes layer outputs before scaling and shifting them.The normalized outputs have mean 0 and variance 1 before multiplication by γ and addition of β.
  • Theoretical Results: BatchNorm makes the activation-space landscape more well-behaved, improving Lipschitz continuity and gradient predictability, with corresponding favorable worst-case weight-space bounds.The analysis first studies normalized activations and then translates those improvements to layer weights.
  • Theoretical Results: BatchNorm reduces the loss landscape’s Lipschitz constant under conditions involving activation-gradient correlation or nonzero gradient mean, including an additive reduction beyond simple rescaling.The reduction can persist even when the BatchNorm scaling equals the original layer scaling, σj = γ.
  • Theoretical Results: BatchNorm decreases a Hessian quadratic-form term and rescales it by input variance, making gradients more predictive and the landscape more resilient to mini-batch variance.The term represents the second-order contribution in the Taylor expansion of the gradient.
  • Theoretical Results: BatchNorm preserves the minima of the original landscape through an appropriate configuration and provides a favorable initialization property under the stated inner-product condition.For any input data and network configuration, a BatchNorm configuration with γ = σj can reproduce the same activations; the initialization result is stated for ⟨W0, W*⟩ > 0.

5 Related work

Related work identifies additional BatchNorm benefits, alternative normalization schemes, and optimization methods that exploit weight direction–length decoupling. These lines of work motivate, but do not replace, the paper’s focus on BatchNorm’s optimization landscape.

  • Related work: Alternative normalization methods normalize across layers, batch subsets, image dimensions, or weights, while ELU and SELU offer alternative nonlinearities.The alternatives differ in whether they normalize activations, weights, or modify nonlinear activation slopes.
  • Related work: Prior studies report that BatchNorm can reduce dependence on initialization and mitigate weak correlations among gradient coordinates or unit activations.The reported correlation issues are especially pronounced in deeper models without BatchNorm.
  • Related work: Other work exploits BatchNorm- and Weight-Normalization-like decoupling of weight length and direction to obtain faster convergence with separate adaptive step sizes.The approach designs optimization algorithms that update these parameters separately.

6 Conclusions

BatchNorm’s effectiveness is not well explained by internal covariate shift; instead, it reparametrizes optimization to produce a smoother, more stable landscape and more predictive gradients. The authors also suggest a possible connection between this smoothening effect and generalization, while leaving that question for future work.

  • BatchNorm’s performance is not well predicted by internal covariate shift, which may not even be reduced from an optimization perspective.
  • BatchNorm reparametrizes optimization to make the loss more stable and effectively smoother, yielding more predictive gradients and faster optimization.
  • Other natural normalization strategies can produce similar or stronger smoothening effects and comparable training-performance improvements.
  • The findings challenge conventional explanations of BatchNorm and motivate more systematic study of deep-learning algorithms and their effectiveness.
  • The authors propose that smoothening may encourage convergence to flatter minima, which are believed to facilitate better generalization, but identify this as future work.

A Experimental Setup

The experimental setup section points readers to architectural details and to the setup and measurements used in the experiments.

  • Architectural details are provided in Section A.1, while experimental setup and measurement specifics are discussed in Section A.2.

A.1 Models

The study uses a VGG-like CIFAR10 classifier and a 25-layer deep linear network on synthetic Gaussian regression data to examine BatchNorm across nonlinear and linear settings.

  • The experiments use a VGG-like network and a deep linear network, and both show clear performance benefits with BatchNorm.
  • The VGG model is trained on CIFAR10 with stochastic gradient descent for 15,000 steps, using batch size 128 and learning rate 0.1 unless specified otherwise.
  • The deep linear network solves a factorized regression problem with 25 fully connected layers and an ℓ2 loss.
  • For the deep linear network, inputs are Gaussian, the target matrix is 10 × 10, training uses full-batch gradient descent for 10,000 steps, and the learning rate is 10^-6 unless specified otherwise.
  • Both networks use standard Glorot initialization, with learning rates selected through hyperparameter optimization for best training performance.

A.2.1 “Noisy” BatchNorm Layers

The experiments define BatchNorm activation distributions over batch-related dimensions, evaluate noisy normalization variants, and probe local loss landscapes by measuring losses after gradient-direction steps.

  • “Noisy” BatchNorm Layers: BatchNorm fixes the mean and variance of each unit’s activations across examples, with these moments possibly learnable.
  • “Noisy” BatchNorm Layers: The noisy BatchNorm model perturbs activations with time-varying, independently sampled noise having non-zero mean and non-unit variance.
  • “Noisy” BatchNorm Layers: The study excludes ResNets because they provide benefits similar to BatchNorm and could introduce confounding factors.
  • “Noisy” BatchNorm Layers: Activation distributions are plotted across the batch dimension for fully connected layers and across batch, height, and width for convolutional layers.
  • Loss Landscape: Loss-landscape smoothness is evaluated, rather than trained, by measuring losses after steps of different lengths in the gradient direction.
  • Loss Landscape: Gradient-direction step lengths range from [1/2, 4] × step size for VGG and [1/100, 30] × step size for the deep linear network.

B Omitted Figures

The figures compare BatchNorm and alternative normalization strategies through internal covariate shift, activation distributions, training performance, and optimization-landscape smoothness. Across these comparisons, improved optimization is associated with smoother landscapes rather than more stable activation distributions.

  • Internal covariate shift: BatchNorm models can have similar or worse internal covariate shift while achieving better accuracy and loss.ICS is measured using gradient cosine angles and ℓ2 differences before and after updates to preceding layers.
  • Internal covariate shift: Noisy BatchNorm produces significantly higher internal covariate shift than standard networks yet performs better from an optimization perspective.The comparison tracks changes in activation means and variances between successive training steps.
  • Optimization landscape: BatchNorm improves the smoothness measures of the optimization landscape in deep linear networks, including loss variation, gradient change, and effective β-smoothness.The analysis evaluates movement in the gradient direction at particular training steps.
  • Gradient behavior: With BatchNorm, gradients are significantly more predictive and change slowly within a local neighborhood along the gradient direction.Predictiveness is assessed by the ℓ2 error between the current gradient and gradients observed after moving along its direction.
  • Alternative normalizations: All tested ℓp normalization strategies improve performance and loss-landscape smoothness over standard training in deep linear networks.The figure reports the same evaluation structure as the VGG experiments.
  • Alternative normalizations: In VGG networks, ℓp normalization strategies perform comparably or better than BatchNorm, despite causing larger distributional covariate shift.The comparisons include training performance, loss variation, gradient changes, and maximum β-smoothness.

C Proofs

This section establishes basic facts about gradients through BatchNorm that are then used to prove the paper’s optimization-landscape theorems. The setup relies on BatchNorm’s closed-form expressions and partial-derivative notation.

  • Proof setup: The proofs begin with facts derived directly from BatchNorm’s closed form.These facts are used freely in the subsequent theorems.
  • Proof setup: The theorem proofs use the setup from Figure 5 and express derivatives with partial-derivative notation instead of gradient notation.The setup also relies on several simple but key facts.
  • Gradient identities: Fact C.1 gives the gradient through BatchNorm when the normalized output is scaled by γ and shifted by β.It defines the relevant composition using BatchNorm followed by another function.
  • Gradient identities: Fact C.2 provides a convenient expression for gradients of normalized outputs.This identity is used in the later smoothness analysis.

C.2 Lipschitzness proofs

The proofs formalize how BatchNorm changes the loss landscape’s Lipschitzness and smoothness, under stated assumptions on the loss Hessian. They also derive bounds for weight-space behavior, gradient predictiveness, and initialization.

  • Proof strategy: The proofs use BatchNorm gradient identities, chain-rule expansions, operator-norm bounds, and Hessian calculations to establish the stated results.Intermediate steps derive gradients with respect to weights and rewrite the Hessian and smoothness expressions.
  • Loss-landscape Lipschitzness: Theorem 4.1 compares the loss-landscape Lipschitzness of a BatchNorm network with an otherwise identical non-BatchNorm network.The two networks are defined with identical losses.
  • Loss-landscape smoothness: Theorem 4.2 states BatchNorm’s effect on smoothness using the gradient and Hessian of the loss with respect to layer outputs.Its additional conclusion requires the Hessian to preserve relative norms of the relevant gradients.
  • Loss-landscape smoothness: The proof evaluates effective β-smoothness through the quadratic form g⊤Hg for the activation gradient g and Hessian H.The calculation expands this product using identities for the normalized activations and gradients.
  • Minimax bounds: The derivation obtains minimax bounds for Lipschitzness in activation and weight spaces under the theorem’s stated conditions.The weight-space result is introduced as Theorem 4.4, while the minimax smoothness result is Theorem C.1.
  • Gradient predictiveness: The argument also analyzes gradient predictiveness using the derived BatchNorm gradient and maximization over the input matrix.The preceding smoothness proof is then applied to conclude the bound.
  • Initialization: BatchNorm’s scale invariance under positive weight rescaling supports a favorable-initialization result relative to corresponding standard-network optima.The proof maps each standard-network optimum to a scaled BatchNorm optimum and compares distances from initialization.
Loading 1805.11604v5…