PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Batch Normalization

Johan Bjorck, Carla Gomes, Bart Selman, Kilian Q. Weinberger

arXiv:1806.02375v4cs.LGcs.AIstat.ML

TL;DR

BN’s improvements lack a settled mechanism, so the paper empirically investigates how normalization changes optimization in deep networks. It finds that BN primarily enables larger learning rates, which support faster convergence and better generalization while avoiding activation divergence.

  • Problem

    The paper addresses the lack of consensus about why BN accelerates training, enables higher learning rates, and improves generalization.

  • Method

    The authors conduct experiments comparing deep networks with and without BN across learning rates, training behavior, activations, gradients, and initialization.

  • Results

    BN primarily enables larger learning rates; unnormalized networks can exhibit exploding activations, diverging loss, and ill-behaved gradients that limit those rates.

  • Takeaways & Limitations

    The paper argues that BN’s faster convergence and better generalization are mainly mediated by the larger learning rates it makes viable.

Abstract

from arXiv · show

Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have established BN as a favorite technique in deep learning. Yet, despite its enormous success, there remains little consensus on the exact reason and mechanism behind these improvements. In this paper we take a step towards a better understanding of BN, following an empirical approach. We conduct several experiments, and show that BN primarily enables training with larger learning rates, which is the cause for faster convergence and better generalization. For networks without BN we demonstrate how large gradient updates can result in diverging loss and activations growing uncontrollably with network depth, which limits possible learning rates. BN avoids this problem by constantly correcting activations to be zero-mean and of unit standard deviation, which enables larger gradient steps, yields faster convergence and may help bypass sharp local minima. We further show various ways in which gradients and activations of deep unnormalized networks are ill-behaved. We contrast our results against recent findings in random matrix theory, shedding new light on classical initialization schemes and their consequences.

1 Introduction

The paper argues that BN’s main benefit is enabling larger learning rates, while preventing activation and gradient pathologies that constrain unnormalized deep networks.

  • Batch Normalization: BN extends input normalization to intermediate layers by normalizing activations across mini-batches.For convolutional networks, normalization is applied per channel and followed by learned affine parameters.
  • Unnormalized networks: Without BN, deep-network activations and gradients tend to become heavy-tailed, with a small subset of deep activations exploding during divergence.These instabilities motivate using smaller learning rates to avoid steep directions that cause divergence.
  • Learning rates and optimization: Larger learning rates accelerate progress along flat directions and may reduce convergence to sharp minima with poorer generalization.The paper links small learning rates to slower progress and greater susceptibility to sharp local minima.
  • BN’s mechanism: BN repeatedly corrects activations to zero mean and unit standard deviation, preventing uncontrolled growth and permitting larger learning-rate updates.The normalization acts as a safety precaution against activation explosion.
  • Experimental setup: The experiments use CIFAR10 image classification with a 110-layer ResNet, SGD with momentum, weight decay, augmentation, and learning-rate reductions on plateaus.The study compares networks with and without BN across multiple initial learning rates.

2 Disentangling the benefits of BN

The experiments disentangle BN’s benefits from normalization itself by comparing learning-rate schedules and examining how SGD noise relates to generalization.

  • Learning-rate dependence: α = 0.0001 is required for the unnormalized ResNet to converge, taking roughly 2400 epochs; BN with this schedule performs no better.Using α = 0.003 requires 1320 epochs for the BN network and produces intermediate results.
  • Learning-rate dependence: Matching low learning rates produce comparable testing accuracy, whereas larger rates improve BN networks and cause unnormalized networks to diverge.Figure 1 averages results over five runs and shows standard deviations as shaded regions.
  • SGD noise: SGD estimates the full-dataset gradient by averaging per-example gradients from a sampled mini-batch, introducing a typically unbiased noise term.The paper models sampling with replacement and separates the estimate into the true gradient plus noise.
  • Learning rate and generalization: SGD-step noise is affected by learning rate similarly to inverse mini-batch size, so higher learning rates may favor wider minima and better generalization.The paper uses this analogy to explain why BN’s higher learning rates can improve generalization.

3 Batch Normalization and Divergence

The paper investigates why unnormalized networks diverge at large learning rates and finds that BN makes training more tolerant by controlling gradients and activation growth.

  • Unnormalized networks are limited by divergence at large learning rates, motivating analysis of gradients and loss along update directions.The study focuses on early training, when divergence typically occurs within the first few mini-batches.
  • Gradients in unnormalized networks are larger and have heavier tails than corresponding gradients in BN networks.The contrast is illustrated for midpoint layer 55 at initialization.
  • BN networks reduce loss across a broader range of step sizes, whereas unnormalized networks diverge when gradient steps become larger.Relative loss is measured as new_loss/old_loss for representative mini-batches.
  • During a diverging update without BN, activation means and variances grow progressively with depth, causing the network output and loss to explode.The heatmap tracks channel moments as the update is applied; later layers reach much larger scales than earlier layers.
  • BN mitigates divergence by correcting each channel’s activations to zero mean and unit standard deviation, preventing lower-layer growth from propagating uncontrollably upward.This normalization is presented as the primary mechanism enabling higher learning rates.

4 Batch Normalization and Gradients

Without BN, activations and gradients become increasingly unstable with depth, producing biased, correlated, and uneven gradients. BN stabilizes activations and reduces these gradient pathologies, allowing substantially larger learning rates.

  • Activation growth: Unnormalized networks amplify channel means and variances with depth, reaching output scales up to 10^2 at initialization.Batch-normalized networks keep these moments relatively stable, while unnormalized networks show near-exponential growth in the plotted depth trend.
  • Output gradients: Without BN, large channel means bias the final classifier toward one class, producing highly correlated gradients across minibatch images.With BN, gradient dependence on the input is much larger and positive gradients are distributed across classes.
  • Convolutional gradients: For unnormalized convolutional networks, gradient summands largely share signs and are data independent, whereas BN makes their aggregate cancellation differ by about two orders of magnitude.The absolute sum and sum of absolute values are similar without BN but differ substantially with BN.
  • Convolutional gradients: Unnormalized convolutional gradients exhibit a low-rank channel structure, with some input/output-channel pairs consistently large and others extremely small.This pattern is less pronounced with batch normalization and is consistent with large channel means driving larger outgoing-weight gradients.

5 Random initialization

The paper connects instability in deep unnormalized networks to random initialization and the conditioning of products of random matrices. Increasing depth makes singular-value distributions more heavy-tailed, worsening conditioning and optimization behavior.

  • Initialization: Traditional initialization controls channel variance under random weights but does not prevent ill-conditioned behavior in deep unnormalized networks.The paper questions whether Xavier-style reasoning extends adequately to many-layer networks.
  • Random matrix theory: Products of independent Gaussian matrices develop increasingly heavy-tailed singular-value distributions as the number of factors grows.The theoretical and empirical distributions both become more heavy-tailed for deeper products.
  • Conditioning: Greater depth increases the ratio between the largest and smallest singular values, thereby increasing the condition number of the linearized network.The paper interprets this as a conditioning consequence of multiplying more random matrices.
  • Conditioning: Higher condition numbers are associated with slower convergence, smaller required learning rates, and greater gradient imbalance across subspaces.These numerical-optimization effects parallel the behavior observed in deep learning.

6 Related Work

The paper situates its explanation of BN among competing accounts, including internal covariate shift, update stability, and conditioning. It argues that BN’s primary benefits can be explained without relying on internal covariate shift.

  • Related explanations: The original BN account attributes its benefits to internal covariate shift, while other work proposes improved update stability or conditioning.The paper does not claim internal covariate shift is absent, but disputes its status as the primary explanation.
  • Scope: The authors defer discussion of normalization variants, random matrix theory, generalization, and further related work to the online appendix.

7 Conclusions

The paper concludes that BN’s benefits are mainly mediated by larger learning rates, which improve optimization and generalization while avoiding instability in deep unnormalized networks.

  • Conclusion: BN benefits are mainly mediated by larger learning rates, whose implicit regularization improves generalization.
  • Conclusion: Large updates in unnormalized networks can cause activation magnitudes to grow dramatically with depth, limiting viable learning rates.
  • Conclusion: Unnormalized networks also produce large, ill-behaved outputs and input-independent gradients, which random matrix theory links to random initialization.

Appendices

The appendices review competing explanations for BN, alternatives and application constraints, and related work connecting batch size, learning rates, SGD regularization, and generalization.

  • Related explanations: The paper contrasts internal covariate shift with explanations based on improved conditioning and stability of concurrent updates.It does not claim internal covariate shift is absent, but argues BN’s success can be explained without it.
  • Alternatives and scope: Many alternatives to BN exist, but the original formulation remains popular and generally competitive for image classification.Applying BN to recurrent neural networks is problematic, motivating specialized normalization schemes.
  • Related optimization findings: The work corroborates reported links among batch size, learning rates, SGD’s regularizing effects, and generalization.Prior work gives differing accounts of whether smaller or optimally sized batches are most beneficial.

B Details on experimental setup

These appendices detail the CIFAR10 ResNet experiments, singular-value analysis, partial BN tests, and investigations of normalization, affine transformations, and gradient behavior.

  • B Details on experimental setup: Experiments use CIFAR10 with a 110-layer ResNet, SGD with momentum and weight decay, batch size 128, standard preprocessing, and data augmentation.The setup follows the original ResNet study and reduces learning rates when training plateaus.
  • B Details on experimental setup: The experiments compare initial learning rates from 0.1 to 0.00003 and report each architecture’s best terminating test accuracy.Training duration and learning-rate schedules are adjusted when BN is removed.
  • B Details on experimental setup: CIFAR10 experiments use a lightweight dataset and a familiar ResNet architecture to permit detailed manual analysis of the architecture and data.The implementation uses PyTorch on AWS p3.2xlarge instances, with source code planned for public release.
  • C Distribution of the singular values: For products of M random matrices, the singular-value density is parameterized by ϕ through a strictly decreasing, one-to-one relation with x.The resulting density becomes unbounded as x approaches zero.
  • E BN on top: BN applied only near the network top achieves somewhat weaker performance than complete BN, using learning rate 0.03 and 330 epochs.The top placement follows the residual blocks and precedes average pooling into the fully connected layer.
  • F Components of BN: Removing BN’s affine transformation improves final accuracy only slightly, while the experiments separately assess mean-and-variance normalization.Figure 13 compares these components in a 110-layer ResNet.
  • G.1 The effect of non-zero means: Because weight gradients scale linearly with preceding activations, nonzero channel means create gradient disparities that BN largely removes.The relationship is examined using channel means and average gradients for convolutional-kernel parameters.
  • G.2 Conditioning: Gradient magnitudes differ across convolutional channels in unnormalized networks, whereas BN produces more similar scaling across layers.The appendix relates this behavior to singular subspaces and conditioning in the linearized network.

H Learning rates for other architectures

Experiments across architectures show that viable learning rates depend on depth, initialization strategy, and normalization, while gradient influence can concentrate in one output class.

  • H Learning rates for other architectures: Table 3 evaluates highest non-divergent learning rates across ResNet depths, warm starts, and alternating normalization.Warm-starting unnormalized networks tests whether training can rectify initialization artifacts.
  • H Learning rates for other architectures: A batch-normalized network does not diverge at learning rate 3.0 but diverges at 10.0.The reported criterion excludes networks that diverge or remain at random-guessing accuracy.
  • H Learning rates for other architectures: Less deep ResNets enable larger learning rates, and warm-starting unnormalized networks also permits higher learning rates.These comparisons use rates drawn from {0.1, 0.003, 0.001, 0.0003, 0.0001, 0.00003}.
  • I Class-wise experiments: Class-wise gradient masking shows that one output class accounts for most gradients in a representative convolutional kernel.The influential class is the class the unnormalized network tends to guess at initialization.

J Kernel gradients experiments

The kernel-gradient experiments examine how gradient contributions are distributed across parameters and classes in unnormalized and batch-normalized networks.

  • J Kernel gradients experiments: The experiments compare per-parameter gradient quantities in convolutional kernels, including a representative parameter in layer 90.The analysis uses the gradient decomposition introduced in equation (4).
  • J Kernel gradients experiments: One class produces about an order of magnitude larger average absolute gradients than other classes across layer-52 kernel parameters.Similar class-wise effects are reported for other layers.
  • J Kernel gradients experiments: The appendix uses scatterplots to examine the relationship between summed gradient contributions and individual parameter gradients in normalized and unnormalized networks.The comparison is made for all parameters within a convolutional kernel.
  • J Kernel gradients experiments: The gradient-contribution distribution is skewed far from zero rather than identical across parameters.Parameters with smaller gradients typically have distributions centered around zero.

K Miscellaneous Experiments

These experiments examine matrix conditioning, activation distributions, loss behavior, gradient structure, and normalization schemes. Unnormalized networks develop increasingly irregular activations and gradients, while normalized networks maintain more stable activation distributions and smaller gradients.

  • Gradient behavior: For layer 54, unnormalized-network gradients form an almost straight-line parameter relationship, whereas normalization removes this relationship and makes gradients much smaller.The unnormalized pattern reflects similar gradients of outgoing activations across spatial dimensions and examples within a batch.
  • Random matrix behavior: Condition numbers of products of independent random square matrices increase dramatically as the number of products grows.The matrices have size 1000 with independent Gaussian entries, and the plot uses a logarithmic scale.
  • Normalization schemes: Across normalization schemes, performance generally degrades at smaller learning rates; layer normalization appears to require a slightly smaller optimal learning rate than batch normalization.Training durations vary by learning rate, ranging from 660 epochs at 0.1 and 0.01 to 2400 epochs at 0.0001.
  • Random matrix behavior: Maximum singular values increase with the number of matrix products, though not at the linear rate expected for infinitely large matrices.Means and standard deviations are reported over five runs.
  • Activation behavior: Deeper layers of an unnormalized network show activation distributions with increasingly nonzero means and multimodality, unlike normalized networks.For the normalized network, distributions remain unimodal with means close to zero even in deep layers.
  • Activation behavior: In an unnormalized network trained with learning rate 0.0001, activation means and variances shrink during the first batches, especially in higher layers.The observed moment changes suggest that gradients point toward smaller activations.
Loading 1806.02375v4…