Understanding Batch Normalization

1 Introduction

• BN prevents activation explosion by repeatedly normalizing activations, enabling larger learning rates that speed convergence along flat directions and reduce the likelihood of getting stuck in sharp minima.Without BN, a small subset of activations can explode early in divergence; smaller learning rates avoid this but yield less progress along flat directions.Source: §1 · pp. 1–2
• The experiments examine viable learning-rate ranges, depth-related activation growth, and how random initialization affects gradients and robustness to ill-conditioned weights.The paper also reports that activations and gradients in deep networks without BN tend to be heavy-tailed.Source: §1 · pp. 1–2

1.1 The Batch Normalization Algorithm

• BN normalizes activations channel-wise across mini-batch examples and spatial locations, uses running mean and variance averages at test time, and then applies learned channel-wise scale and shift.The normalization divides centered activations by the standard deviation plus ϵ for numerical stability.Source: §1.1 · p. 2

1.2 Experimental Setup

• The study uses CIFAR-10 image classification with a 110-layer ResNet, SGD with momentum and weight decay, standard augmentation and preprocessing, and learning-rate drops when learning plateaus.It reports the best results across initial learning rates {0.1, 0.003, 0.001, 0.0003, 0.0001, 0.00003}, training for enough epochs to reach a plateau.Source: §1.2 · p. 2

2 Disentangling the benefits of BN

• At matched low learning rates, BN and unnormalized ResNets achieve comparable test accuracy, while larger rates improve BN accuracy but cause unnormalized networks to diverge.The unnormalized model requires α=0.0001 and roughly 2400 epochs; BN at α=0.003 takes 1320 epochs, and its low learning-rate schedule has a larger train-test gap than BN at α=0.1.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Learning rate and generalization

• The authors argue that higher learning rates add SGD noise similarly to smaller batches, biasing BN-trained networks toward wider minima and better generalization.The passage relates this proposed mechanism to evidence that larger mini-batches can lead to sharp minima that generalize poorly [25].Source: §2.1 · p. 3

3 Batch Normalization and Divergence

• At initialization, comparable parameters in unnormalized networks have larger, heavier-tailed gradients than in BN networks.For the illustrated layer, unnormalized-network gradients are almost two orders of magnitude larger.Source: §3 · p. 4, Figure 2 · p. 4
• Unnormalized networks reduce loss only with small gradient steps, whereas BN networks reduce loss across a far broader range of step sizes.The comparison examines gradient-direction loss changes during the first mini-batches, using learning rates 0.1 with BN and 0.0001 without BN.Source: §3 · p. 4, Figure 3 · p. 4
• BN prevents depth-wise activation explosions by correcting each channel and layer to zero mean and unit standard deviation, enabling larger learning rates.During a diverging update without BN, later-layer activation means and variances reached scales orders of magnitude above earlier layers; the authors argue normalization statistics must be updated every batch for this safeguard.Source: §3 · p. 5, Figure 4 · p. 5
• Updating BN’s normalization statistics only every other batch does not enable high learning rates, and shallower networks allow larger rates.The authors relate the shallower-network observation to having fewer layers through which activation explosions can propagate.Source: §3 · p. 5

4 Batch Normalization and Gradients

• A single BN layer at the final layer allowed learning rates up to 0.03 and yielded 90.1% final test accuracy, capturing two-thirds of the overall BN improvement.The layer normalized the output of the last residual block but no intermediate activation.Source: §4 · p. 5
• At initialization, unnormalized networks’ channel means and variances grow with depth, seemingly almost exponentially, while BN keeps them relatively constant.The growth is partly data independent, and some unnormalized output channels reach scales up to 102.Source: §4 · p. 5, Figure 5 · p. 6
• Without BN, final-layer gradients across images are highly correlated because the network tends to predict one class, producing large summed gradients that rectify this initial bias.With BN, positive gradients are distributed across classes and predictions depend more on the input.Source: §4 · p. 6, Figure 6 · p. 6

4.1 Gradients of convolutional parameters

• For convolutional weights at initialization, unnormalized networks’ absolute gradient sum and sum of absolute gradients agree within a factor of 2 or less, versus a factor of 102 with BN.The unnormalized summands tend to match in sign and be similar across examples and spatial dimensions, suggesting a large input-independent component limits the learning rate.Source: §4.1 · p. 7, Table 1 · p. 7
• At initialization, some input-output channel pairs have consistently larger convolutional-weight gradients than others, associated with channels that have large means.The paper argues that weights outgoing from channels with large means have larger gradients.Source: §4.1 · p. 7

5 Random initialization

• The paper argues that constant variance across channels is too weak a guarantee: even weights set to 0 or 100 with some probability can satisfy it.This challenges the assumption that Xavier initialization necessarily ensures smooth information flow between layers.Source: §5 · pp. 7–8
• The analysis models a linear network as a product of random weight matrices and applies a random-matrix result for Gaussian matrix products.The paper notes that this toy model abstracts away important features, while CNNs can be flattened into fully connected layers with shared weights.Source: §5 · p. 8
• Products of more random matrices become increasingly heavy-tailed, raising the condition number as network depth increases.The paper illustrates this trend theoretically and empirically for random matrix products.Source: §5 · p. 8, Figure 8 · p. 8, Figure 9 · p. 9
• A higher condition number slows gradient-descent convergence, requires a smaller learning rate, and increases gradient differences across subspaces.The paper connects these effects to lower learning rates in deeper unnormalized networks and differing channel scales in unnormalized networks.Source: §5 · pp. 8–9, Figure 7 · p. 8
• Xavier and Kaiming [16] initialization scale iid entries by n^-1/2, but the analysis suggests this scaling leaves matrices ill-conditioned regardless of constant factors.The paper argues that this makes robustness to initialization schemes important.Source: §5 · p. 9

• The authors argue that BN’s success can be explained without making the elimination of internal covariate shift its primary benefit, challenging the original BN paper’s explanation [23].Source: §6 · p. 9
• [34] proposes an initialization scheme that normalizes all layers; despite internal covariate shift remaining, it can replace BN with relatively small performance loss.Source: §6 · p. 9
• Studies [48] and [47] investigate relationships among network parameters, accuracy, and convergence speed; [48] argues that batch normalization is important for facilitating a phenomenon dubbed super convergence.Discussion of batch-normalization variants, random matrix theory, generalization, and further related work is deferred to Appendix A in the online version [4].Source: §6 · p. 9

• Batch normalization’s benefits are mainly mediated by larger learning rates, whose increased implicit regularization in SGD improves generalization.Source: §7 · p. 9
• Large parameter updates in unnormalized networks can make activation magnitudes grow dramatically with depth, limiting the learning rates they can use.Source: §7 · p. 9
• Unnormalized networks have large, ill-behaved outputs that produce gradients independent of the input.Source: §7 · p. 9
• Random matrix theory supports the conclusion that ill-conditioned activations are a natural consequence of random initialization.Source: §7 · p. 9

A.1 On internal covariate shift and other explanations

• Prior accounts linked normalization to biased updates from nonzero means, internal covariate shift [23], improved conditioning [42], or stability of concurrent updates.Internal covariate shift describes shifting features during training but lacks a precise formal definition; some papers found it persuasive [53] [35].Source: §A.1 · p. 13
• This paper takes a first-principles approach, arguing BN’s success can be explained without internal covariate shift while not claiming that the phenomenon does not exist.The authors note that its vague definition makes falsification difficult; earlier empirical work was a master’s thesis.Source: §A.1 · p. 13

A.2 Random matrix theory and conditioning

• Conditioning research spans linear systems, general optimization metrics, sigmoid-network Hessians, and random-matrix studies of neural networks [31] [8].The paper distinguishes its focus on products of random matrices, for which theory had recently developed, from prior broad neural-network studies.Source: §A.2 · p. 13
• Initialization research seeks to ease optimization and avoid exploding or vanishing gradients, with approaches including deep-network methods [55], mean-field information propagation [44], and structured unitary RNN layers [1].The passage also notes that [55] uses more sophisticated analysis than, while [44] argues for initialization near order-to-chaos phase transitions.Source: §A.2 · p. 13

A.3 Alternatives to Batch Normalization

• Batch normalization alternatives [42] [26] exist, but the original formulation remains popular and generally competitive for image classification; applying it to RNNs is problematic [28].RNN-specific schemes [9] [2] and independently computed worker statistics to reduce communication overhead [20] have also been proposed.Source: §A.3 · p. 13

A.4 Generalization in deep learning

• The work corroborates an emerging approximate equivalence between batch size and learning rate [49] [14] [24] [33] and relates SGD’s regularizing effects to generalization [25] [57] [6].Prior accounts variously emphasize small batches [25] or an optimal batch size, with [50] arguing that [25] studied sizes above that optimum.Source: §A.4 · pp. 13–14
• Other cited work on deep-learning generalization includes [5] [36], while [47] explores relationships among network parameters.Source: §A.4 · p. 14

B Details on experimental setup

• The experiments use a 110-layer CIFAR10 ResNet and select each architecture’s best test accuracy across six initial learning rates.Training uses SGD with momentum 0.9, weight decay 5e-4, batch size 128, Xavier initialization, and standard CIFAR10 preprocessing and augmentation; training begins at 165 epochs and extends when learning has not plateaued.Source: B Details on experimental setup · p. 14

C Distribution of the singular values

• The singular-value analysis parameterizes x using ϕ, with an injective relationship, and notes that the resulting density blows up as x approaches 0.The exposition follows and plots the parameter relationship for different numbers of multiplied matrices.Source: C Distribution of the singular values · p. 14, Figure 10 · p. 15

D Deriving variance of SGD update

• The SGD-update variance derivation uses independent batch-sampling indicators and the fact that the gradient estimate is unbiased.The discussion notes that its derivation is similar to and relates the learning-rate and batch-size relationship to [24].Source: D Deriving variance of SGD update · p. 15

E BN on top

• With batch normalization only at the network top, performance is somewhat weaker than with full batch normalization, using a learning rate of 0.03 and 330 epochs.Figures 11 and 12 show the training and test learning curves for this architecture.Source: E BN on top · p. 16, Figure 11 · p. 16, Figure 12 · p. 16

F The parts of batch normalization

• Removing mean and variance normalization has a more dramatic accuracy effect than removing the affine transformation, and both normalization components matter.The affine transformation improves final accuracy by roughly 0.6%; without normalization, smaller learning rates and longer training are needed, and the highest non-diverging learning rates perform best.Source: F The parts of batch normalization · p. 16, Figure 13 · p. 17, Table 2 · p. 17

G.1 The effect of non-zero means

• In unnormalized networks, channel means strongly affect weight-gradient scales, while batch normalization removes this dependence and makes channel gradients similarly scaled.Large negative means are associated with channels that are effectively dead after ReLU and extremely small gradients; large positive means are associated with larger gradients.Source: §G.1 · p. 17, Figure 14 · p. 18

G.2 Channel gradients

• In unnormalized networks, gradient magnitudes vary more in early layers; with batch normalization, gradients are similarly scaled across layers.The paper relates the early-layer variation to the greater depth ahead of those layers and the resulting worse conditioning.Source: §G.2 · p. 18, Figure 15 · p. 18

H Learning rates for other architectures

• The tested architectures include 32- and 56-layer ResNets and unnormalized networks, with candidate learning rates from 0.1 to 0.00003.Achievable rates are those at which networks neither diverge nor remain at random-guessing accuracy.Source: H Learning rates for other architectures · pp. 18–19
• Alternating normalization computes mean and variance every other batch and uses the latest moments on intervening batches.Source: H Learning rates for other architectures · p. 18
• Less deep ResNets enable larger learning rates than deeper ResNets.Table 3 reports the highest achievable learning rates for the miscellaneous architectures, but the supplied passage does not provide its cell values.Source: Table 3 · p. 19
• Warm-starting unnormalized networks allows them to achieve higher learning rates.The warm start is one epoch at learning rate 0.0001.Source: H Learning rates for other architectures · p. 18, Table 3 · p. 19

I Class-wise experiments

• Class-wise gradient masking isolates each class by zeroing the other classes’ top-layer loss gradients before backpropagation.The experiment tests whether parameter gradients are driven mainly by their influence on the activation for the class the network almost always guesses at initialization.Source: I Class-wise experiments · p. 19
• Gradients from the class the network almost always guesses are at least one order of magnitude larger than those from other classes.The paper links this dominance to increasing channel means with depth and the resulting preference for one class at initialization.Source: I Class-wise experiments · p. 19
• The initially dominant class accounts for most gradients in a representative convolutional kernel, with similar results in other layers.Figure 16 compares absolute parameter-gradient values across top-layer classes for the layer 52 kernel.Source: Figure 16 · p. 19, I Class-wise experiments · p. 19

J Kernel gradients experiments

• One class has about one order of magnitude larger convolutional-kernel gradients than other classes, with similar results in other layers.The comparison is for layer 52 across classes at the top layer.Source: Figure 17 · p. 20
• Kernel-parameter gradient distributions are not identical and are skewed far from zero; distributions for parameters with smaller gradients are typically centered around zero.The distribution is shown for a single parameter in layer 90.Source: Figure 18 · p. 20, J Kernel gradients experiments · p. 20

K Miscellaneous Experiments

• For individual parameters in layer 54, unnormalized networks show an almost straight-line relationship between the two gradient measures, whereas this relationship disappears and gradients become much smaller with normalization.The unnormalized pattern reflects similar gradients of outgoing activations across spatial dimensions and batch examples.Source: Figure 19 · p. 21
• As the number of independent 1000-by-1000 Gaussian matrix products grows, their condition number rises dramatically and their maximum singular value increases, but not at the rate expected for infinitely large matrices.The plotted means and standard deviations are over five runs.Source: Figure 20 · p. 21, Figure 21 · p. 22
• Across normalization schemes, performance generally degrades at smaller learning rates; layer normalization appears to require a slightly smaller rate for optimal performance.Training lasted 660 epochs at learning rates 0.1 and 0.01, 1320 epochs at 0.003, and 2400 epochs at 0.0001; some schemes required slightly more training than batch normalization to reach final performance.Source: Table 4 · p. 21
• Deep unnormalized networks develop increasingly nonzero-mean, multimodal activation distributions, while normalized networks remain unimodal with means near zero, even in deep layers.The unnormalized network’s higher-layer activation means and variances also shrink during early batches when trained at learning rate 0.0001, suggesting gradients point toward smaller activations.Source: Figure 22 · p. 22, Figure 23 · p. 22, Figure 24 · p. 23
• On the first minibatch, loss diverges for both normalized and unnormalized networks, but the increase is seemingly exponential without normalization and linear with it; this behavior is transient.Figure 3 shows that the divergent behavior is transient.Source: Figure 25 · p. 24