PaperlayerView exact revision on arXiv

Source-linked AI summary

Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

Devansh Arpit, Yingbo Zhou, Bhargava U. Kota, Venu Govindaraju

arXiv:1603.01431v6stat.MLcs.LG

TL;DR

Deep-network training is hindered by shifting hidden-layer input distributions and Batch Normalization’s reliance on potentially unrepresentative mini-batch statistics for validation. Normalization Propagation uses data-independent parametric estimates propagated through hidden layers, achieving more stable convergence than Batch Normalization, especially early in training.

  • Problem

    Shifting hidden-layer input distributions slow training convergence, while Batch Normalization relies on mini-batch statistics that may not represent the full data distribution during validation.

  • Method

    Normalization Propagation uses closed-form, data-independent normalization estimates and propagates them through hidden layers under Gaussian pre-activation and roughly incoherent-weight assumptions.

  • Results

    Normalization Propagation achieves more stable convergence than Batch Normalization, particularly during initial training.

  • Takeaways & Limitations

    Normalization Propagation avoids moving-average batch statistics for validation and enables training with batch size 1.

  • Takeaways & Limitations

    Batch Normalization’s validation estimates may be unrepresentative because they rely on mini-batch statistics affected by changing preceding-layer parameters.

Abstract

from arXiv · show

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of mean and standard deviation of input (distribution) to hidden layers inaccurate for validation due to shifting parameter values (especially during initial training epochs). Also, BN cannot be used with batch-size 1 during training. We address these drawbacks by proposing a non-adaptive normalization technique for removing internal covariate shift, that we call Normalization Propagation. Our approach does not depend on batch statistics, but rather uses a data-independent parametric estimate of mean and standard-deviation in every layer thus being computationally faster compared with BN. We exploit the observation that the pre-activation before Rectified Linear Units follow Gaussian distribution in deep networks, and that once the first and second order statistics of any given dataset are normalized, we can forward propagate this normalization without the need for recalculating the approximate statistics for hidden layers.

1. Introduction and Motivation

Internal Covariate Shift arises when hidden-layer input distributions shift, complicating learning a fixed P(Y | X); Normalization Propagation addresses this with batch-independent, closed-form normalization propagated through hidden layers.

  • Internal Covariate Shift shifts hidden-layer input distributions, making learning a fixed P(Y | X) difficult when P(X) changes continuously.
  • Batch Normalization addresses this shift by estimating each layer’s pre-activation mean and standard deviation from mini-batch statistics during training.Its validation estimates are affected by shifting parameters.
  • Normalization Propagation uses a parametric normalization technique that does not depend on batch statistics and is less affected by parameter shifts during validation.
  • The method obtains data-independent normalization components in closed form assuming Gaussian pre-activations and roughly incoherent hidden-layer weight matrices.
  • Normalization Propagation forwards the data distribution’s normalization property through all hidden layers using Gaussian pre-activation knowledge and algebraic manipulations.

2. Background

Input whitening and decorrelation were recognized as ways to accelerate deep-network training, with prior work recommending normalization before every layer. Batch Normalization (Ioffe & Szegedy, 2015) instead normalizes hidden-layer pre-activations independently using mini-batch statistics, but its estimates depend on shifting network parameters.

  • Input whitening and decorrelation can speed deep-network training, and LeCun et al. (2012) recommend applying whitening before every layer to produce zero-mean inputs.
  • Batch Normalization (Ioffe & Szegedy, 2015) normalizes hidden-layer pre-activations independently for each unit using mini-batch statistics, avoiding full whitening.
  • Because W participates in the normalization, it also becomes part of the gradient-descent step.
  • Both normalization approaches rely on approximations affected by shifting network parameters.

3. Normalization Propagation (NormProp) Derivation

NormProp propagates a canonical input distribution through hidden layers by normalizing pre-activations with weight norms and post-ReLU activations with analytically determined statistics. This construction approximately yields dynamical isometry, with Jacobian singular values near 1, helping avoid exploding or diminishing gradients.

  • Normalization Propagation: For normalized inputs, the pre-activation covariance is approximately diagonal when the weight matrix has a tight coherence-dependent error bound, enabling unit-variance scaling by each weight vector’s ℓ2 norm.Normalizing by ∥W_i∥2 effectively gives each hidden weight vector unit ℓ2 length, leaving coherence as the determinant of the approximation error.
  • Normalization Propagation: Analytical mean and standard-deviation values for the Rectified Gaussian output let NormProp standardize ReLU activations without explicit data-dependent normalization.After pre-activation normalization, ReLU outputs are Rectified Gaussian; applying its calculated statistics restores the dynamics used for the next layer.
  • Normalization Propagation: NormProp recursively makes every layer’s input approximately canonical by dividing pre-ReLU activations by corresponding weight norms and standardizing post-ReLU activations.The same two normalization operations are repeated layer by layer and back-propagated during training.

4. NormProp: Implementation Details … 4.3. Propagate Normalization

NormProp propagates input normalization through ReLU hidden layers without hidden-layer batch statistics, supporting fully connected and convolutional networks with learnable scaling and bias parameters. Its implementation specifies data-normalization choices, parameter initialization, and layer-specific propagation rules.

  • 4. NormProp: Implementation Details: NormProp propagates normalization through hidden layers and can accompany optimization algorithms including stochastic gradient descent with or without momentum.
  • 4.2. Initialize Network Parameters: Weights use Normalized Initialization, biases start at zero, and scaling vectors start at one or receive the layer-specific initialization.
  • 4.3. Propagate Normalization: The method uses learnable scaling and bias parameters γ and β, analogous to Batch Normalization.
  • 4.3.1. FULLY CONNECTED LAYERS: For fully connected layers, NormProp replaces the traditional output computation with a normalization-aware transformation using the layer’s weights, bias, scaling, input, and ReLU activation.
  • 4.3.1. FULLY CONNECTED LAYERS: Fully connected scaling factors initialize to 1/1.21 to keep the ReLU Jacobian near one and enable larger learning rates without divergence.The initialization may alternatively be treated as a hyper-parameter.
  • 4.3.2. CONVOLUTIONAL LAYERS: For convolutional layers, NormProp applies normalization-aware filtering, broadcasts each filter’s scaling and bias across outputs, and performs pooling afterward as usual.
  • 4.1. Normalize Data: NormProp performance is unaffected by choosing global or batch data normalization at the input.Global normalization uses dataset-wide feature statistics, whereas batch normalization is proposed for streaming data; hidden layers remain normalized by NormProp.

4.4. Training

Training uses backpropagation with layerwise normalization gradients, momentum SGD, adaptive learning-rate reductions, and weight constraints. Weight decay is used, whereas Dropout is excluded because it can disrupt the output distributions required by NormProp.

  • 4.4. Training: Backpropagation trains the network while propagating normalization gradients through every layer.
  • 4.4. Training: Optimization uses stochastic gradient descent with momentum 0.9, shuffled data, and halved learning rates when training error saturates.Larger initial learning rates improve performance for larger batch sizes.
  • 4.4. Training: After each iteration, every convolutional and fully connected hidden weight vector is constrained to unit ℓ2 length to reduce parameter-search space.The constraint is motivated by weight scale not affecting network representation.
  • 4.4. Training: Training applies weight decay with coefficient 0.0005 − 0.005 but omits Dropout because changing layer-output distributions can affect NormProp.

4.5. Validation and Testing

NormProp uses training-consistent, data-derived normalization statistics during validation and testing before forwarding samples through the learned network. On CIFAR-10, it produces more stable lower-layer hidden-input distributions and convergence closer to zero than BN.

  • Validation and Testing: NormProp normalizes each validation/test sample using statistics consistent with the training normalization strategy, then forwards it through the network with learned parameters.Global Data Normalization reuses the global training mean and standard deviation, whereas Batch Data Normalization uses a running estimate.
  • Validation and Testing: On CIFAR-10, NormProp achieves more stable lower-layer hidden-input distributions and overall convergence closer to zero than BN.The comparison uses the validation-set mean of a randomly chosen unit across hidden layers in a 9-layer convolutional network.

4.6. Extension to other Activation Functions

NormProp extends beyond ReLU as a general framework for propagating data-level normalization through hidden layers. For each activation, it requires the post-activation mean and standard deviation, computed analytically or empirically as appropriate.

  • Generalization: NormProp can extend to other activation functions by computing the output distribution mean (c2) and standard deviation (c1) after the chosen activation.These statistics enable normalization propagation from the data level to all hidden layers.
  • Parameterized activations: For parameterized activations such as PReLU (He et al. (2015)), the post-activation mean and standard deviation depend on the parameter a and must enter normalization.PReLU statistics are derived by applying PReLU_a to a standard normal input X ∼ N(0, 1).
  • Non-parameterized activations: For fixed activations such as Tanh and Sigmoid, the required statistics can be computed analytically or estimated by simulation from the fixed Normal input distribution.The same activation-specific statistics support extending NormProp to these non-parameterized functions.
  • Implementation: The appropriate Jacobian factor permits a larger learning rate, although NormProp also works without it.This is an optional enhancement rather than a requirement for the framework.

5. Empirical Results and Observations

NormProp matched across its two data-normalization strategies, stabilized hidden-layer distributions and convergence more effectively than BN, remained robust across batch sizes including 1, and trained faster than BN.

  • Normalization strategies: NormProp’s Global and Batch Data Normalization strategies achieved very similar performance for batch sizes 50 and 100, converging within 30 epochs.The comparison used initial learning rates of 0.05 and 0.08 for batch sizes 50 and 100, respectively.
  • Internal Covariate Shift and convergence: NormProp produced more stable lower-layer input distributions than BN, reducing Internal Covariate Shift and supporting more stable convergence, especially early in training.The convergence comparison used batch size 50 and initial learning rate 0.05; the figure reports 95% confidence intervals.
  • Dataset evaluation: The authors evaluated NormProp and BN on CIFAR-10, CIFAR-100, and SVHN using the stated Network in Network protocol and also reported existing state-of-the-art results.For CIFAR-100, the final convolutional layer used 100 output channels instead of 10; CIFAR experiments used batch size 50 and trained for 200 epochs.
  • Training efficiency: NormProp required approximately 84 seconds per CIFAR-10 training epoch versus 96 seconds for BN on the reported hardware.The speed advantage is attributed to avoiding running-average estimates of hidden-layer input means and standard deviations.

6. Conclusion

NormProp is proposed as a parametric approach to address Internal Covariate Shift while overcoming key Batch Normalization drawbacks. It normalizes dataset statistics and conditions the weight matrix so normalization propagates through hidden layers without moving-average batch statistics for validation or testing.

  • NormProp addresses Internal Covariate Shift in deep-network training through a parametric approach designed to overcome drawbacks of Batch Normalization.
  • Table 1 compares NormProp with Batch Normalization and other state-of-the-art methods across various datasets.
  • Normalizing dataset statistics and conditioning the weight matrix propagates normalization to all hidden layers without estimating their input statistics from mini-batches.
  • NormProp avoids maintaining moving-average estimates of hidden-layer batch statistics during validation and testing, making normalization more representative of the entire data distribution.This is especially relevant during the initial training period, when network parameters change drastically.

Appendices · A. Proofs

Appendix A proves the covariance approximation underlying Normalization Propagation and derives the moments of ReLU and PReLU transforms applied to standard Gaussian inputs.

  • A. Proofs: For u = Wx with zero-mean, spherical-covariance x, Proposition 1 establishes that u’s covariance is approximately canonical.The proposition assumes E_x[x] = 0 and E_x[xx^T] = σ^2I.
  • A. Proofs: The proof characterizes the approximation using the covariance Σ, row coherence µ, and a vector α whose diagonalization gives the closest canonical ellipsoid.The optimal α* is obtained by minimizing the approximation objective element-wise.
  • A. Proofs: Because x has spherical covariance, its off-diagonal covariance elements vanish and all diagonal elements equal the variance of an individual unit.This identity enables the subsequent covariance calculation for u.
  • A. Proofs: Substituting the element-wise optimizer α* into the covariance approximation completes the proof’s canonical-covariance derivation.The appendix explicitly performs this substitution into equation 18.
  • A. Proofs: For X ∼ N(0, 1) and Y = max(0, X), the appendix derives the ReLU mean using the half-normal representation Z = |X|.It also computes the variance from E[Y^2] = 0.5E[Z^2].
  • A. Proofs: For Y = PReLU_a(X), the proof models positive and negative halves with half-normal distributions, scaling the negative half by |a|.It derives E[Y] from the two half-normal components and uses E[Y^2] = 0.5E[Z^2](1 + a^2) to obtain the variance.
  • A. Proofs: The PReLU variance is obtained as var(Y) = E[Y^2] − E[Y]^2, completing the stated moment derivation.The appendix notes that the resulting expression yields the claimed result.
Loading 1603.01431v6…