PaperlayerView exact revision on arXiv

Source-linked AI summary

Iterative Normalization: Beyond Standardization towards Efficient Whitening

Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao

arXiv:1904.03441v1cs.CVcs.LG

TL;DR

BN standardization and DBN whitening face limitations involving decorrelation, batch size, and GPU efficiency. The paper proposes IterNorm, which uses Newton’s iterations and SND-based analysis; it reports consistent improvements over BN and DBN on CIFAR-10 and ImageNet.

  • Problem

    DBN improves normalization through whitening but depends heavily on large batches or GPU-inefficient eigen-decomposition, motivating a more efficient whitening approach.

  • Method

    IterNorm approximates the whitening matrix with Newton’s iterations, normalizes covariance eigenvalues for convergence, and introduces SND to measure normalization stochasticity.

  • Results

    IterNorm consistently improves performance over BN and DBN on CIFAR-10 and ImageNet, including a 3.56% test error on WRN-28-10 on CIFAR-10.

  • Takeaways & Limitations

    IterNorm offers a better trade-off between optimization and generalization while improving whitening efficiency by avoiding eigen-decomposition.

  • Takeaways & Limitations

    The Newton iteration requires the convergence condition ∥A − I∥2 < 1, and the method uses extra learnable γ and β parameters because normalization constrains representation capacity.

Abstract

from arXiv · show

Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-batches. Decorrelated Batch Normalization (DBN) further boosts the above effectiveness by whitening. However, DBN relies heavily on either a large batch size, or eigen-decomposition that suffers from poor efficiency on GPUs. We propose Iterative Normalization (IterNorm), which employs Newton's iterations for much more efficient whitening, while simultaneously avoiding the eigen-decomposition. Furthermore, we develop a comprehensive study to show IterNorm has better trade-off between optimization and generalization, with theoretical and experimental support. To this end, we exclusively introduce Stochastic Normalization Disturbance (SND), which measures the inherent stochastic uncertainty of samples when applied to normalization operations. With the support of SND, we provide natural explanations to several phenomena from the perspective of optimization, e.g., why group-wise whitening of DBN generally outperforms full-whitening and why the accuracy of BN degenerates with reduced batch sizes. We demonstrate the consistently improved performance of IterNorm with extensive experiments on CIFAR-10 and ImageNet over BN and DBN.

1. Introduction

BN standardizes activations but does not decorrelate them, while DBN whitens them at the cost of large batches and inefficient decompositions. IterNorm uses Newton’s iterations for efficient whitening, reduces batch-size sensitivity, and is consistently improved over BN and DBN in CIFAR-10 and ImageNet experiments.

  • BN centers and scales mini-batch activations so each neuron has zero mean and unit variance, but it does not decorrelate them.
  • DBN whitens activations and improves training efficiency and generalization over BN, but relies on large batches and GPU-inefficient eigen-decomposition or SVD.
  • IterNorm approximates the whitening matrix with Newton’s iterations, avoiding eigen-decomposition or SVD while progressively adjusting dimensions along eigenvectors.
  • Small or zero covariance eigenvalues can be largely ignored with a fixed number of IterNorm iterations, significantly reducing sensitivity to batch size.
  • SND measures stochastic uncertainty from mini-batch normalization and supports analysis showing IterNorm’s trade-off between optimization and generalization.
  • IterNorm consistently improves performance over BN and DBN on CIFAR-10 and ImageNet.

2. Related Work

Prior work studies activation normalization, standardization, whitening, weight normalization, and Newton-based covariance operations. IterNorm approximates DBN’s ZCA-whitening matrix more efficiently and provides a generic data-dependent normalization module.

  • Activation normalization methods estimate or transform statistics to normalize neural-network activations, but population-statistics approaches can become unstable with inaccurate local estimates.
  • Mini-batch standardization methods include L2, L1, and L∞ variants, while many approaches address their dependence on reasonable batch sizes for estimating mean and variance.
  • DBN applies ZCA-whitening through eigen-decomposition, whereas IterNorm targets a more efficient approximation of DBN’s whitening matrix.
  • IterNorm is data-dependent, unlike related approaches that normalize weights or implicitly normalize activations independently of the data.
  • Unlike Newton-based bilinear or second-order pooling methods limited to fully connected activations, IterNorm is designed as a generic module for various neural-network frameworks.

3. Iterative Normalization

IterNorm whitens mini-batch activations by approximating the covariance inverse square root with Newton’s iterations, avoiding eigen-decomposition or SVD. The method supports back-propagation, running statistics for inference, and implementation choices that control efficiency and representation capacity.

  • Whitening with Newton’s iteration: IterNorm computes the covariance of centered mini-batch activations and applies Newton’s iterations to approximate its inverse square root for ZCA whitening.The algorithm centers inputs, forms the covariance, normalizes it, iterates, and multiplies the resulting whitening matrix by the centered inputs.
  • Whitening with Newton’s iteration: Trace-normalizing the covariance ensures a convergence condition for Newton’s iteration before the whitening matrix is computed.The normalized covariance has nonnegative eigenvalues whose sum is 1, supporting the required convergence condition.
  • Whitening with Newton’s iteration: The iteration number controls whitening extent because dimensions with larger covariance eigenvalues converge faster than those with smaller eigenvalues.This mechanism allows the method to progressively stretch dimensions toward unit eigenvalues while controlling the amount of whitening.
  • Back-propagation: IterNorm back-propagates through its covariance normalization and Newton iterations, including the minibatch-dependent trace normalization.The backward pass uses saved centered activations and iteration states to compute gradients with respect to the inputs.
  • Training and inference: During inference, running averages provide the population mean and whitening matrix, while learnable scale and shift parameters help recover representation capacity.The running statistics are updated during training with momentum and used for inference.
  • Efficiency and implementation: IterNorm has computation comparable to convolution, can use group-wise whitening when dimension is large, and introduces no extra inference cost after merging with adjacent convolutions.Its iteration computation costs Td^3, while the covariance and output operations are equivalent to 1 × 1 convolutions.

4. Stochasticity of Normalization

Mini-batch normalization introduces stochastic disturbance that depends on batch size and feature dimension, affecting optimization as well as inference. SND explains why full whitening can trade improved conditioning for instability, whereas IterNorm adapts whitening to achieve a better balance.

  • 4. Stochasticity of Normalization: Undersized batches can degrade both inference and training because mini-batch normalization introduces stochasticity, measured here by Stochastic Normalization Disturbance (SND).The paper reports that BN can hardly learn with batch size 2, while an unnormalized network learns well.
  • 4.1. Stochastic Normalization Disturbance: SND measures the variation of a sample’s normalized output across random sample sets XB, and normalization-level SND averages this disturbance over examples.For a fixed sample, changing the batch changes the normalized output because XB is random.
  • 4.1. Stochastic Normalization Disturbance: Larger BN batches reduce SND and make transformed outputs more compact, while small batches and high dimensions increase BN’s disturbance.With batch size fixed at 2, BN’s SND decreases in low dimensions but increases in high-dimensional spaces.
  • 4.2. Controlling the Extent of Whitening: With a full batch of 60,000 on MNIST, all normalization methods improve performance, but IterNorm trains more efficiently than full whitening.The paper attributes full-whitening’s disadvantage to amplifying directions associated with small eigenvalues, which may correspond to noise.
  • 4.2. Controlling the Extent of Whitening: DBN achieves condition number 1 but enlarges SND, especially in high dimensions, so its conditioning benefit does not consistently improve training efficiency.This trade-off also explains why group-wise whitening can outperform full whitening from the training perspective.
  • 4.2. Controlling the Extent of Whitening: IterNorm improves conditioning over BN while reducing or stabilizing SND in high-dimensional spaces through eigenvalue-adaptive normalization.Its Newton iterations stretch dimensions associated with large eigenvalues faster and largely ignore small or zero eigenvalues for a fixed iteration count.

5. Experiments

Experiments compare IterNorm with BN and DBN across CIFAR-10, ImageNet, architectures, batch sizes, iteration counts, and group sizes. IterNorm generally improves optimization and generalization, with especially strong results on wide residual networks and selected ImageNet placements.

  • Effect of Iteration Number: T = 0 and T = 7 have the worst training efficiency, while T = 7 also has significantly worse test performance.The ablation varies IterNorm’s iteration number over {0, 1, 3, 5, 7}.
  • Comparison of Baselines: IterNorm converges fastest and generalizes best among IterNorm, BN, and DBN across four batch-size and learning-rate experiments.It reduces BN’s absolute test error by 0.79%, 0.53%, 1.11%, and 0.75%, and DBN’s by 0.22, 0.37, 1.05, and 0.58.
  • Results on CIFAR-10 with Wide Residual Networks: 3.56% test error on WRN-28-10 represents an 8.5% relative improvement over the Baseline on CIFAR-10.IterNorm improves the original WRN with BN and DBN.
  • Results on ImageNet with Residual Network: Replacing only the first BN improves residual-network performance with either DBN or IterNorm on ImageNet.Replacing all layers with IterNorm provides no significant improvement over replacing only the first layer.
  • Results on ImageNet with Residual Network: IterNorm-L1 plus decorrelated features significantly improves residual-network performance while introducing negligible computational cost.The method adds IterNorm after the last average-pooling layer and before the final linear layer.
  • Results on ImageNet with Residual Network: IterNorm-L1 achieves lower test errors than the original ResNet-50 and ResNet-101 models, while IterNorm-L1 + DF improves performance further.These results use single-model, single-crop testing.

6. Conclusions

The paper concludes that IterNorm improves optimization, generalization, and whitening efficiency over BN and DBN, while SND explains normalization behavior across batch sizes and feature dimensions.

  • 6. Conclusions: IterNorm uses Newton’s iterations to decorrelate activations, improving optimization and generalization over BN while avoiding DBN’s expensive eigen-decomposition.The paper reports consistent improvements on CIFAR-10 and ImageNet.
  • 6. Conclusions: SND measures stochastic uncertainty in normalization and supports analysis of batch-size and feature-dimension effects.The paper presents IterNorm as having a better trade-off between optimization and generalization.

A. Derivation of Back-propagation

The appendix derives IterNorm’s backward pass by applying the chain rule through its normalization operations, including covariance and iterative whitening terms.

  • A. Derivation of Back-propagation: IterNorm’s forward pass takes mini-batch inputs X and computes whitened outputs using the iterative normalization procedure.The appendix uses column-vector matrix notation for the forward computation.
  • A. Derivation of Back-propagation: The backward derivation computes gradients through IterNorm’s normalized covariance, inverse covariance, trace normalization, and whitening transformations.The displayed derivation expresses gradients with respect to covariance-related quantities.
  • A. Derivation of Back-propagation: Gradients with respect to the iterative matrices P_k are computed recursively in reverse order and then used in the chain-rule backpropagation.The recursion runs from k = T to 1, while the iterative derivatives are calculated for k = 1 to T.

B. Comparison of Wall Clock Time

Wall-clock experiments compare IterNorm with DBN and convolution implementations, finding that IterNorm’s cost is comparable to convolution and substantially lower than DBN in VGG training.

  • Computational Cost: IterNorm costs 2hwmd^2 + Td^3, compared with 9hwmd^2 for a same-sized 3 × 3 convolution.Its relative cost is 2/9 + Td/mhw.
  • Wall-clock comparison: IterNorm-iter5 is faster than Torch’s ‘nn’ convolution and slightly slower than ‘cudnn’ convolution in the reported TITAN Xp measurements.The comparison uses input configurations with m = 64, h = w = 32, and d = 64 or 128.
  • Wall-clock comparison: 0.343s per iteration for IterNorm versus 1.366s for DBN with group size 16 on VGG at batch size 256.The reported training-time comparison uses additional VGG experiments.

C. Experiments of IterNorm with Different Iterations

The experiments compare IterNorm across iteration counts with BN and DBN, and analyze how iteration count affects performance, conditioning, and stochasticity. Increasing iterations improves conditioning but also increases SND, creating a trade-off in whitening strength.

  • Experimental setup: IterNorm is evaluated with different iteration numbers alongside Batch Normalization (BN) and Decorrelated Batch Normalization (DBN).The comparison uses the experiments described in Section 4.2 of the main paper.
  • Performance across iterations: Iteration counts T = 1 and T = 9 produce the worst training-efficiency performance on MNIST, while T = 9 also has significantly worse test performance.The tested values are T ∈ {1, 3, 5, 7, 9}.
  • Conditioning and stochasticity: Increasing iteration count T improves conditioning and increases Stochastic Normalization Disturbance (SND).The results indicate that T controls the extent of whitening.
  • Conditioning and stochasticity: The iteration number can balance improved conditioning against the stochasticity introduced by whitening.This trade-off is reported from the conditioning and SND analysis.

D. Details of the VGG Network

The experiment uses VGG networks tailored for 32 × 32 inputs, with their architectural details provided in Table A2.

  • Network configuration: The experiments use VGG networks tailored for 32 × 32 inputs.These networks are introduced for the experiments described in Section 5.1 of the main paper.
  • Network configuration: Table A2 provides the details of the VGG network used in the experiment.The table corresponds to the VGG setup in Section 5.1 of the main paper.
  • Network configuration: The VGG architecture is documented as part of the experimental setup.The supplied passage identifies the network details but does not enumerate its layers.
Loading 1904.03441v1…