PaperlayerView exact revision on arXiv

Source-linked AI summary

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, Ling Shao

arXiv:2009.12836v1cs.LGcs.CVstat.ML

TL;DR

Training deep neural networks is difficult despite their strong representation capacity, motivating normalization techniques. This paper unifies their optimization motivations, organizes their methods, and reviews applications, while identifying limitations such as instability from population-statistics estimation.

  • Problem

    Deep neural networks are difficult to train because their depth and complex structure complicate optimization despite their powerful representation capacity.

  • Method

    The paper reviews normalization methods through an optimization-based taxonomy, decomposes activation normalization into partitioning, operation, and representation recovery, and surveys task-specific applications.

  • Results

    The paper provides a unified picture of normalization motivations, a comprehensive method taxonomy, and a review showing applications can address key issues in domain adaptation, style transfer, GAN training, and efficient deep models.

  • Takeaways & Limitations

    The resulting landscape offers guidelines for selecting normalization techniques and designing methods tailored to specific tasks or efficiency-performance trade-offs.

  • Takeaways & Limitations

    Normalization by population statistics can be unstable because estimates are limited, change with weight updates, and become increasingly inaccurate across layers.

Abstract

from arXiv · show

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

1 INTRODUCTION

The paper surveys normalization techniques for training DNNs, addressing the lack of a unifying lens for comparing their motivations, analyses, and applications. It organizes methods through optimization, a taxonomy, a three-component framework, theoretical analysis, and task-specific applications.

  • DNN training is difficult despite the representation capacity provided by deep, complex architectures.
  • The survey addresses the absence of a unifying lens for describing, comparing, and analyzing normalization techniques.
  • Most normalization methods seek nearly equal layer input and output-gradient statistics to avoid ill-conditioned optimization landscapes.
  • The paper decomposes normalizing-activations-as-functions into normalization area partitioning, normalization operation, and normalization representation recovery.
  • It reviews normalization applications in domain adaptation, style transfer, GAN training, and efficient deep models, including editing activation statistics.

2 DENOTATIONS AND DEFINITIONS

This section defines the notation, learning setup, and normalization operations used throughout the paper. Normalization is treated as a transformation that imposes specified statistical properties on data.

  • The paper represents scalars, vectors, matrices, and higher-order tensors with distinct notation, while DNNs are modeled as stacked transformations.
  • Training minimizes empirical risk by tuning model parameters to reduce the discrepancy between predicted and desired outputs.
  • The paper focuses on empirical risk from optimization and does not explicitly analyze risk under the true data distribution from a generalization perspective.
  • Normalization: Normalization is defined as a function mapping data to transformed data with certain statistical properties.
  • Normalization operations: The main operations are centering, scaling, decorrelating, standardizing, and whitening, which impose zero-mean, unit-variance, decorrelation, or spherical-covariance properties.

3 MOTIVATION AND OVERVIEW OF NORMALIZATION IN DNNS

The paper motivates normalization as a way to improve optimization conditioning in DNNs, while distinguishing the challenges of evolving layer activations and stochastic gradient estimation. It groups methods by whether they normalize activations, weights, or gradients.

  • Normalizing inputs can remove feature-magnitude differences and offers theoretical advantages for linear models.
  • For quadratic optimization, the Hessian spectrum governs learning dynamics, while whitening can make the curvature matrix the identity and permit one-step GD convergence.
  • These linear-model results do not directly extend to deeper neural-network weight matrices, motivating layer-wise curvature approximations such as K-FAC.
  • Normalization methods aim to equalize layer input and output-gradient statistics, corresponding to improved conditioning under the paper’s criteria.
  • The review covers activation normalization using population statistics or functions, weight normalization, and gradient normalization.

4 NORMALIZING ACTIVATIONS BY POPULATION STATISTICS

Population-statistics methods normalize activations using distribution estimates treated as constant during backpropagation. Their effectiveness is constrained by inaccurate estimates, changing activation distributions, and error amplification in deep networks.

  • Population-statistics normalization estimates activation statistics over their distribution and treats those estimates as constant during backpropagation.
  • Centering and whitening were proposed to improve optimization conditioning, using estimated population means or covariance-related quantities.
  • Running-average estimates can be inaccurate; one reported observation is that scaling did not improve performance in a centering-based approach.
  • Whitening methods use interval updates, a preconditioning hyperparameter, or online SVD to address estimation and computational difficulties.
  • Population-statistics normalization can become unstable because estimates are limited, activation distributions shift during training, and estimation errors amplify with depth.

5 NORMALIZING ACTIVATIONS AS FUNCTIONS

This section frames normalization as a function applied to mini-batch activations and explains how standardization, affine recovery, and reshaping form the pipeline. It also identifies BN’s training–inference inconsistency and small-batch limitations.

  • BN as a normalization function: BN standardizes each neuron across mini-batch data using mean and variance computed from the current mini-batch.During inference, running population statistics are used instead.
  • Limitations: BN’s training and inference operations differ because training uses mini-batch statistics while inference uses population statistics estimated during training.The discrepancy becomes more significant when batch size is small because mean and variance estimates become less accurate.
  • Representation recovery: Standardization constrains activations, so BN adds learnable scale and shift parameters to recover representation capacity.These affine parameters are denoted γ and β.
  • Function pipeline: Normalization functions decompose into partitioning the activation area, applying a normalization operation, recovering representation, and reshaping the output.Algorithm 1 lists these stages as Π, Φ, Ψ, and Π^-1.
  • Limitations: Feature normalization methods were developed to address BN’s inconsistent operations and rapidly increasing estimation error at small batch sizes.The review organizes subsequent approaches around these limitations.

5.1 A Framework for Decomposing Normalization

The framework decomposes activation normalization into area partitioning, normalization operation, and representation recovery, then uses these components to compare normalization methods and their design choices.

  • Framework: Normalization area partitioning determines which activation elements are grouped as samples for computing normalization statistics.For convolutional activations X ∈ R^d×m×h×w, partitioning maps them into groups indexed by the selected sample sets.
  • Representation recovery: Representation recovery commonly uses a channel-wise affine transformation with learnable scale γ and shift β.The framework expresses this as cX ⊙ (γ1^T) + (β1^T).
  • Partitioning variants: LN normalizes across channels, height, and width independently for each sample, while GN divides neurons into groups and normalizes within each group.GN reduces to LN when g = 1 and supports small-batch visual tasks by changing the group number.
  • Partitioning variants: IN removes instance-specific contrast information by normalizing each image, whereas PN normalizes each spatial position across channels.IN is widely used for style transfer, while PN is designed for spatial information and generative models.
  • Normalization operation: The normalization operation transforms partitioned activations, with standardization as the default and whitening as a more general alternative.The review also covers variations and reduced forms of standardization.
  • Whitening: Whitening methods face covariance-estimation and inverse-square-root backpropagation challenges, with ZCA and Cholesky whitening offering distinct practical outcomes.ZCA avoids stochastic axis swapping and improves classification over standardization, while Cholesky whitening achieves state-of-the-art GAN training performance.
  • Standardization variants: L1 normalization avoids the costly square and root operations of L2 normalization and achieved a 1.5× speedup with 50% lower power consumption on an FPGA.The cited result is reported for the proposed sign and absolute operations in L1 normalization.
  • Representation recovery: Conditional normalization can inject side information into representation recovery for conditional generative models, including conditional BN and conditional IN.The side information is introduced through the NRR operation.

5.2 Multi-Mode and Combinational Normalization

Multi-mode methods normalize heterogeneous distributions using multiple estimated modes, while combinational methods learn or design mixtures of normalization strategies to adapt their behavior.

  • Multiple modes: MixNorm identifies distributional subregions by disentangling modes with a Gaussian mixture model before normalizing samples using the estimated parameters.Its two-stage process fits the GMM with expectation-maximization and K-means++ initialization, then performs normalization.
  • Multiple modes: ModeNorm addresses heterogeneous datasets with multiple means and variances, assigning samples probabilistically through jointly trained mixture-of-experts gates.Each sample is normalized according to its gate assignment.
  • Combination: Switchable normalization combines channel-wise, layer-wise, and mini-batch-wise statistics from IN, LN, and BN using learned softmax importance weights.The method switches among normalization strategies by learning their relative importance.
  • Combination: Exemplar normalization learns data-dependent normalizations for individual images, whereas switchable normalization fixes importance ratios for the entire dataset.Dynamic normalization further unifies IN, LN, GN, and BN and can interpolate among them.
  • Combination: BIN adaptively balances IN and BN through a learnable gate controlling how much style information passes through each feature-map channel.AdaLIN similarly balances LN and IN for image-to-image translation tasks.
  • Combination: EvoNorms were discovered by searching over normalization-activation layer combinations with AutoML, producing structures that sometimes go beyond existing methods.The passage characterizes EvoNorms as a set of new normalization-activation layers.

5.3 BN for More Robust Estimation

This section reviews methods that make BN estimation more robust by reducing training–inference discrepancy, improving statistics across batches, or correcting estimates during inference.

  • Training–inference discrepancy: BN uses different normalization operations during training and inference, and the discrepancy becomes significant when small batches make mean and variance estimates inaccurate.This mismatch can limit BN in complex networks and particular tasks.
  • Training-time correction: Batch renormalization combines estimated population statistics during training by augmenting normalized outputs with a bounded affine correction.When correction factors remain within bounds, training uses population statistics consistently with inference; otherwise mini-batch statistics contribute.
  • Cross-batch estimation: Combining population and mini-batch statistics can improve BN performance in small-batch-size scenarios.Related approaches include diminishing BN, full normalization, online normalization, and moving-statistics methods.
  • Cross-batch estimation: Memorized BN and cross-iteration BN use information from recent batches or iterations to produce more accurate and stable statistics.Cross-iteration BN approximates statistics for current weights with a low-order Taylor polynomial.
  • Inference correction: Inference-only methods correct normalization statistics without changing the model’s training scheme.EvalNorm optimizes sample weights during inference so normalized activations resemble those produced during training.
  • Inference correction: Post-training population-statistics estimation can help small-batch models but may degrade generalization when batch size is moderate.The original BN paper recommended estimating population statistics after training rather than relying only on running averages.
  • Whitening estimation: Estimating a whitening matrix is more challenging than estimating BN statistics, and indirectly using the mini-batch covariance matrix is more stable.The whitening matrix can then be calculated after training.

6 NORMALIZING WEIGHTS

Weight normalization constrains weights to preserve activation statistics or control weight geometry, offering scale-invariant optimization and theoretical tractability. These constraints require suitable optimization strategies and may reduce representation capacity or practical accuracy without complementary normalization.

  • Weight constraints: Weight normalization constrains each neuron's input weight to unit norm, while centered weight normalization additionally enforces zero mean.CWN is intended to preserve activation statistics between layers under certain assumptions.
  • Weight constraints: Orthogonal weights can preserve activation and output-gradient norms between linear transformations, while spectral normalization constrains the maximum singular value to control a discriminator’s Lipschitz constant.ONI interpolates between spectral normalization and full orthogonalization by changing its iteration number.
  • Weight constraints: Weight constraints can harm representation capacity, so an extra learnable scalar is commonly used to recover the lost flexibility.The scalar plays a role similar to BN’s affine parameters.
  • Training with constraints: Re-parameterization, penalty regularization, and Riemannian optimization are three strategies for training networks while maintaining constrained weights.Riemannian optimization treats constrained matrices as embedded submanifolds, such as the Stiefel manifold for orthogonality.
  • Practical scope: Weight normalization is data-independent and easier to analyze theoretically, but may underperform BN in large-scale image classification and can be less effective with residual connections or unsuitable nonlinearities.Combining weight and activation normalization is widely studied; WS may cause exploding activations and often needs BN or GN.

7 NORMALIZING GRADIENTS

Gradient normalization addresses exploding or vanishing gradients directly rather than designing a better-conditioned optimization landscape. Layer-wise scaling controls gradient magnitudes, while gradient centralization centers weight gradients and improves DNNs with activation normalization.

  • Motivation: Gradient normalization targets ill-conditioned optimization by operating solely on gradients and exploiting curvature information for GD or SGD.This differs from activation and weight normalization, which aim to improve the optimization landscape by design.
  • Gradient scaling: Block-wise gradient normalization scales each layer’s weight gradients to unit norm, reducing large gradients and increasing small ones.The method was proposed to address gradient explosion or vanishing.
  • Gradient centralization: Gradient centralization centers the gradient with respect to each neuron’s input weight in every layer.It implicitly constrains the input weights so their elements retain a constant sum during training.
  • Gradient centralization: Gradient centralization effectively improves the performance of DNNs using activation normalization such as BN or GN.Its benefit is reported specifically for networks with activation normalization.

8 ANALYSIS OF NORMALIZATION

The paper analyzes normalization through scale invariance, optimization conditioning, and stochasticity. These analyses explain training stabilization, convergence behavior, and generalization effects, while applications such as ghost batch normalization exploit normalization noise for large-batch training.

  • Scale invariance: BN’s scale invariance leaves predictions unchanged under parameter rescaling and adaptively adjusts the layer-wise learning rate, stabilizing training.The property also applies to several activation- and weight-normalization methods.
  • Scale invariance: Scale-invariant weights are perpendicular to their gradients, producing an auto-tuning effect; theory establishes convergence results for BN under broad learning-rate conditions.For OLS, BN-based GD converges under arbitrary weight learning rates and remains linear under mild conditions.
  • Scale invariance: With scale-invariant normalization, weight decay reduces parameter norms and thereby increases the effective learning rate.The original learning-rate schedule and weight decay can be folded into a new exponential schedule.
  • Optimization conditioning: BN can improve conditioning of activation covariance and output-gradient covariation, and placing it after the linear layer may work better in certain situations.This practical placement differs from the placement required by some theoretical whitening arguments.
  • Stochasticity and generalization: BN’s stochasticity arises because normalized outputs and population statistics depend on minibatch inputs, and this noise is linked to improved generalization.Ghost batch normalization uses small virtual batches to reduce generalization error during large-batch training.

9 APPLICATIONS OF NORMALIZATION

Normalization methods are reviewed as task-specific tools for stabilizing DNN training and editing activation statistics across domain adaptation, style transfer, image translation, and GAN training.

  • Overview: Normalization methods can be integrated as general modules to stabilize and accelerate training across computer-vision and NLP architectures.The review focuses on domain adaptation, style transfer, training GANs, and efficient deep models.
  • Domain Adaptation: AdaBN adapts batch-normalization statistics between source and target domains to learn domain-invariant features without extra loss terms or parameters.Follow-up methods couple source and target samples or align their feature distributions to a canonical one.
  • Style Transfer: Normalization can edit activation statistics so they represent style or domain-specific information, supporting controllable transformations.The normalization operation can remove style information, while representation recovery introduces it.
  • Style Transfer: Conditional and adaptive instance normalization encode or match style statistics, while whitening and coloring transformations match covariance matrices for richer style control.CIN stores styles in affine parameters, whereas WCT addresses AdaIN’s variance-only matching limitation.
  • Image Translation: Image-translation methods extend normalization through local, regional, adaptive-convolutional, and higher-order-statistics transformations for tasks including inpainting and unsupervised translation.Examples include GDWCT, AdaCoN, region normalization, and attentive normalization.
  • Training GANs: GAN training uses weight and spectral normalization to constrain discriminators, while generator normalization or orthogonal regularization can improve training stability.Spectral normalization enforces Lipschitz continuity on discriminator parameters; generator spectral normalization can reduce discriminator updates per iteration.

10 SUMMARY AND DISCUSSION

The paper presents normalization as a landscape connecting methods, theory, and applications, while identifying theoretical and task-specific gaps that remain open.

  • Summary: The review organizes normalization techniques, analyses, and applications into a research landscape intended to guide method selection and future design.It links choices of normalization area partitioning and operation to task tailoring and efficiency–performance trade-offs.
  • Theoretical Perspective: Theoretical analyses of DNNs remain limited because commonly studied networks often differ from practical architectures that use normalization.The paper identifies this mismatch as an open theoretical perspective.
  • Theoretical Perspective: BN and LN can invalidate theoretical assumptions by introducing nonlinearity and batch-dependent optimization behavior.BN analyses may depend on sampling strategy and mini-batch size, requiring revised optimization frameworks.
  • Theoretical Perspective: Weight normalization preserves or strengthens some theoretical analyses by controlling layer Lipschitz constants, but remains less effective than activation normalization for training performance.Approximate orthogonality is connected to certified adversarial defense and generalization analysis.
  • Applications Perspective: Normalization-based editing of activation statistics is widely exploited in computer vision, but its relationship to domain knowledge in NLP remains underexplored.The paper also highlights the unresolved contrast between BN/GN effectiveness in vision and LN effectiveness in NLP.
  • Applications Perspective: Normalization is uncommon in deep reinforcement learning, motivating investigation of GAN-inspired normalization techniques for DRL training.The proposed connection is based on similarities between certain DRL frameworks and GANs.
Loading 2009.12836v1…