PaperlayerView exact revision on arXiv

Source-linked AI summary

Norm matters: efficient and accurate normalization schemes in deep networks

Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry

arXiv:1803.01814v3stat.MLcs.LG

TL;DR

The paper examines unresolved shortcomings of batch normalization, including unclear interactions with regularization, task-specific limitations, and poor suitability for low-precision computation. It reframes normalization as decoupling weight norms from the training objective, proposes L1/L∞ alternatives and bounded weight normalization, and reports half-precision viability plus improved large-scale weight-normalization results.

  • Problem

    Batch normalization has unresolved interactions with regularization, task-specific limitations for correlated samples, and poor suitability for low-precision computation.

  • Method

    The paper analyzes normalization as decoupling weight norms from the training objective and proposes L1/L∞ batch normalization plus bounded weight normalization.

  • Results

    L1 normalization enables half-precision batch normalization without visible accuracy loss, while bounded weight normalization improves ImageNet results to near batch-normalization performance.

  • Takeaways & Limitations

    L1/L∞ normalization offers a low-precision alternative to standard batch normalization, while bounded weight normalization extends weight normalization to large-scale tasks.

  • Takeaways & Limitations

    Weight normalization can exhibit unstable convergence and lower ImageNet accuracy than the original method on large-scale usage.

Abstract

from arXiv · show

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used $L^2$ batch-norm, using normalization in $L^1$ and $L^\infty$ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.

1 Introduction

The paper frames normalization as a way to address unresolved interactions between weight decay, computational cost, numerical precision, and task-specific limitations in Batch-Normalization. It proposes analyzing weight norms and developing alternative normalization and weight-normalization schemes.

  • Motivation: Batch-Normalization has unresolved issues involving regularization, task assumptions, computational cost, and numerical precision.Weight decay’s interaction with BN is unclear, while BN can be problematic for correlated samples, memory-bound computation, and low-precision arithmetic.
  • Task-specific limitations: BN assumes independence among batch samples, limiting its use in time-series prediction, reinforcement learning, generative modeling, and some classification settings.The passage also notes instability in some GAN training and the need for modifications in recurrent networks.
  • Computational costs: BN can constitute up to 24% of total model computation and generally requires saving pre-normalized activations for back-propagation.The operation is described as difficult to parallelize and typically memory-bound on current hardware.
  • Numerical precision: Existing normalization methods use L2 normalization, which is poorly suited to low-precision training and inference because variance computation can overflow or introduce quantization noise.The paper identifies the need for alternatives to L2-based activation-scale control.
  • Contributions: The paper interprets normalization as decoupling weight norms from the training objective and uses this view to reassess weight decay and propose new normalization methods.Its contributions include studying learning-rate or normalization adjustments, L1 and L∞ BN variants, and bounded weight normalization.
  • Contributions: L1 and L∞ normalization alternatives preserve accuracy while supporting low-precision computation, and bounded weight normalization improves large-scale performance while remaining nearly comparable with BN.The paper also reports computational and memory benefits for bounded weight normalization.

2 Consequences of the scale invariance of Batch-Normalization

After a linear layer, Batch-Normalization makes the output invariant to the channel weight vector’s norm, so the weight direction becomes the relevant feature for subsequent layers. The paper derives how this scale invariance affects gradient scaling and directional updates during training.

  • Scale invariance: BN after a linear layer makes the output invariant to the channel weight vector norm.The same property applies to per-channel weights in convolutional layers.
  • Scale invariance: With rescaling invariance, the weight vector’s direction is the key feature affecting the following layers.The norm is irrelevant to consecutive-layer inputs, although it still affects the gradient scale.
  • Training dynamics: During training, stochastic-gradient updates use the mini-batch loss gradient and learning rate to update the weights.The section analyzes these updates for the normalized weight direction.
  • Training dynamics: For a linear layer followed by BN and a small learning rate, the weight-direction step size depends on the scale-invariant update dynamics rather than directly on the layer output norm.The analysis also extends to scale-preserving nonlinearities such as ReLU and beyond vanilla SGD.

3 Connection between weight-decay, learning rate and normalization

With batch normalization, weight decay stabilizes optimization by controlling weight norms and therefore the effective step size. Experiments show that learning-rate or norm adjustments can mimic weight-decay training accuracy.

  • Weight decay fixes channel weight norms to a small range, producing a more stable effective step size for the weight direction.Without weight decay, norms grow unbounded and the effective step size decreases despite an unchanged learning-rate hyperparameter.
  • Accuracy gained with weight decay can be achieved without it by adjusting the learning rate using norms measured from weight-decay training.The correction is designed to mimic the effective step size.
  • The learning-rate correction requires access to norms from a weight-decay run, so it demonstrates the mechanism rather than providing a practical replacement.
  • Figure 1 compares weight decay on, weight decay off, weight-decay-free learning-rate correction, and learning-rate replacement by norm scheduling in VGG11 on CIFAR-10.
  • In CIFAR-10 experiments, corrected step sizes and norm scheduling produced accuracy similar to weight-decay training throughout learning.Norm scheduling normalized each convolution-channel norm to match the corresponding weight-decay run while keeping the learning rate constant.

4 Alternative Lp metrics for batch norm

The paper replaces L2 batch normalization’s squared-deviation computation with scale-invariant L1, L∞, and Top(k) alternatives to improve efficiency and low-precision robustness. With appropriate scale adjustment, L1 and Top(10) preserve accuracy, while L1 also reduces computation, memory use, and half-precision instability.

  • L1 batch norm: L1 batch normalization uses average absolute deviation instead of L2 batch normalization’s average squared deviation, avoiding square and square-root operations.The paper motivates this change as more suitable for low-precision training because squaring an n-bit number generally requires 2n bits.
  • L1 batch norm: With the normalization constant CL1, L1 batch norm converged at the same rate and to the same accuracy as L2 batch norm; without it, reaching 20% validation error took more than twice as long.The authors identify precise scale adjustment as important, despite CL1 being close to one.
  • L1 batch norm: L1 normalization improved running time and memory consumption by using cheaper absolute-value operations and caching sign values rather than full activations for gradients.The derivative of |x| is sign(x), enabling the stated memory-saving mechanism.
  • L∞ and Top(k): L∞ batch normalization uses maximum absolute deviation but was less robust to noise, so Top(10) averages the ten largest deviations to improve outlier robustness.Top(k) contains L∞ as Top(1) and L1 as Top(n); Top(10) was sufficient to close the gap to L2 performance in Figure 2.
  • Half-precision training: L1 batch normalization remained accurate in half precision, while standard L2 batch normalization suffered overflow, quantization noise, and divergence.The paper reports no visible validation-accuracy loss for L1 and describes the half-precision L2 run as diverging before epoch 20.
  • Accuracy and efficiency: On ImageNet, L1 and Top(10) reached final accuracy within 0.2% of L2 baselines, whereas L∞ accuracy was 3% lower.The comparison covers ResNet-18 and ResNet-50.

5 Improving weight normalization

The paper modifies weight normalization by fixing each layer’s weight norm, separating weight scale from learned values and addressing instability in large-scale tasks. The resulting bounded method approaches batch-normalization accuracy and extends weight normalization beyond its earlier limitations.

  • 5.1 The advantages and disadvantages of weight normalization: Weight normalization separates each output channel’s direction from its scale, but its scalar scale parameter leaves weight norm free to change.The paper identifies this remaining scale freedom as the target for its modification.
  • 5.2 Norm bounded weight-normalization: The proposed bounded weight normalization fixes each layer’s norm with a scalar ρ determined by layer size or initialized from the weights’ initial norm.For scale-insensitive nonlinearities such as ReLU, ρ can be incorporated into the final classifier’s weights and biases.
  • 5.2 Norm bounded weight-normalization: The modification resolves the unstable convergence previously reported for weight normalization on large-scale tasks and achieves accuracy comparable to BN, though still behind it.The authors note that hyper-parameter tuning, including learning-rate scheduling, might further reduce the remaining gap.
  • 5.3 Lp weight normalization: On ImageNet with ResNet50, BN reaches 75.3% final accuracy, BWN reaches 73.8%, and WN reaches 67%.The WN result is taken from prior work, while the paper reports that its own WN implementation could not converge.
  • 5.3 Lp weight normalization: L1 weight normalization can replace the original L2 form, whereas the proper top-1 absolute-maximum L∞ version causes noticeable degradation.The authors attribute the L1 alternative’s viability to suitable computation of the scaling constant ρ.

6 Discussion

The discussion reframes normalization as decoupling weight norms from the training objective and uses that perspective to motivate alternatives to standard batch normalization. The proposed L1, L∞, and bounded weight schemes target precision, resource, and large-scale-task limitations.

  • 6 Discussion: The paper views normalization methods as tools that decouple weights’ norms from the training objective.This perspective also motivates re-evaluating weight decay and its relationship with learning-rate adjustments.
  • 6 Discussion: L1- and L∞-based normalization can provide results similar to standard BN while supporting low-precision computation.The paper presents these schemes as alternatives targeting computational and numerical deficiencies.
  • 6 Discussion: L1 normalization enabled batch normalization in half-precision floating-point, whereas prior work had reported failure and required full or mixed precision hardware.The scale constant CL1 must be chosen precisely to match L2 performance.
  • 6 Discussion: Bounded weight normalization improves results on large-scale ImageNet tasks and is nearly comparable with BN.The paper also associates it with lower computational costs and potential use in reinforcement learning and temporal modeling.
  • 6 Discussion: The paper proposes that effective norm or temperature, batch size, weight decay, and learning rate mutually affect one another.This is presented as a suggested connection for how networks are devised and trained.

A Implementation Details for weight-decay experiments

The weight-decay experiments use VGG11 with batch normalization after every convolution, a fixed last-layer decay factor, and a scheduled learning rate.

  • A Implementation Details for weight-decay experiments: The experiments use VGG11 with batch normalization after every convolution layer and weight decay only on the last layer.The last-layer decay factor is λ = 0.0005.
  • A Implementation Details for weight-decay experiments: The learning rate starts at 0.1 and is divided by 10 every 20 epochs, except in the norm-scheduling experiment.All runs use the same random seed.

B Importance of normalization constants

The appendix shows that the CL1 normalization constant materially affects optimization and validation performance, with π/2 identified as the optimal tested value.

  • B Importance of normalization constants: Removing CL1 slows convergence and produces a higher final validation error when training ResNet-56 on CIFAR-10.The comparison is shown in the left side of Figure 5.
  • B Importance of normalization constants: Figure 5 further evaluates CL1 with Res18 on ImageNet.The supplied passage identifies the ImageNet comparison but does not include its complete reported outcome.
  • B Importance of normalization constants: π/2 ≈ 1.25 is optimal, while modifying CL1 to nearby values degrades performance.The result is reported for the normalization constant studied in Figure 5.

B.1 Deriving normalization constants

The section derives normalization constants by assuming pre-normalization activations follow a Gaussian distribution and relating maximum absolute deviation to the standard deviation.

  • The derivation assumes the inputs to the normalization layer follow a Gaussian distribution N(µ_k, σ^2).
  • Multiplying the deviation bound by C_L∞(n) yields an approximation of the original standard deviation measure σ.
  • The resulting scale relation is 8π ln(2) · σ ≈ 0.793σ, with u defined as 1+√π ln(4).

C Bounded-weight-norm experiments

The experiments evaluate bounded-weight norm against unnormalized and normalized baselines, including recurrent-network training and L1-norm results.

  • Figure 6 compares bounded weight-norm with a no-normalization baseline during recurrent-network training on WMT14 de-en.
  • Table 1 compares the baseline, L2-based normalization with weight-normalization, and bounded-weight norm.
  • Table 2 reports results comparing the baseline with L1-norm normalization, using perplexity as the metric.
Loading 1803.01814v3…