PaperlayerView exact revision on arXiv

Source-linked AI summary

Fixup Initialization: Residual Learning Without Normalization

Hongyi Zhang, Yann N. Dauphin, Tengyu Ma

arXiv:1901.09321v2cs.LGcs.CVstat.ML

TL;DR

Normalization is widely believed to provide several optimization and generalization benefits, but their necessity for residual networks lacks general consensus. The paper analyzes exploding gradients, introduces Fixup by rescaling initialization, and finds normalization-free residual networks can match or exceed normalized systems across image classification and translation. With proper regularization, Fixup achieves state-of-the-art test performance.

  • Problem

    There is no general consensus on why normalization helps train residual neural networks or whether its perceived benefits require normalization.

  • Method

    Fixup rescales standard initialization according to residual-network architecture and adds bias parameters to obtain properly scaled updates without normalization.

  • Results

    Fixup matches normalization techniques in training deep residual networks and achieves state-of-the-art test performance with proper regularization.

  • Takeaways & Limitations

    Residual networks can be trained reliably without normalization, with Fixup matching normalized image-classification baselines and outperforming the baseline on machine translation benchmarks.

  • Takeaways & Limitations

    The paper leaves the training dynamics of Fixup and the regularization benefits of normalization methods for future analysis.

Abstract

from arXiv · show

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

1 INTRODUCTION

The paper asks whether deep residual networks can train without normalization while retaining reliable optimization and generalization. It answers yes, proposing Fixup and reporting comparable or better benchmark performance.

  • Normalization is widely believed to stabilize learning, permit higher learning rates, accelerate convergence, and improve generalization across deep-network tasks.
  • The paper studies whether residual networks can train without normalization at the same learning rate, convergence speed, and generalization level.
  • The paper derives a gradient-norm lower bound explaining why standard initialization makes normalization essential for training deep residual networks at maximal learning rate.
  • Fixup rescales standard initialization according to network architecture, enabling very deep residual networks to train stably at maximal learning rate without normalization.
  • With proper regularization, Fixup matches normalization-trained baselines on CIFAR-10 and ImageNet and outperforms the baseline on IWSLT and WMT translation benchmarks.
  • Figure 1 contrasts a batch-normalized ResNet block, a simple stably trainable block, and Fixup with additional bias parameters.

2 PROBLEM: RESNET WITH STANDARD INITIALIZATIONS LEAD TO EXPLODING GRADIENTS

The paper attributes exploding gradients in plainly initialized residual networks to activation and logit growth with depth. Its positively homogeneous-function analysis identifies affected parameter sets and motivates a new initialization.

  • Standard initialization does not properly account for residual connections, causing exploding gradients in normalization-free residual networks with positively homogeneous activations.
  • For zero-mean residual blocks, output variance approximately doubles across layers, producing exponential variance growth with depth.
  • At initialization, exploding logits can cause gradient explosion because certain activation and weight-tensor gradient norms are lower bounded by the cross-entropy loss.
  • The analysis applies to convolutional and linear weights in ReLU networks with possible skip connections, but without normalization.
  • A positively homogeneous function satisfies f(αx) = αf(x) for α > 0, and a positively homogeneous parameter set scales the network output likewise when those parameters are scaled.
  • Compositions of positively homogeneous functions remain positively homogeneous, covering operations such as bias-free linear layers, convolutions, pooling, addition, concatenation, dropout, and ReLU.
  • Theorems identify gradient-norm lower bounds under the stated assumptions, while examples include early convolution, final fully connected, and downsampling-related sets.

3 FIXUP: UPDATE A RESIDUAL NETWORK Θ(η) PER SGD STEP

Fixup designs residual-network initialization so each branch contributes a Θ(η/L) output update, yielding a depth-independent Θ(η) network update per SGD step. The method combines architecture-aware rescaling with zero initialization, biases, and branch multipliers to train without normalization while matching normalized-network performance.

  • Update-scale design: Fixup rescales standard initialization to ensure the network function receives updates at the proper scale, independent of depth.Its design starts from the goal of making SGD updates to the network function have the right scale rather than merely controlling individual layers.
  • Update-scale design: Residual branches receive correlated SGD updates, so each of L branches should change the output by Θ(η/L) to produce an overall Θ(η) update.The Shortcut is treated separately from the residual branches, and the Shortcut is assumed to use standard initialization.
  • Fixup initialization: Residual-branch weight layers can be rescaled by setting each layer’s scalar factor to L−1 2m−2, with an alternative that initializes the final branch layer to zero.The zero-function option avoids unlearning a potentially bad random residual-branch state.
  • Biases and multipliers: Fixup adds scalar biases before weight and activation layers because preferred input/output means can differ, and this improves training performance.Biases restore useful representation shifts without normalization.
  • Biases and multipliers: A scalar multiplier per residual branch mimics normalization-like weight-norm dynamics and avoids searching for a new learning-rate schedule.Multipliers scale branch outputs and interact with learning-rate decay and L2 weight decay.
  • Fixup initialization: Fixup initializes the classification layer and each branch’s last layer to zero, rescales other branch weight layers, and adds scalar biases and multipliers.Rule 2 is the essential component; Rules 1 and 3 provide further improvements toward normalized-network performance.
  • Fixup initialization: Fixup introduces only O(K) additional parameters beyond convolution and linear weights, compared with normalization methods requiring O(KC) or O(KCWH).Here K is the number of layers, C the maximum channels per layer, and W,H the spatial dimensions of the largest feature map.

4 EXPERIMENTS

The experiments test whether Fixup can replace normalization while preserving stable, fast training across increasing depth, image classification, and machine translation. Fixup matches normalization-based training behavior in deep networks and reaches competitive or state-of-the-art performance with appropriate regularization.

  • Training at increasing depth: Fixup matches BatchNorm test accuracy after the first epoch even at 10,000 layers using the default BatchNorm learning rate.LSUV and 1/2-scaling cannot train at the same learning rate past 100 layers.
  • Image classification: On CIFAR-10 ResNet-110, Fixup yields a 7% relative improvement in test error over standard initialization while training at the same learning rate as BatchNorm.The Xavier-initialized network fails at 0.1x maximal learning rate and may require more epochs to converge.
  • Image classification: With better regularization, Fixup eliminates the CIFAR-10 and SVHN test-error gap and achieves state-of-the-art results without normalization layers.The initial gap is attributed to BatchNorm's regularization effect rather than optimization difficulty.
  • Image classification: On ImageNet, Fixup training is fast and stable but initially overfits relative to BatchNorm; stronger Mixup regularization makes its performance on par with GroupNorm.The evaluation uses ResNet-50 for 100 epochs and ResNet-101 for 200 epochs.
  • Machine translation: Replacing LayerNorm with Fixup in Transformer produces stable, baseline-speed training and matches or supersedes state-of-the-art results on both translation datasets.Dropout provides strong regularization, preventing the overfitting observed in the ImageNet setting.

5 RELATED WORK

Related work studies normalization, signal propagation, optimization, and initialization in deep networks. This paper extends those lines by analyzing gradient growth through positive homogeneity and proposing an initialization recipe for residual networks.

  • Normalization methods: Normalization methods standardize activations or weights using statistics, whereas Fixup does not compute statistics at initialization or during training.Fixup is therefore distinct from normalization methods despite targeting similar training behavior.
  • Theoretical analysis of deep networks: Prior analyses examine variance propagation, dynamical isometry, activation scale, gradient variance, and related initialization properties in deep and residual networks.Dynamical-isometry work enabled training 10,000-layer CNNs from scratch.
  • Theoretical analysis of deep networks: The paper's theoretical analysis shares the conclusion that standard residual-network initialization is problematic but introduces positive-homogeneity-based gradient-norm lower bounds applicable to broader architectures and initializations.The stated examples include ResNet, DenseNet, Xavier, and LSUV.
  • ResNet initialization in practice: Earlier work studies zero-initialized residual networks, loss landscapes near zero, and initialization effects on test performance; this paper gives a concrete initialization recipe.The related methods include recurrence-based scaling and data-dependent LSUV initialization.
  • Understanding batch normalization: The analysis supports the view that standard initialization creates a steep loss surface at initialization, aligning with claims that BatchNorm may smooth optimization rather than merely reduce internal covariate shift.This connects the paper's analysis to prior interpretations of BatchNorm's effectiveness.

6 CONCLUSION

The paper develops Fixup to train deep residual networks reliably without normalization, using update-scale analysis to match normalization during training and reach strong test performance with regularization.

  • 6 CONCLUSION: Fixup targets reliable training of deep residual networks without normalization by controlling the scale of network and residual-branch updates.The method is motivated by the connection between initialization-time exploding gradients and logit blowup.
  • 6 CONCLUSION: Extensive experiments show that Fixup matches normalization techniques for training deep residual networks and achieves state-of-the-art test performance with proper regularization.
  • 6 CONCLUSION: The authors identify analyzing Fixup training dynamics, extending the scheme to other applications, and improving regularizers as open directions.

A PROOFS FOR SECTION 2

The proofs analyze initialization-time gradients through positive homogeneity, cross-entropy scaling, and entropy bounds to establish a lower bound tied to the output logits.

  • Proof strategy: Positive homogeneity lets the proof differentiate the cross-entropy loss with respect to an input scaling factor at initialization.The composed network output scales linearly with the scaled input.
  • Proof strategy: The directional derivative with respect to the scaling factor equals the cross-entropy loss minus the prediction entropy.The displayed relation is ∂ϵ = ℓ(z, y) − H(p).
  • Proof strategy: Symmetric, mean-zero initialization makes the logits have a symmetric density with mean zero, enabling a lower bound based on the expected largest logit.
  • Result: The resulting gradient norm of a positively homogeneous parameter set is of order Ω(E[max_i∈[c] z_i]) at initialization.

B.1 RESIDUAL BRANCHES UPDATE THE NETWORK IN SYNC

The analysis shows that residual-branch updates align rather than cancel, so each of L branches must contribute only Θ(η/L) to keep the network update at Θ(η).

  • B.1 RESIDUAL BRANCHES UPDATE THE NETWORK IN SYNC: Updates to different residual branches are highly correlated, causing their network-function changes to accumulate rather than cancel at initialization.
  • Assumptions and architecture: Under the stated assumptions, the network is modeled as sequential residual blocks with fully connected layers, ReLUs, and additive residual branches.The final block is a fully connected layer, and the network has no bias parameter in the theoretical setup.
  • Update analysis: The gradient update changes the network output according to a first-order expression involving the residual-branch Jacobian terms.The proof replaces ReLU layers by input-dependent diagonal matrices and uses a Taylor expansion for sufficiently small η.
  • Update analysis: Because the summed Jacobian trace norm scales with L, the expected output-update scale is proportional to L and the average branch contribution.
  • Implication: To obtain an overall Θ(η) network update independent of depth, each residual branch must contribute Θ(η/L) on average.
  • Scalar branch guidance: For a scalar branch, Theorem 4 provides initialization guidance that achieves the desired update scale under its stated nonnegative-parameter and bounded-gradient assumptions.

C.1 ABLATION STUDIES OF FIXUP

The ablations compare Fixup with normalization and altered initialization or regularization settings, examining training behavior and generalization on CIFAR-10 and SVHN.

  • Initialization and architecture ablations: The ResNet-110 ablation compares batch normalization, Fixup, altered initialization scales, bias removal, and removal of all residual branches.
  • Regularization ablations: The experiments use CIFAR-10 WideResNet-40-10 and SVHN WideResNet-16-12 with Mixup and Cutout as additional regularization.
  • Training curves: Figure 4 reports minibatch training accuracy during the first 3 epochs for ResNet-110 on CIFAR-10, using minibatches of 128 and 10-step smoothing.

C.3 TRAINING AND TEST CURVES ON IMAGENET

On ImageNet with ResNet-50, Fixup fits the training data strongly but overfits without additional regularization; with Mixup, it closely matches normalization-based performance.

  • Fixup fits the ImageNet training data better than the compared normalized models but overfits without additional regularization.The figure attributes the overfitting to the need for more regularization.
  • With Mixup, Fixup is competitive with networks trained with normalization on ImageNet.The reported final results closely match those obtained with GroupNorm, despite using no normalization.
  • GroupNorm and BatchNorm comparison results are reproduced from prior work for the ImageNet ResNet-50 curves.
Loading 1901.09321v2…