Characterizing signal propagation to close the performance gap in unnormalized ResNets

Motivation

• BatchNorm has practical drawbacks including batch-size dependence, train–inference mismatch, memory overhead, implementation errors, and reproducibility difficulties.Existing alternatives can generalize poorly or add inference compute.Source: §1 · p. 1

Contributions

• The paper introduces Signal Propagation Plots to inspect forward signal propagation and design unnormalized ResNets with batch-normalized-like propagation properties.These visualizations support the proposed design process rather than adding activation-normalization layers.Source: §1 · pp. 1–2
• Scaled Weight Standardization prevents depth-wise growth of per-channel means caused by ReLU or Swish activations, substantially improving unnormalized ResNet performance.The failure arises because these nonlinearities have positive mean outputs, causing squared hidden-activation means to grow rapidly with depth.Source: §1 · p. 2
• On networks as deep as 288 layers, normalization-free ResNets attain performance comparable to or better than batch-normalized ResNets on ImageNet.The result combines the proposed normalization-free structure with Scaled Weight Standardization.Source: §1 · p. 2
• Across FLOP budgets, normalization-free RegNet models are competitive with the state-of-the-art EfficientNets on ImageNet.The models combine the normalization-free approach with RegNet and compound scaling.Source: §1 · p. 2

Deep ResNets at initialization

• BatchNorm and skip connections enabled training ResNets with hundreds or thousands of layers by shaping signal propagation and gradients.Analyses of normalized ResNets study this behavior at initialization.Source: §2 · p. 2
• Zero-initialized residual branches preserve trainability in very deep unnormalized ResNets, but can degrade generalization relative to well-tuned baselines.This strategy biases early computation toward the skip path.Source: §2 · p. 2
• Theory suggests that identity-skip ResNets avoid exploding or vanishing gradients when forward signals remain controlled, with residual scaling of O(1/d) or less sufficient at initialization.Here d denotes network depth.Source: §2 · p. 2

Alternate normalizers

• LayerNorm, InstanceNorm, and GroupNorm remove batch-size dependence, but their benefits and drawbacks vary across batch-size regimes and can differ from BatchNorm.The passage notes that these alternatives often work better for very small batches.Source: §2 · p. 2

SIGNAL PROPAGATION PLOTS

• SPPs measure activation statistics at every residual-block output after initializing one weight set and feeding Gaussian input batches through identity residual blocks.The plots use an identity residual form x_ℓ+1 = f_ℓ(x_ℓ) + x_ℓ and summarize forward propagation.Source: §3 · p. 3
• The three core SPP statistics are average channel squared mean, average channel variance, and residual-branch variance before merging with the skip path.Channel-squared means detect nonzero per-channel means, channel variance tracks signal magnitude, and branch variance assesses initialization.Source: §3 · p. 3

Scope and limitations

• SPPs are useful practical diagnostics but cover only forward-pass statistics and do not capture every property of signal propagation.The authors report that real-data inputs do not meaningfully change the key trends.Source: §3 · p. 3

Signal propagation behavior

• In a 600-layer BatchNorm ResNet, average channel variance grows linearly within stages and resets near 1 at transition blocks.The figure compares BN-ReLU-Conv and ReLU-BN-Conv orderings using blue and red plots, respectively.Source: §3 · p. 3, Figure 1 · p. 3
• With BN-ReLU-Conv ordering, average squared channel means also grow linearly between transition blocks because rectified residual-branch inputs produce nonzero-mean outputs.This explains why the residual-branch variance can be approximately zero in the described analysis.Source: §3 · p. 4

Normalizer-free ResNets

• The proposed residual block x_ℓ+1 = x_ℓ + αf_ℓ(x_ℓ/β_ℓ) mimics BatchNorm by normalizing block inputs and controlling variance growth through α.The residual function is variance preserving at initialization, while β_ℓ is based on the expected activation standard deviation.Source: §4 · p. 4
• Transition-layer shortcut convolutions also receive normalized inputs, resetting the signal variance to 1 + α^2 at each new stage.This reproduces the variance reset that occurs in normalized ResNets at transition layers.Source: §4 · p. 4

ReLU activations induce mean shifts

• ReLU with Gaussian He-initialized weights causes squared channel means to grow rapidly with depth and exceed channel variance, creating a substantial mean shift.Removing ReLU keeps squared channel means near zero and residual-branch variance around one.Source: §4.1 · p. 5

Scaled Weight Standardization

• Scaled Weight Standardization reparameterizes convolutional filters using fan-in mean and variance so outputs have zero mean and can be variance preserving with a suitable gain γ.The transformation is applied differentiably during training and does not depend on batch elements.Source: §4.2 · p. 5
• Scaled Weight Standardization eliminates the depth-wise growth of average channel squared means, producing SPPs nearly identical to batch-normalized ReLU-BN-Conv networks.The residual-branch variance remains close to one, with slight late-network decay attributed to zero padding.Source: §4.2 · pp. 5–6

Determining nonlinearity-specific constants

• The gain γ depends on the nonlinearity: analytical variance is available for ReLU or tanh, whereas nonlinearities such as SiLU may require numerical approximation.The method assumes approximately unit-normal inputs when deriving γ, an approximation reported to work well in practice.Source: §4.3 · p. 6

5.1 AN EMPIRICAL EVALUATION ON RESNETS

• 79.5% top-1 accuracy was achieved by the regularized NF-ResNet-288, with NF-ResNets outperforming FixUp and SkipInit and remaining competitive with BN across depths.Unregularized 288-layer models sometimes collapsed, while reducing the learning rate prevented instability at the cost of test accuracy.Source: §5.1 · p. 7, Table 1 · p. 7
• NF-ResNets achieved significantly better performance than BN-ResNets at very small batch sizes and were unaffected when the batch size decreased from 8 to 4.These experiments used ImageNet ResNet-50s trained for 15 epochs without stochastic depth or dropout.Source: §5.1 · pp. 7–8, Table 2 · p. 8

5.2 DESIGNING PERFORMANT NORMALIZER-FREE NETWORKS

• Normalizer-Free EfficientNets underperformed their batch-normalized baselines; the B0 variant reached 73.5% top-1, 3.2% below baseline.Removing Scaled WS from depth-wise convolutions improved accuracy but reduced training stability, motivating NF-RegNets with grouped convolutions.Source: §5.2 · pp. 8–9
• Across FLOP and augmentation settings, NF-RegNets attained comparable but slightly lower test accuracy than EfficientNets while training substantially faster.The comparison spans EfficientNet-style scaling and augmented settings using CutMix+MixUp for NF-RegNets.Source: §5.2 · p. 9
• AutoAugment and RandAugment degraded NF-RegNet performance and stability, while inserting one BatchNorm layer after the first convolution restored stable training without improving test accuracy.The authors suggest these augmentations may implicitly depend on activation normalization.Source: §5.2 · p. 9

• Normalizer-Free Networks remove activation normalization layers from residual networks while achieving performance competitive with EfficientNets on ImageNet across FLOP budgets.Source: §6 · p. 9

Stability, learning rates, and batch sizes

• Unnormalized ResNets can lose the robust learning-rate–batch-size relationship observed for Batch-Normalized ResNets beyond batch size 1024.Both optimal and highest stable learning rates decrease at larger batch sizes, increasing tuning expense.Source: §A.1 · p. 15
• Collapsed networks can produce NaN outputs because poorly conditioned weight spectra make the network Lipschitz constant large, although weight magnitudes remain moderate.Scaled Weight Standardization does not prevent this spectral-conditioning failure.Source: §A.1 · p. 15
• SPPs provide no guarantees far from initialization, so good initial signal propagation is necessary but not sufficient for stable optimization.This limitation motivates parameterizations such as Scaled Weight Standardization rather than relying only on initialization strategies.Source: §A.1 · p. 16

Training speed

• NF-RegNets are substantially faster to train than EfficientNets while attaining comparable but slightly lower test accuracies across FLOP and augmentation settings.NF-RegNets use CutMix+MixUp because AutoAugment and RandAugment degrade their performance and stability.Source: §A.2 · p. 16, Table 5 · p. 16

Affine gains and biases

• Affine gains and biases are not responsible for training instability; collapse instead arises from the spectra of the weights.Removing the affine transform does not improve stability in collapsing settings.Source: §B.1 · p. 17

Stochastic depth

• Adding or removing Stochastic Depth does not reduce or improve stability in practice, despite the possibility that many dropped blocks could attenuate signals.A more principled skip-connection rescaling variant was not necessary.Source: §B.2 · p. 17

Model modifications

• Squeeze-and-Excitation layers reduce signal magnitude by a factor of 0.5, which the models correct by multiplying the output by 2.Unmodified S+E blocks do not train stably in the reported NF-RegNet experiments.Source: §B.3 · p. 17
• Replacing strided 1x1 convolutions with average pooling followed by 1x1 convolutions attenuates signal by a factor of k for kernel size k × k, without requiring correction.The resulting β mis-estimation at initialization was not treated as requiring an adjustment.Source: §B.4 · p. 17

Architecture

• An expansion convolution to 1280w channels substantially improves performance by giving the classifier a larger feature basis.Without sufficient feature width, the classifier underfits and performs worse.Source: Appendix C · p. 18
• NF-RegNets use average pooling before transition convolutions, SiLU activations with EMA, and wider Squeeze-and-Excitation layers to improve the architecture.The wider S+E design increases parameter counts with minimal FLOP effect.Source: Appendix C · p. 18

Compound scaling

• The models apply EfficientNet compound scaling by increasing width, depth, and input resolution to target approximately matched FLOP budgets.EfficientNet B0–B5 width and depth multipliers are reused, with image resolution tuned for similar computational cost.Source: Appendix C · p. 18

Training

• Training uses SGD with Nesterov momentum, 360 epochs, batch size 1024, a five-epoch warmup to learning rate 0.4, cosine annealing, and EMA decay 0.99999.The larger EMA decay accounts for EfficientNet’s additional averaging of BatchNorm statistics.Source: Appendix C · p. 18
• Some models reach peak performance before training ends, but the reported results use final rather than early-stopped weights.As learning-rate decay slows weight changes, averaged weights can move back toward less performant underlying weights.Source: Appendix C · p. 19

Activation scaling

• The SiLU normalization constant γ can be estimated empirically by averaging output variances over many random vectors.The appendix illustrates this procedure with 1024 random vectors of size 256.Source: §D.1 · p. 20

Scaled Weight Standardization

• ScaledWSConv2d standardizes convolution weights using per-output-channel means and variances, with optional gains and numerical stabilization.The implementation computes fan-in and divides centered weights by (var * fan_in + eps)^0.5 before convolution.Source: §D.2 · p. 20

Residual blocks

• NFBlock applies activation, bottleneck and spatial convolutions, squeeze-and-excitation, skip scaling, and shortcut addition.The residual output is multiplied by skipinit_gain and alpha before being added to the shortcut.Source: §D.2 · pp. 21–22

Activations

• The model supports ReLU, SiLU, and identity activations with baked-in scaling constants intended to preserve signal magnitudes.The listed nonlinearities divide ReLU and SiLU outputs by fixed gains.Source: §D.2 · p. 22

Variance calibration

• NFRegNet tracks expected variance across blocks, resets it at stage transitions, and uses its square root as the block beta scaling factor.The implementation increments expected variance by alpha^2 after each block and resets it when a transition block begins.Source: §D.2 · p. 23

Measurement definitions

• Signal Propagation Plots measure average channel mean squared, average channel variance, and residual average channel variance over the forward pass.These metrics are computed by averaging channel-wise statistics over spatial and batch dimensions.Source: Appendix F · p. 27

Unnormalized ResNets

• Unnormalized ResNetV2 with ReLU and He initialization exhibits logarithmic growth in all three measured signal properties because of signal explosion.The plot uses a semilog scale and reports growth for average channel mean squared, average channel variance, and residual average channel variance.Source: Figure 6 · p. 27, Appendix F · p. 27

Linear activations

• Normalizer-free scaling and BatchNorm are compared in 600-layer linear-activation ResNetV2s after removing stem max-pooling so first-block inputs remain centered.Source: Figure 7 · p. 27

FixUp

• FixUp produces constant within-stage variance through zero-initialized residual weights while retaining variance resets across stages.Source: Figure 10 · p. 28

Post-activation ordering

• Post-activation ResNetV1 keeps residual average channel variance at 1 through BatchNorm and also shows transition resets from normalized shortcut convolutions.Source: Figure 11 · p. 28

Decoupled parameterization

• Decoupled Weight Standardization ultimately performed worse than forward-mode Scaled WS, despite reduced weight-decay sensitivity and better early EMA accuracy.The authors describe these experiments as cursory and suggest deeper analysis of the interactions.Source: §G.1 · p. 29

Soft WS

• Soft WS became drastically unstable unless its regularization scale remained at the maximal value.The method penalized the squared difference between raw weights and their Scaled WS parameterization.Source: §G.1 · p. 29

SPP design

• The authors found activation means and average pixel norms less informative than the selected channel and residual variance measures for Signal Propagation Plots.Average pixel norm also added no information beyond those variance measures and was harder to interpret.Source: §G.2 · p. 29

Alternative activations

• SeLU-based normalizer-free networks trained stably but were difficult to tune and did not readily recover BatchNorm-ResNet performance.Source: §G.2 · p. 29

Pascal VOC semantic segmentation

• Pascal VOC semantic segmentation experiments use FCN with modified final-stage dilation and added dilated convolutions on the normalizer-free ResNet backbones.Source: §H.1 · p. 30, Table 6 · p. 30

NYUv2 depth estimation

• NYUv2 depth-estimation experiments use downsampled, cropped images with random flips and color augmentations, following a stated prior protocol.Source: §H.2 · p. 30, Table 7 · p. 30