Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

Jonathan Frankle; David J. Schwab; Ari S. Morcos

Source-linked AI summary

Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

Jonathan Frankle, David J. Schwab, Ari S. Morcos

arXiv:2003.00152v3Feb 29, 2020cs.LGcs.AIcs.NEstat.ML

TL;DR

The paper asks how much expressive power lies in affine feature parameters apart from the learned features they transform. It freezes CNN weights at random initialization and trains only BatchNorm’s γ and β, finding surprisingly strong accuracy and feature disabling. The study shows that these parameters can build performant random-feature networks, while its evidence is limited to selected architectures, tasks, and standard hyperparameters.

Problem
The role and expressive power of learned affine feature parameters are difficult to distinguish from the learned features they transform.
Method
The paper freezes all network weights at random initialization and trains only BatchNorm’s γ and β in ResNets.
Results
82% CIFAR-10 accuracy and 32% ImageNet top-5 accuracy are achieved by sufficiently deep networks trained only through BatchNorm’s γ and β.
Takeaways & Limitations
Affine BatchNorm parameters have substantial expressive power and can produce performant networks from random features when distributed throughout the network.
Takeaways & Limitations
The study considers only ResNets on CIFAR-10 and ImageNet and uses standard hyperparameters without targeted hyperparameter search.

Abstract

from arXiv · show

A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this paper, we aim to understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, we investigate the performance achieved when training only these parameters in BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the expressive power of affine parameters in deep learning, but - in a broader sense - they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.

1 INTRODUCTION

Affine feature transformations are widely used, but their independent role and expressive power remain difficult to isolate. The paper studies BatchNorm’s γ and β by freezing random network weights and training only these affine parameters.

Affine transformations multiply each feature by learned γ and add learned β across techniques including multitask learning, style transfer, and generation.
BatchNorm normalizes pre-activations across the mini-batch, then scales and shifts each normalized feature with trainable affine parameters.
0.5% to 2% higher CIFAR-10 accuracy and 2% higher ImageNet accuracy result when γ and β are enabled in ResNets.
The paper isolates affine parameters by freezing all weights at initialization and training only BatchNorm’s γ and β.
At most 0.6% of parameters are trainable, forcing learning to occur through scaling and shifting random features rather than learned features.
82% CIFAR-10 accuracy and 32% ImageNet top-5 accuracy are reached by sufficiently deep networks trained only through γ and β.

2 RELATED WORK

Prior work studies BatchNorm’s optimization effects, random-feature models, and frozen-weight networks, while this paper focuses specifically on affine parameters throughout random-feature CNNs. It also extends earlier BatchNorm-only experiments with broader scale and comparisons against equally sized random parameter subsets.

BatchNorm: BatchNorm has been linked to smoother optimization, protection against exploding activations, improved neuron usage, and other optimization effects.
BatchNorm: Earlier analyses often emphasize normalization or treat BatchNorm as a black box rather than isolating γ and β.
Training only BatchNorm: 61% and 30% CIFAR-10 accuracy were reported in earlier short BatchNorm-only experiments, compared with 80% and 70% for comparable settings here.
Training only BatchNorm: The study expands evaluation across network scales and includes large-scale ImageNet experiments plus analysis of the resulting representations.
Random features: Random-feature research commonly trains a linear readout over random nonlinear features, whereas this work trains affine parameters throughout the network before nonlinearities.
Freezing weights at random initialization: Prior frozen-network work showed that some random-initialized layers can coexist with high accuracy, but did not establish the distinct expressive power of BatchNorm parameters.

3 METHODOLOGY

The experiments use deep and wide ResNets on CIFAR-10 and ImageNet, varying depth, width, and which parameters are trainable. BatchNorm is placed before activation, and results average multiple runs with distinct initializations and data orders.

ResNet architectures: ResNets are used because depth and width can increase features while preserving feasible optimization and parameter scaling.
ResNet architectures: CIFAR-10 and ImageNet ResNets vary depth, while CIFAR-10 models also vary width by multiplicatively increasing channels per layer.
BatchNorm: BatchNorm is placed before activation, with β initialized to 0 and γ sampled uniformly between 0 and 1.
Replicates: Five CIFAR-10 runs and three ImageNet runs are averaged across different initializations, data orders, and augmentation.
Evaluation: Figure 2 compares CIFAR-10 deep and wide ResNets and ImageNet top-1 and top-5 accuracy under different trainable-parameter sets.

4 TRAINING ONLY BATCHNORM

Training only BatchNorm can achieve surprisingly high accuracy using random frozen features, and deeper or wider networks improve performance. The advantage is specific to per-feature affine parameters rather than merely the number of trainable parameters.

Performance: 69.5% test accuracy is achieved by training only 8.3K (0.48%) affine parameters in ResNet-110 on CIFAR-10, versus 93.3% when all 1.7M parameters are trainable.The frozen features are shifted and rescaled through BatchNorm parameters alone.
Scaling depth and width: 82% CIFAR-10 accuracy and 32% ImageNet top-5 accuracy are reached by sufficiently deep networks when only γ and β are trained.ResNet-14 reaches 48% on CIFAR-10, while ResNet-866 reaches 82%; ResNet-50 reaches 17% top-5 on ImageNet, while ResNet-200 reaches 32%.
Scaling depth and width: Increasing depth produces higher CIFAR-10 accuracy than increasing width for a fixed BatchNorm parameter budget, with the depth trend 18% steeper.ResNet-434 is 7 percentage points more accurate than WRN-14-32 despite similar BatchNorm parameter counts.
Interpretation: The results indicate that γ and β provide particular expressive power as per-feature coefficients and biases rather than merely reflecting the raw number of trainable parameters.The experiment characterizes networks that combine and transform random features through learned scaling and shifting.

5 EXAMINING THE VALUES OF γ AND β

BatchNorm-only training changes γ toward wider, more concentrated distributions near zero, naturally disabling many random features. Clamping these small values confirms that a substantial fraction of features is expendable, while activation sparsity is studied through γ and β jointly.

Examining γ: 27% of γ values in ResNet-110 and 33% in ResNet-101 have magnitude below 0.01 when only BatchNorm is trained, compared with 4% and 5% when all parameters are trained.This near-zero spike appears without additional sparsity-inducing steps beyond standard weight decay.
Examining γ: The fraction of γ values with |γ| < 0.01 remains between a quarter and a third across depths and widths, while |γ| < 0.05 rises from 3% in ResNet-14 to 48% in ResNet-866.γ values tend to become smaller in deeper and wider networks.
Small γ values: Clamping all |γ| < 0.01 values to zero does not affect accuracy, indicating that 24% to 38% of features can be disabled when only BatchNorm is trained.The same clamping test leaves accuracy unchanged when all parameters are trainable.
Training only BatchNorm sparsifies activations: γ and β jointly alter normalized pre-activation distributions, so β values near zero do not by themselves establish feature removal.The paper therefore examines ReLU activations directly to assess joint sparsification.
Comparison with full training: Compared with standard training, BatchNorm-only training produces a larger γ variance and a spike at zero, whereas fully trained networks retain activations without disabling features.The authors report that γ still plays a role in fully trained networks even when its values become smaller.

6 DISCUSSION AND CONCLUSIONS

The paper shows that BatchNorm’s affine parameters can produce high-accuracy networks from frozen random features, while also identifying practical limits and directions for broader validation.

Training only BatchNorm gives affine parameters substantial expressive power even when all other weights remain at random initialization.
The learned affine parameters disable more than a quarter of activations while still producing surprisingly high accuracy.
Distributing affine parameters throughout random-feature networks appears more expressive than training only a linear output layer.
The method does not reduce training costs because updating deep BatchNorm parameters still requires full backpropagation.
The study evaluates only ResNets on CIFAR-10 and ImageNet and uses standard rather than specially tuned hyperparameters.

TABLE OF CONTENTS FOR SUPPLEMENTARY MATERIAL

The supplementary material contains additional experimental details, data, and experiments.

The appendices provide additional experiment details, data omitted from the main paper, and further experiments.

Appendix B. The details of the ResNet architectures and training hyperparameters we use.

The supplementary experiments describe ResNet and VGG architectures, training settings, BatchNorm behavior, and additional comparisons involving random parameters.

Appendices D and F: The supplementary material compares BatchNorm-only training with equivalent random-parameter baselines and with training the output layer.
Appendix E: Appendix E varies initializations and finds that setting β to 1 improves BatchNorm-only performance.
Appendix I: Values of γ near zero can be set to zero without affecting accuracy, indicating that those features are not important to learned representations.
Appendix B: The CIFAR-10 and ImageNet ResNet sections specify architecture depth, width, channel structure, BatchNorm placement, and optimization hyperparameters.
Appendix C: The appendices repeat the main experiments for non-residual VGG networks and report qualitatively similar results.
Appendix C: The VGG experiments use VGG-11, VGG-13, VGG-16, and VGG-19 with convolutional layers followed by BatchNorm and max-pooling.

C.2 RESULTS

VGG networks retain substantial performance when only BatchNorm is trained, outperform random per-channel parameters, and learn many near-zero scaling coefficients.

Disabling γ and β has no effect on fully trained VGG performance, unlike the reported ResNet behavior.
13 to 14 percentage points higher accuracy is achieved than with two random parameters per channel.
44% of VGG-11 γ parameters and 48% of VGG-19 γ parameters have magnitudes below 0.01 under BatchNorm-only training.

D COMPARING TRAINING ONLY BATCHNORM TO SMALL RESNETS

For a fixed trainable-parameter budget, small ResNets that learn features outperform larger random-feature networks trained only through BatchNorm affine parameters.

The comparison asks whether learning a smaller number of features is preferable to shifting and scaling a larger number of random features.
Small ResNets outperform training only BatchNorm by up to 5 to 10 percentage points at a similar trainable-parameter budget.The comparison uses grid-searched depths and widths to match total parameter counts.
The study also examines whether alternative random initializations can produce higher accuracy than He normal initialization.It explores uniform, binarized, and orthogonalized normal initializations.

E.2 BATCHNORM INITIALIZATION

BatchNorm-only training benefits from initializing β at 1, while standard training can respond differently; random parameters require substantially larger per-channel budgets to match BatchNorm.

1 to 3 percentage points: setting β = 1 improves BatchNorm-only accuracy across depths and widths.This reaches 84% accuracy for ResNet-866, 75% for WRN-14-32, and 37% top-5 accuracy for ResNet-200.
The β = 1 initialization lowers accuracy when all parameters are trainable and can cause failures on the deepest CIFAR-10 networks.It lowers WRN-14-32 accuracy by 3.5% and ResNet-200 top-5 accuracy by 0.5%.
The authors conclude that ideal BatchNorm initializations differ between BatchNorm-only and standard training, and standard training is initialization-sensitive.
8 random parameters per channel match BatchNorm-only performance in shallower ResNets, while deeper ResNets require 16.These correspond to 4x and 8x the number of BatchNorm parameters, respectively.
Training two random parameters per channel matches or underperforms BatchNorm on CIFAR-10 and only matches it in deeper ImageNet networks.It remains far below training BatchNorm and the output layer in all cases.

G MAKING SHORTCUTS TRAINABLE

Making shortcut and output parameters trainable can improve shallow networks, but the gains diminish with depth and are consistent with adding trainable capacity rather than restoring gradient propagation.

The shortcut experiment tests whether freezing downsampling 1x1 convolutions restricts gradient propagation to earlier layers.Nearly all other shortcuts are identity functions without trainable parameters.
Five points: training 3.2k additional shortcut and output parameters raises ResNet-110 accuracy to 74.6%, but gives no improvement on ResNet-866.The lack of deeper-network gains does not support freezing these parameters as the main gradient-propagation impediment.
Shallow networks benefit most when shortcut and output parameters are trainable, consistent with accuracy gains from adding more trainable parameters.ResNet-14 improves from 48% to 63%, WRN-14-32 from 73% to 87%, and ResNet-101 top-5 from 25% to 72%.
Training only shortcuts and outputs reaches 49% for ResNet-14 and 35% top-5 for ResNet-101, slightly exceeding BatchNorm-only training.
Training only BatchNorm produces many ReLUs that are almost always off or on, unlike all-parameter training, which yields more intermittently active units.The appendix reports distributions of γ and β alongside per-ReLU activation frequencies across ResNet, WRN, ImageNet, and VGG settings.