Source-linked AI summary
Data-dependent Initializations of Convolutional Neural Networks
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, Trevor Darrell
TL;DR
CNNs are difficult to train from scratch because initialization affects gradient behavior and convergence, while small fine-tuning datasets can leave generalization sensitive to starting weights. The paper proposes a fast, data-dependent rescaling procedure that equalizes activation statistics and gradient ratios across layers. It reports improved representations on limited-label tasks, faster early training on ImageNet, and better results when rescaling existing unsupervised models.
Problem
CNNs are difficult to initialize reliably from scratch, and different initializations can yield different test performance when fine-tuning on small datasets.
Method
The method uses actual data to normalize activations and rescale layers so gradient ratios are roughly constant, with random-label-free backpropagated errors for initialization.
Results
The initialization improves representations for limited-labeled tasks, accelerates early CNN training on ImageNet, and improves two of three rescaled unsupervised models.
Takeaways & Limitations
Data-dependent rescaling provides a simple initialization that can support deeper CNN exploration and complement unsupervised or self-supervised pre-training.
Takeaways & Limitations
For non-feed-forward DAG architectures, identically distributed affine-layer inputs may not hold, so the method should be applied with care.
Abstract
from arXiv · showhide
Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable. Despite this, few researchers dare to train their models from scratch. Most work builds on one of a handful of ImageNet pre-trained models, and fine-tunes or adapts these for specific tasks. This is in large part due to the difficulty of properly initializing these networks from scratch. A small miscalibration of the initial weights leads to vanishing or exploding gradients, as well as poor convergence properties. In this work we present a fast and simple data-dependent initialization procedure, that sets the weights of a network such that all units in the network train at roughly the same rate, avoiding vanishing or exploding gradients. Our initialization matches the current state-of-the-art unsupervised or self-supervised pre-training methods on standard computer vision tasks, such as image classification and object detection, while being roughly three orders of magnitude faster. When combined with pre-training methods, our initialization significantly outperforms prior work, narrowing the gap between supervised and unsupervised pre-training.
1 INTRODUCTION
CNN performance has advanced across computer-vision tasks, but training from scratch remains difficult and small fine-tuning datasets leave generalization sensitive to initialization. Existing approaches control activation statistics or architecture-specific gradient behavior, motivating a broader principle for equalizing learning across layers.
- CNNs have improved performance across a wide variety of computer-vision tasks, aided by their ability to use large datasets.
- ImageNet pre-training commonly initializes CNNs before fine-tuning on smaller datasets whose labels are more expensive to obtain.
- Different initializations can reach similarly high training-set performance yet produce substantially different test-time performance on small datasets.
- Freezing some layers during fine-tuning generally harms performance, suggesting that small datasets favor adjusting all layers rather than changing only a few.
- Equal learning rates do not ensure equal learning rates across layers because function-preserving weight rescalings can change their gradients.
- Existing initialization methods target activation variance or particular architectures, but deeper networks make direct gradient-preservation strategies difficult.
2 PRELIMINARIES
The paper models CNNs as feed-forward transformations of hidden activations and parameters, with image-shaped intermediate representations and a task loss at the output. It defines a good parameterization as one in which weights across all layers learn equally quickly.
- A CNN is represented as successive transformations z_k = f_k(z_{k−1}; θ_k), where z_k denotes hidden activations and θ_k denotes layer parameters.
- Transformations may be affine layers or fixed nonlinearities such as ReLUs, while θ_k contains weights and biases for affine layers.
- Image inputs and hidden activations are arranged on two-dimensional grids with channels, whereas the final representation is generally non-spatial.
- The final output is converted into a label-dependent loss, and learning minimizes expected loss over the training dataset using backpropagation and stochastic gradient descent.
- The proposed parameterization measures learning speed through gradient-induced weight changes and initializes every layer so its weights learn equally fast.
3 DATA-DEPENDENT INITIALIZATION
The method initializes CNNs in three stages: normalize activation statistics within layers, equalize gradient-based change rates between layers, and optionally use data-driven filter initializations. It is designed to make parameters learn at approximately the same rate while accommodating broad network operations, with care required for non-feed-forward architectures.
- Rate calibration: The method simplifies equal-rate learning by equalizing weight-matrix columns and assuming individual activations are independent of the backpropagated gradient norm.This decouples column-wise change rates from the global layer-wise change rate, but the underlying nonlinear dependence is difficult to control exactly.
- Within-layer weight normalization: Empirical activation normalization rescales each affine layer so outgoing activations have unit variance and mean β using statistics from actual data samples.The implementation estimates per-channel means and standard deviations, then adjusts weights and biases accordingly.
- Within-layer weight normalization: Dozens of input samples are typically sufficient for CNN initialization because the variance of empirical statistic estimates decreases with sample size.The sample set used for estimating activation statistics need not be large in practice.
- Scope and robustness: The empirical strategy guarantees a chosen activation center and scale without assumptions beyond non-zero input variance, extending to arbitrary operations and DAG architectures.However, for non-feed-forward networks such as concatenative architectures, identically distributed affine inputs may not hold, so the method requires care.
- Between-layer scale adjustment: The between-layer procedure iteratively scales parameters toward a geometric-mean change ratio, using damping and roughly 10 iterations to equalize initial change rates.The geometric mean preserves outputs in completely homogeneous networks, while damping prevents unstable corrections.
- Overall procedure: The procedure first initializes weights or filters, normalizes activations to a common distribution, and rescales layers so gradient ratios are constant across layers.This three-stage pipeline is intended to make all weights learn at approximately the same rate.
4 EVALUATION
Across VOC 2007 and ImageNet experiments, the proposed initializations improve early learning and perform competitively across architectures and pre-training comparisons. Their benefits depend on appropriate scaling and can complement stronger optimization or existing self-supervised features.
- Evaluation setup: 5011 training images and 4952 test images define the PASCAL VOC 2007 evaluation for classification and detection.Experiments use CaffeNet primarily, with additional results on GoogLeNet and VGG.
- Scaling and learning rates: Low within-layer change-rate variation shows that the proposed initialization makes weights train more uniformly across layers.The coefficient of variation is low throughout the layers, similarly to the hand-tuned Caffe initialization.
- Scaling and learning rates: Combining within-layer and between-layer scaling makes both Gaussian and k-means initializations perform very well on VOC classification.Within-layer scaling alone can worsen between-layer scaling, whereas the combined adjustment addresses both effects.
- Scaling and learning rates: The initialization complements ADAM and batch normalization, with the combined methods improving over those optimization methods alone.Neither ADAM nor batch normalization alone or together matches simple SGD with the k-means initialization.
- Weight initialization: The proposed initialization substantially improves on carefully chosen random Gaussian weights, while Glorot and He initializations perform worse in the tested architecture.The authors attribute this robustness partly to handling additional pooling and LRN layers not considered by those alternatives.
- Comparison to unsupervised pre-training: K-means initialization keeps pace with most unsupervised pre-training methods despite containing little semantic information.Rescaling two of three unsupervised models significantly improves results, including an 18% relative classification improvement for Doersch et al. (2015).
- Training dynamics: During early CaffeNet and GoogLeNet training, the proposed initializations reduce error or loss faster than reference initialization strategies.After 320,000 CaffeNet iterations, all initializations reach similar accuracy, although the reference initialization later pulls ahead by an uncertain margin.
- Training dynamics: The k-means variant reduces loss slightly faster than the random variant, and removing LRN layers does not slow learning in the comparable model.This suggests LRN may not be necessary under a well-chosen initialization, within these experiments.
5 DISCUSSION
The method is a simple data-dependent CNN initialization that equalizes activation distributions locally and approximately equalizes gradient scaling across layers. Experiments show improved representations, faster early training, and benefits when combined with unsupervised or self-supervised methods.
- The initialization enforces empirically identical activation distributions within layers and roughly uniform global weight-gradient scaling across arbitrarily deep networks.This is achieved through data-dependent rescaling of network weights.
- It improves CNN representations for tasks with limited labeled data, including PASCAL VOC classification and detection.
- It improves representations learned by existing self-supervised and unsupervised methods.
- It substantially accelerates the early stages of CNN training on large-scale datasets such as ImageNet.
- The authors propose the initialization as a way to facilitate further progress in unsupervised and self-supervised learning and more efficient exploration of deeper, larger architectures.