PaperlayerView exact revision on arXiv

Source-linked AI summary

All you need is a good init

Dmytro Mishkin, Jiri Matas

arXiv:1511.06422v7cs.LG

TL;DR

Very deep networks lack a general, efficient end-to-end training procedure. The paper proposes LSUV, which combines orthonormal weight initialization with sequential output-variance normalization, and reports near-state-of-the-art results across several datasets while outperforming specialized systems.

  • Problem

    Very deep networks are difficult to train end-to-end, particularly thin networks deeper than five layers with uniform initialization.

  • Method

    LSUV initializes convolutional and inner-product layers orthonormally, then sequentially scales each layer’s output variance to one.

  • Results

    LSUV enables very deep networks trained with standard SGD and achieves near-state-of-the-art results on MNIST, CIFAR, and ImageNet while outperforming FitNets and Highway.

  • Takeaways & Limitations

    A simple initialization procedure can support thin, very deep networks with performance comparable to or better than wider, shallower networks.

  • Takeaways & Limitations

    The authors do not claim that proper initialization can always replace batch normalization, especially on large datasets such as ImageNet.

Abstract

from arXiv · show

Layer-sequential unit-variance (LSUV) initialization - a simple method for weight initialization for deep net learning - is proposed. The method consists of the two steps. First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one. Experiment with different activation functions (maxout, ReLU-family, tanh) show that the proposed initialization leads to learning of very deep nets that (i) produces networks with test accuracy better or equal to standard methods and (ii) is at least as fast as the complex schemes proposed specifically for very deep nets such as FitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)). Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets and the state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100 and ImageNet datasets.

1 INTRODUCTION

Very deep networks are difficult to train end-to-end, motivating a simple initialization procedure that supports thin, deep models with standard SGD.

  • 1 INTRODUCTION: Thin deep networks are attractive because they combine accuracy with inference-time efficiency.
  • 1 INTRODUCTION: Very deep networks lack a general, repeatable, and efficient end-to-end training procedure.Prior approaches included progressively adding layers or using specialized initialization and training schemes.
  • 1 INTRODUCTION: The paper proposes a simple initialization procedure that, with standard SGD, leads to state-of-the-art thin and very deep neural networks.
  • 1 INTRODUCTION: Weight-update scaling between 0.1% and 1% leads to convergence, whereas larger scaling diverges and smaller scaling leaves the network near its initial state.

2 INITIALIZATION IN NEURAL NETWORKS

Initialization determines whether signals and gradients remain usable through deep networks, while prior methods address this through scaling formulas, gating, distillation, or orthonormal weights.

  • 2 INITIALIZATION IN NEURAL NETWORKS: If each layer scales its input by k, the final scale becomes k^L, causing exploding outputs for k > 1 and diminishing signals and gradients for k < 1.
  • 2 INITIALIZATION IN NEURAL NETWORKS: Glorot and He initialization estimate weight scales from layer dimensions, with He et al. extending the approach to ReLU networks.
  • 2 INITIALIZATION IN NEURAL NETWORKS: Alternative approaches regulate error norms, mimic teacher networks, or control information and gradient flow with gates.These include Random Walk Initialization, knowledge distillation and Hints initialization, and Highway-style gating.
  • 2 INITIALIZATION IN NEURAL NETWORKS: Orthonormal matrix initialization performs better than Gaussian noise in linear networks and also works with nonlinearities.

3 LAYER-SEQUENTIAL UNIT-VARIANCE INITIALIZATION

LSUV combines orthonormal weights with layer-by-layer output-variance normalization, using data to handle nonlinear and composite architectures beyond existing formulas.

  • 3 LAYER-SEQUENTIAL UNIT-VARIANCE INITIALIZATION: Existing initialization formulas do not cover nonlinearities such as tanh and maxout or layers such as pooling and local normalization.
  • 3 LAYER-SEQUENTIAL UNIT-VARIANCE INITIALIZATION: LSUV first initializes each convolutional or fully connected layer with orthonormal matrices, then iteratively rescales weights using the layer’s output variance.
  • 3 LAYER-SEQUENTIAL UNIT-VARIANCE INITIALIZATION: The method can be viewed as orthonormal initialization combined with batch normalization applied only to the first mini-batch.Unlike full batch normalization, it uses initial weight decorrelation and does not continuously transform batch outputs.
  • 3 LAYER-SEQUENTIAL UNIT-VARIANCE INITIALIZATION: The desired variance is reached in 1–5 iterations in the reported experiments, while the tolerance parameter has little performance effect from 0.01 to 0.1.
  • 3 LAYER-SEQUENTIAL UNIT-VARIANCE INITIALIZATION: Input-activation normalization brought no improvement over LSUV output normalization, and Gaussian pre-initialization caused a small but consistent performance decrease.

4 EXPERIMENTAL VALIDATION

The experiments evaluate LSUV on very deep, thin FitNet architectures across MNIST and CIFAR-10/100, including comparisons with alternative initializations and training setups.

  • Experimental setup: Very deep and thin nets were trained in a single stage using architectures proposed by Romero et al. (2015).The experiments use the FitNet architectures listed in Table 1.
  • MNIST: LSUV outperformed orthonormal and Hints initialization on the FitNet-MNIST architecture.The MNIST experiment used 60,000 28x28 grayscale digit images without data augmentation.
  • MNIST: Replacing the MNIST softmax classifier with SVM further improved the LSUV result.This produced the FitNet-LSUV-SVM result in Table 2.
  • Training setup: FitNets were trained with momentum SGD for 230 epochs, whereas the cited Highway and FitNets studies trained their networks for 500 epochs.The training schedule reduced the learning rate by a factor of 10 after epochs 100, 150, and 200.
  • CIFAR-10/100: LSUV and orthonormal initialization outperformed Hints-trained FitNets on CIFAR-10/100 with mirroring and random-shift augmentation.The gain was much larger on fine-grained CIFAR-100 than on CIFAR-10.
  • CIFAR-10/100: FitNets with LSUV initialization outperformed larger networks on CIFAR-100, including Large-All-CNN and Fractional Max-pooling.These larger networks used affine and color dataset augmentation.

5 ANALYSIS OF EMPIRICAL RESULTS

Experiments show that LSUV supports very deep networks across activation functions and residual architectures, while matching batch normalization in accuracy and reducing initialization overhead. On ImageNet, results differ by architecture: LSUV improves GoogLeNet but trails the standard CaffeNet initialization in final precision.

  • 5.1 INITIALIZATION STRATEGIES AND NON-LINEARITIES: Orthonormal-based methods outperform scaled Gaussian initialization across tested activations except tanh, while LSUV consistently improves on orthonormal initialization.LSUV also converges faster and achieves better generalization error for all tested activations except tanh.
  • 5.1 INITIALIZATION STRATEGIES AND NON-LINEARITIES: LSUV is the only initialization that converges for all tested residual-net nonlinearities without additional tuning, except sigmoid.Residual training improves ReLU and maxout results but does not help the tanh-based network.
  • 5.2.1 WHERE TO PUT BN – BEFORE OR AFTER NON-LINEARITY?: Batch normalization performs better after the nonlinearity in most runs, while LSUV matches batch-normalized FitNet-4 in accuracy versus wall-clock time.Batch normalization needs fewer iterations, but each iteration is slower because of extra computations.
  • 5.3 IMAGENET TRAINING: LSUV reduces CaffeNet’s starting flat-loss period from 0.5 to 0.05 epochs but finishes 1.3% below standard CaffeNet precision; GoogLeNet reaches 0.680 versus 0.672.LSUV learns faster than the original GoogLeNet initialization and has better test accuracy throughout training.
  • 5.4 TIMINGS: LSUV initialization takes 3.5 minutes in the slowest case, which is negligible compared with training time.The overhead includes SVD decomposition of weight matrices, including a 9216x4096 matrix for CaffeNet’s fc6 layer.

6 CONCLUSIONS

The paper concludes that LSUV is a simple, fast initialization strategy that enables very deep networks to train with standard SGD. Across MNIST, CIFAR, and ImageNet, it achieves near-state-of-the-art results and outperforms more specialized systems in the reported experiments.

  • 6 CONCLUSIONS: LSUV is a simple initialization strategy that performs as well as complex learning schemes requiring auxiliary networks.The procedure is described in six lines of pseudocode.
  • 6 CONCLUSIONS: LSUV enables very deep networks to learn with standard SGD, works quickly across activation functions, and achieves near-state-of-the-art results on MNIST, CIFAR, and ImageNet.The reported comparisons include FitNets and Highway networks, which were designed specifically for very deep nets.
  • 6 CONCLUSIONS: Experiments confirm that thin, deep networks can match or outperform wider, shallower networks while using fewer parameters and faster inference.This conclusion follows the reported comparison between thin deep networks and wider shallow ones.

A.1 INFLUENCE OF MINI-BATCH SIZE TO LSUV INITIALIZATION

The experiment tests whether LSUV’s variance-estimation mini-batch size affects training under tanh, where LSUV performs worst. Results show little difference between small and large mini-batches except at the extreme two-sample case.

  • LSUV mini-batch size was tested with tanh activation, the setting where LSUV showed its worst performance.
  • The training mini-batch remained fixed across initializations; only the mini-batch used for variance estimation differed.
  • There was no difference between small and large variance-estimation mini-batches except in the extreme case of two samples.

A.2 LSUV WEIGHT STANDARD DEVIATIONS IN DIFFERENT NETWORKS

The paper compares LSUV’s resulting filter-weight standard deviations with those produced by other initialization schemes across layers and nonlinearities in FitNet4 on CIFAR-10.

  • Tables 8 and 9 report filter-weight standard deviations for LSUV and other initialization schemes.
  • The comparisons cover different initializations in FitNet4 on CIFAR-10 with ReLU and different nonlinearities.

A.3 GRADIENTS

The paper evaluates how activation-variance normalization affects gradient variance across layers. After ten mini-batches, convolutional-layer gradient variance is close to 10^-9 and more stable than reference methods except MSRA.

  • After 10 mini-batches, the average gradient variance was measured across all layers to assess activation-variance normalization.
  • Gradient variance was close to 10^-9 for all convolutional layers.
  • Gradient variance was more stable than for the reference methods, except MSRA.
  • Table 10 presents initial gradient variance per layer for different initializations in FitNet4 with ReLU.
Loading 1511.06422v7…