PaperlayerView exact revision on arXiv

Source-linked AI summary

Greedy Layerwise Learning Can Scale to ImageNet

Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon

arXiv:1812.11446v3cs.LGstat.ML

TL;DR

The paper addresses whether CNNs require jointly learned layers, especially on large-scale tasks where alternative methods often fail. It sequentially trains layers through shallow supervised auxiliary problems and reports competitive ImageNet performance, including results matching or exceeding standard CNN baselines.

  • Problem

    Whether CNN layers need to be learned jointly for high performance remains unclear, while prior alternative training methods often fail on large-scale datasets such as ImageNet.

  • Method

    The paper sequentially builds deep CNNs by training layers with shallow supervised auxiliary problems, including 1-, 2-, and 3-hidden-layer formulations.

  • Results

    Layerwise training scales to ImageNet, matching AlexNet with sequential 1-hidden-layer problems and reaching VGG-level performance and end-to-end learning with deeper auxiliary problems.

  • Takeaways & Limitations

    Layerwise CNN training is a competitive alternative to end-to-end learning and may support memory-constrained training, model prototyping, compression, and parallelized training.

  • Takeaways & Limitations

    The study does not address residual connections or very deep modern networks, and further progress may require optimization designed for industrial-scale architecture and hyper-parameter search.

Abstract

from arXiv · show

Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate layers.

1. Introduction

The paper asks whether CNN layers must be learned jointly for high performance and proposes sequential supervised layerwise learning as an alternative. It targets large-scale settings, where prior layerwise methods had not been convincingly demonstrated to work.

  • The paper asks whether CNN layers need joint learning to achieve high performance, testing this question on challenging ImageNet data.
  • End-to-end learning leaves the functional behavior of intermediate layers indirectly specified and may be inefficient in computation and memory.
  • Sequentially solving shallow supervised problems directly specifies each layer’s objective and can encourage properties such as progressive linear separability.
  • Greedy methods can draw on shallow-network theory while requiring fewer intermediate activations and gradients, benefiting memory-constrained settings.
  • Prior alternative training strategies often worked on smaller datasets but failed on large-scale datasets such as ImageNet.
  • Sequentially solving 1-hidden-layer problems matched AlexNet on ImageNet, while 3-hidden-layer auxiliary problems reached VGG-level performance and end-to-end learning.

2. Related Work

Related work includes unsupervised pretraining, early supervised layerwise construction, boosting-based sequential training, and progressive network growth. This paper distinguishes itself by never fine-tuning the whole network and by targeting settings where deep CNNs lack established competitors.

  • Greedy unsupervised learning was used to initialize deep supervised architectures, while early supervised greedy learning was not shown effective with contemporary techniques.
  • Earlier supervised layerwise methods addressed simple problems and primarily grew architectures according to the data.
  • Huang et al. sequentially trained residual layers, but reported limited-dataset results and often required end-to-end learning for competitive performance.
  • The paper focuses on large-scale settings where deep CNN approaches do not currently have competitors and uses a simpler objective function.
  • Progressive-growth approaches may freeze layers or apply end-to-end learning after stacking, whereas this work never fine-tunes the whole network.

3. Supervised Layerwise Training of CNNs

The paper constructs deep CNNs by training layers successively through shallow supervised auxiliary problems, giving each layer a direct objective while retaining analyzable properties of shallow networks. The framework uses convolutional blocks, auxiliary classifiers, and optional down-sampling, with theoretical motivation from progressive separability and shallow-network optimization.

  • Architecture and auxiliary tasks: The architecture adds one layer at a time through a succession of auxiliary learning tasks for k-hidden-layer CNNs.Each intermediate representation feeds an auxiliary classifier that produces an intermediate prediction.
  • Architecture and auxiliary tasks: The CNN uses convolutional operators, ReLU nonlinearities, auxiliary classifiers, and an optional down-sampling operator between successive representations.The down-sampling operation is applied at selected layers; the cited framework example uses k = 2 and applies P at the input and j = 2.
  • Shallow auxiliary problems: For k = 1, each auxiliary classifier is linear, so the procedure trains a sequence of 1-hidden-layer CNN problems.This formulation permits use of theoretical results and bounds developed for shallow networks.
  • Layerwise optimization: At depth j, the method optimizes the current convolutional parameters and auxiliary classifier while keeping other parameters fixed.The procedure trains a shallow classifier on the current representation, typically using an optimization method such as SGD.
  • Theoretical properties: Theoretical analysis states that the cascade can inherit properties of auxiliary problems, including stability and near-optimality when individual subproblem solutions are near-optimal.The stability example assumes 1-Lipschitz layer operators and bounded biases; the optimization procedure is also required not to worsen its initialization objective.
  • Properties and extensions: Layerwise training is intended to progressively improve linear separability and to build layers that become better inputs for shallow CNNs.The paper also considers auxiliary problems with hidden layers, such as k = 2 or k = 3, while retaining advantages associated with shallower optimization.

4. Experiments and Discussion

The experiments show that sequentially trained auxiliary problems can scale layerwise CNN learning to ImageNet and CIFAR-10, while inducing progressively more separable representations. Using deeper auxiliary problems improves ImageNet performance further, matching end-to-end VGG-11 accuracy and exceeding several benchmarks.

  • Experimental setup: The method uses scalable layerwise CNN architectures with spatial averaging, invertible downsampling, auxiliary classifiers, and optional ensembles of layer outputs.Spatial averaging is critical for scalability at large image sizes, while the ensemble computes a weighted average of auxiliary predictions.
  • 4.1. AlexNet Accuracy with 1-Hidden Layer Auxiliary Problems: 88.3% CIFAR-10 accuracy nearly matches AlexNet’s 89.0%, while end-to-end training reaches 89.7% using 5× more GPU memory than k = 1 training.Earlier sequentially trained 1-hidden-layer methods reached 82.0% and did not exceed leading hand-crafted or unsupervised methods.
  • 4.1. AlexNet Accuracy with 1-Hidden Layer Auxiliary Problems: 79.7% top-5 single-crop ImageNet validation accuracy, or 80.8% with a weighted ensemble, exceeds AlexNet and improves from approximately 23% to approximately 80% across layers.The per-layer progression is near-linear during the first five layers, while final training accuracy reaches 87%, suggesting room for regularization improvements.
  • 4.2. Empirical Separability Properties: Linear separability monotonically increases with depth for k = 1, but deeper auxiliary classifiers produce slower initial gains and higher separability at later layers.The k = 1 objective maximizes linear separation locally, yet this may not be optimal for progressive linear separability.
  • 4.3. Scaling up Layerwise CNNs with 2 and 3 Hidden Layer Auxiliary Problems: 89.8% top-5 accuracy from the k = 3 layerwise ensemble is comparable to VGG-13 and largely exceeds AlexNet, while k = 3 training matches end-to-end VGG-11 accuracy.Reducing the final auxiliary width from 2048 to 512 substantially reduces model size with only limited accuracy loss, and the ensemble is better than the baseline VGG-11 comparison.
  • Limitations: Residual connections and very deep networks remain outside the study’s scope, and newer ImageNet models benefit from industrial-scale architecture and hyper-parameter search.The authors suggest extending layerwise learning to these modern techniques, but do not evaluate that extension here.
  • Discussion: Layerwise learning can train larger models than end-to-end learning on the same hardware, with possible applications to 3-D vision and medical imaging.The authors also observe that subsequent layers can improve even when earlier-layer outputs have not converged, suggesting partial parallelization of auxiliary problems.

5. Conclusion

The paper shows that layerwise CNN training using simpler sub-problems and no feedback between layers can scale to ImageNet and remain competitive with standard CNN baselines.

  • Layerwise training scales to ImageNet without end-to-end learning or feedback between layers.The approach trains shallow CNNs with standard ReLU and convolutional components.
  • The method supports larger models under memory constraints and suggests applications including 3-D vision and medical imaging.
  • Layerwise learning also opens possibilities for model prototyping, joint compression and training, parallelized training, and more stable training.

A. Proof of Proposition

The proof develops guarantees for sequential layerwise optimization by combining assumptions on subproblem solutions, stability, Lipschitz operators, and bounded biases.

  • Proposition 3.1 assumes P_j = Id and establishes progressive improvement for a suitable parameter θ0.
  • The analysis considers parameters obtained through optimal layerwise optimization procedures for preceding layers.
  • The subproblem procedure is assumed to produce a 1-Lipschitz operator with a stability bound relating outputs for different input functions.
  • The proof propagates an inductive property from the initialization x̂_0 = x through successive layers.

A Note on Sec.2 From (Mallat, 2016)

The note formalizes samples that remain discriminated by successive network layers and defines the induced label function on their representations.

  • Ω_j contains samples whose distinct labels remain separated after the first j−1 learned transformations.
  • Y_j is the set of representations produced from samples in Ω_j, and f̂_j assigns their original labels.
  • Because Ω_{j+1} ⊂ Ω_j, the set of well-discriminated samples can only shrink as layers are added.

B. Additional Details on Imagenet Models and Performance

Additional ImageNet results show accuracy improving as layers are added, while model-size efficiency depends on the construction and auxiliary-network width.

  • Additional ImageNet performance: Adding layers improves both training and validation accuracy for the ImageNet auxiliary problem solutions.The k = 1 model also substantially overfits, suggesting that regularization may help.
  • Model size: Some models are less parameter-efficient than related literature models because parameter efficiency was not a primary optimization target.
  • Model architecture: The k = 2, 3 model structures use invertible downsampling from 224x224x3 input, with default auxiliary networks of M̃_f = 2048.
  • Model size: The k = 3 model with M̃_f = 512 is more parameter-efficient than VGG-family models while achieving similar performance.

C. Additional Studies

The section reports additional studies aimed at clarifying the system’s critical components and assessing whether greedily learned features transfer.

  • Additional studies examine the system’s critical components.
  • The studies assess the transferability properties of greedily learned features.
  • The reported experiments extend the paper’s analysis beyond its primary system results.

C.1. Choice of Downsampling

The experiments primarily use invertible downsampling to reduce potentially lossy architectural elements and preserve a pure ReLU network. Alternative downsampling methods produce similar CIFAR results, with invertible downsampling slightly better.

  • Choice of Downsampling: Invertible downsampling is primarily used to reduce inherently lossy architectural elements such as average pooling.Compared with maxpooling, it also helps preserve a pure ReLU network, which may aid analysis.
  • Choice of Downsampling: The k = 1 ImageNet network structure uses invertible downsampling on the 224x224x3 input and does not include batch-norm.
  • Choice of Downsampling: On CIFAR with k = 1, average pooling, maxpooling, and strided convolution achieve very similar results, while invertible downsampling is slightly better.The authors interpret this similarity as evidence that the method is rather general.

C.2. Effect of Width

The additional linear-separability analysis finds similar trends when comparing only same-sized models, while larger model sizes primarily differ through increased accuracy.

  • Effect of Width: Same-sized models show a linear-separability trend similar to the aggregated results.
  • Effect of Width: The primary difference across model sizes is increased accuracy.
  • Effect of Width: The analysis provides an additional view of the aggregated linear-separability results.

C.3. Transfer Learning on Caltech-101

The paper evaluates whether the k = 1 ImageNet model’s features support transfer learning on Caltech-101 using linear multinomial logistic regression. The evaluation follows a protocol with 30 training images per class, ten random splits, and average per-class accuracy.

  • Transfer Learning on Caltech-101: The evaluation uses 30 randomly selected images per class for training and the remainder for testing across 10 random splits.Average per-class accuracy is reported.
  • Transfer Learning on Caltech-101: Multinomial logistic regression evaluates features from different network layers, including the final layer, under a linear-model restriction.The study also reports reference accuracy without transfer.
Loading 1812.11446v3…