PaperlayerView exact revision on arXiv

Source-linked AI summary

Systematic evaluation of CNN advances on the ImageNet

Dmytro Mishkin, Nikolay Sergievskiy, Jiri Matas

arXiv:1606.02228v2cs.NEcs.CVcs.LG

TL;DR

Recent CNN improvements and their interactions had not been systematically evaluated, while many design choices lacked guidance or theory. The paper systematically evaluates recent CNN architecture and learning choices individually and in combination on ImageNet classification. The study provides CNN design recommendations and reports that ImageNet-128px preserves the relative ordering of popular architectures versus 224x224 and 300x300 inputs.

  • Problem

    Recent CNN improvements and their interactions had not been systematically evaluated, while many design choices lacked guidance or theory.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically evaluates recent CNN architecture and learning choices individually and in combination on ImageNet classification.

    Source: §1 · p. 2, §3.2.2 · p. 6

  • Results

    The study provides CNN design recommendations and reports that ImageNet-128px preserves the relative ordering of popular architectures versus 224x224 and 300x300 inputs.

    Source: §1 · p. 2, §5 · p. 16

  • Takeaways & Limitations

    The authors report that benchmarking is affordable in time and computation; AlexNet training on ImageNet-128px takes 24 hours on a GTX980.

    Source: §1 · p. 2, §5 · p. 16

    VGGNet224 training would require one month of GPU time, so it was not included in the tests.

    Source: §4 · p. 16

Abstract

from arXiv · show

The paper systematically studies the impact of a range of recent advances in CNN architectures and learning methods on the object categorization (ILSVRC) problem. The evalution tests the influence of the following choices of the architecture: non-linearity (ReLU, ELU, maxout, compatibility with batch normalization), pooling variants (stochastic, max, average, mixed), network width, classifier design (convolutional, fully-connected, SPP), image pre-processing, and of learning parameters: learning rate, batch size, cleanliness of the data, etc. The performance gains of the proposed modifications are first tested individually and then in combination. The sum of individual gains is bigger than the observed improvement when all modifications are introduced, but the "deficit" is small suggesting independence of their benefits. We show that the use of 128x128 pixel images is sufficient to make qualitative conclusions about optimal network structure that hold for the full size Caffe and VGG nets. The results are obtained an order of magnitude faster than with the standard 224 pixel images.

1. Introduction

CNN design has many choices, but little guidance or theory has left researchers relying on trial and error and copying established architectures. This paper systematically evaluates recent improvements, individually and in combination, to provide recommendations for CNN construction.

1. Introduction

  • AlexNet, VGGNet, and GoogLeNet had become de-facto standards following strong ImageNet competition results.The authors describe a field with many CNN design choices but little guidance or theory beyond two classic works and.Source: §1 · pp. 1–2
  • The interactions among recent CNN improvements had not been systematically evaluated, despite their adoption across research areas.The study examines nonlinearity, pooling, learning-rate policy, classifier design, network width, and batch normalization.Source: §1 · p. 2
  • The paper surveys recent CNN developments and evaluates design choices both alone and in combination to derive recommendations.It excludes ResNets because the authors say they were already well covered in papers.Source: §1 · p. 2
  • The paper presents ImageNet-128px as a fast, reliable benchmark whose relative ordering of popular architectures matches results at 224x224 and 300x300 pixels.Training AlexNet on ImageNet-128px takes 24 hours on a GTX980; scripts and data are available online for reproducibility.Source: §1 · p. 2

2. Evaluation

The evaluation uses ImageNet classification experiments and a reduced-resolution benchmark to compare popular architectures. The framework specifies a thinner CaffeNet and fixed learning settings for most experiments.

2.1. Evaluation framework

  • ImageNet-128px limits images to 144xN, where N ≥128, and CaffeNet training takes 24 hours on an NVIDIA GTX980.The test set is not used in experiments; the standard preprocessing rescales images to 256xN and randomly crops a 224x224 square.Source: §2.1 · pp. 2–3

2.1.1. Architectures

  • Reducing input image size leads to a consistent drop in top-1 accuracy around 6% across CaffeNet, GoogLeNet, and VGGNet, without changing their relative order (VGGNet > GoogLeNet > CaffeNet) or accuracy difference.Source: §2.1.1 · p. 3
  • Most experiments use CaffeNet with fully connected layers 6 and 7 reduced from 4096 to 2048 filters to limit overfitting and memory demands.The tested attributes are listed in Table 1, and the architecture is specified in Table 2.Source: §2 · p. 2, §2.1.1 · p. 3, Table 1 · p. 3

2.1.2. Learning

  • The default activation is ReLU after each convolution except the final 1000-way softmax classifier, and fully connected layers are represented as fully convolutional layers for fixed 128x128 inputs.Training uses SGD with momentum 0.9, an initial learning rate of 0.01, and a 320K-iteration schedule with tenfold reductions every 100K iterations.Source: §2.1.1 · p. 4, §2.1.2 · p. 4

3. Single experiments

The single-experiment section isolates CNN design choices, including activation functions. Its activation results favor ELU-based configurations, while showing that some alternatives perform respectably or at higher computational cost.

3.1.1. Previous work

  • Prior ImageNet-scale activation evaluations covered only a small fraction of proposed functions, and the cited ELU evaluation used an architecture designed specifically for that experiment.The paper situates this study against activation-function proposals including LeakyReLU, PReLU, ELU, and maxout.Source: §3.1.1 · p. 4

3.1.2. Experiment

  • The activation study tests ReLU, tanh, sigmoid, VLReLU, RReLU, PReLU, ELU, linear, maxout, APL, and SoftPlus.Maxout is tested at matched effective width and matched computational complexity; the study also tests ReLU in early layers with maxout in later layers.Source: §3.1.2 · p. 5
  • ELU is the best single-performing activation with complexity similar to ReLU; PReLU performs on par, and centered softplus matches ELU.Very Leaky ReLU does not outperform vanilla ReLU.Source: §3.1.2 · p. 5
  • 38.9% top-1 accuracy on ImageNet is achieved even without a nonlinearity, not much worse than the tanh network..Source: §3.1.2 · p. 6
  • ELU after convolutional layers plus maxout after fully connected layers performs best among the tested nonlinearities, with speed close to ReLU.Wide maxout outperforms the other competitors at higher computational cost.Source: §3.1.2 · p. 6

3.2. Pooling

The section evaluates pooling methods and receptive-field sizes, finding that mixed max-and-average pooling performs best and that overlap results depend on zero-padding. Larger pool outputs improve accuracy at a speed cost.

3.2.1. Previous work

  • Pooling combined with striding is used to add invariance while reducing feature-map spatial size; prior work also contrasts overlapping 3x3/2 with non-overlapping 2x2 windows.Krizhevskyi et al. claimed superiority for overlapping pooling, while VGGNet uses non-overlapping pooling.Source: §3.2.1 · p. 6

3.2.2. Experiment

  • The experiments compare average, max, stochastic, and summed average-plus-max pooling, as well as replacing pooling with strided convolutions.Tree and gated pooling were omitted because of convergence problems and results that depended strongly on input image size.Source: §3.2.2 · p. 6
  • Stochastic pooling performs poorly, while the best results come from combining max and average pooling.Removing dropout from the stochastic-pooling network decreased accuracy further; the authors suggest max pooling adds selectivity and invariance while average pooling uses gradients from all filters.Source: §3.2.2 · pp. 6–7, Figure 3 · p. 8
  • Overlapping pooling is inferior to non-overlapping 2x2 pooling without padding, but wins when zero-padding is used.The authors attribute this pattern to larger pool5 outputs and note a speed–performance trade-off.Source: §3.2.2 · p. 7, Figure 3 · p. 8

3.3. Learning rate policy

The section compares learning-rate decay policies and image-preprocessing choices for CNNs. Linear decay performs best among the tested schedules, while learned input color transformations provide an accuracy gain with little computational cost.

3.3. Learning rate policy

  • Linear decay gives the best validation accuracy among the four tested learning-rate policies.The tested policies are step, quadratic, square-root, and linear decay.Source: §3.3 · p. 8, Figure 4 · p. 8, Table 5 · p. 9

3.4.1. Previous work

  • Prior work had explored colorspaces and preprocessing in several applications, but little research had examined optimal choices for CNNs.Examples include vehicle color identification, image super-resolution, and retina-image processing.Source: §3.4.1 · p. 9

3.4.2. Experiment

  • RGB is the best-suited tested colorspace, while Lab fails to improve initial loss after 10K iterations.Replacing training-set mean-and-variance calculation with input batch normalization decreases final accuracy by 0.3%; removing color costs 5.8% to 5.2% accuracy, and grayscale conversion and histogram equalization also hurt performance.Source: §3.4.2 · pp. 9–10, Figure 5 · p. 9
  • The best-performing learned colorspace network gives a 1.4% absolute accuracy gain without significant computational cost.It learns a transformation through 1x1 convolutions jointly with the main network, without involving pixel neighbors.Source: Figure 5 · p. 9, §3.4.2 · p. 10

3.5. Batch normalization

Batch normalization’s effects depend on both network architecture and placement: CaffeNet and GoogLeNet favor different placements, and GoogLeNet’s results are inferior to its plain network.

3.5. Batch normalization

  • CaffeNet and GoogLeNet favored different batch-normalization placements, and GoogLeNet’s results were inferior to its plain network.The placement comparison is reported for ImageNet-128px with ReLU activation.Source: Table 7 · p. 10, §3.5 · p. 10
  • Batch normalization washed out accuracy differences between ReLU-family variants, so the more complex variants were unnecessary.Source: §3.5 · p. 10
  • With batch normalization, sigmoid outperformed ReLU without it, while tanh with batch normalization performed worse than sigmoid with it.Source: §3.5 · p. 11

3.6. Classifier design

The section reviews classifier designs and prior classifier studies, then compares CaffeNet classifier variants and tests how batch size and learning rate affect accuracy and training time.

3.6.1. Previous work

  • Earlier classifier studies considered pre-trained CNN or HOG feature extractors, often with most weights frozen, and sometimes added convolution layers that enhance feature extraction.Source: §3.6.1 · p. 11
  • Common classifier designs pair max pooling with an MLP, spatial pyramid pooling with an MLP, or average pooling with a softmax classifier.Source: §3.6.1 · p. 11

3.6.2. Experiment

  • Averaging spatial softmax predictions and treating MLP layers as convolutional with zero padding gave the best tested results; SPP’s advantage over standard max pooling was less pronounced.Source: §3.6.2 · p. 12, Figure 7 · p. 12

3.7. Batch size and learning rate

  • Keeping learning rate constant across mini-batch sizes hurt performance; Krizhevsky’s constant batch-size–learning-rate product heuristic worked, but batches of 512 or more reduced performance significantly.Source: §3.7 · p. 12, Figure 8 · p. 13
  • Single-example online training did not improve accuracy over batch sizes of 64 or 256 and significantly increased training wall-clock time.Source: §3.7 · p. 12, Figure 8 · p. 13

3.8. Network width

Increasing CaffeNet width produced diminishing accuracy gains, while larger input images improved accuracy at sharply higher computational cost; much of the image-size gain came from deeper-layer spatial extent.

3.8. Network width

  • Increasing CaffeNet width improved accuracy but gains saturated quickly; making it more than three times thicker brought only limited additional accuracy.Reducing filter count caused a quick, significant accuracy drop, while the original network was close to optimal in accuracy per FLOPS.Source: §3.8 · p. 13

3.9. Input image size

  • Across input widths of 96, 128, 180, and 224 pixels, accuracy rose linearly with image size while computation grew quadratically.Source: §3.9 · p. 14, Figure 10 · p. 14
  • After a minimum input size, the gain from larger images came mostly from larger spatial dimensions in deeper layers rather than unseen image details.Source: §3.9 · p. 14, Figure 10 · p. 14

3.10. Dataset size and noisy labels

The experiments examine ImageNet accuracy across training-set sizes and fixed label corruption, alongside a test of removing layer biases.

3.10.1. Previous work

  • Prior dataset-size studies reported accuracy drops when training data were reduced, but their datasets were private and their experiments were not reproducible.Source: §3.10.1 · pp. 14–15

3.10.2. Experiment

  • A larger, more diverse dataset improved accuracy, with performance dropping quickly below a minimum size; clean 400K-image data performed on par with 1.2M images containing 800K correct images.The experiments sampled 200K, 400K, 600K, and 800K training examples and fixed random incorrect labels for 5%, 10%, 15%, or 32% of examples.Source: §3.10.2 · p. 15

3.11. Bias in convolution layers

  • Removing biases from convolutional and fully connected layers reduced accuracy by 2.6% versus the default network.Source: §3.11 · p. 15, Table 8 · p. 15

4. Best-of-all experiments

The paper combines computationally cost-neutral CNN improvements and finds consistent gains for the first three tested networks, but degraded performance for GoogLeNet. VGGNet224 was not tested because training it was prohibitively costly.

4. Best-of-all experiments

  • The combined test applied the improvements to CaffeNet128, CaffeNet224, VGGNet128, and GoogLeNet128.The authors describe these improvements as not increasing computational cost.Source: §4 · p. 15
  • The combined improvements consistently increased performance for CaffeNet128, CaffeNet224, and VGGNet128, while GoogLeNet128 performance degraded.The combination used learned colorspace transform F, ELU in convolution layers, maxout in fully connected layers, linear learning-rate decay, and average plus max pooling.Source: §4 · p. 15, Figure 12 · p. 16
  • VGGNet224 was not included because training it would require one month of GPU time.Source: §4 · p. 16

5. Conclusions

The authors report that systematic benchmarking of recent CNN advances can be done at affordable time and computational cost. They recommend specific choices for network components, training, data, and image resolution.

5. Conclusions

  • Systematic benchmarking of recent CNN advances can be done at affordable time and computational cost.Source: §5 · p. 16
  • Use ELU without batch normalization or ReLU with it, learned RGB colorspace transformation, linear learning-rate decay, and summed average and max pooling.Source: §5 · p. 16
  • Use mini-batches around 128 or 256; if GPU limits require smaller batches, decrease the learning rate proportionally.Source: §5 · p. 16
  • Treat fully connected layers as convolutional and average predictions for the final decision; check for a plateau before expanding training data.Source: §5 · p. 16
  • Prioritize data cleanliness over dataset size, and reduce subsequent-layer stride when image size cannot be increased.Source: §5 · p. 16
  • Use caution when modifying complex, highly optimized architectures such as GoogLeNet.Source: §5 · p. 16

Related papers

Loading 1606.02228v2…