PaperlayerView exact revision on arXiv

Source-linked AI summary

Striving for Simplicity: The All Convolutional Net

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, Martin Riedmiller

arXiv:1412.6806v3cs.LGcs.CVcs.NE

TL;DR

The paper asks which components of standard CNNs are necessary for image recognition and tests whether pooling and fully connected layers can be removed. It finds that an all-convolutional architecture with strided subsampling matches or slightly outperforms state-of-the-art results on CIFAR-10 and CIFAR-100 while remaining competitive on ImageNet.

  • Problem

    The paper investigates which components of standard CNNs are necessary for achieving strong object-recognition performance.

  • Method

    The authors replace pooling with stride-2 convolution and build networks consisting solely of convolutional layers with occasional dimensionality reduction.

  • Results

    All-convolutional networks match or slightly outperform state of the art on CIFAR-10 and CIFAR-100 and achieve competitive results on ImageNet.

  • Takeaways & Limitations

    Pooling is not always necessary for strong CNN performance when convolutional layers provide subsampling.

  • Takeaways & Limitations

    The evaluated models could potentially improve with larger model sizes or more thorough hyperparameter searches.

Abstract

from arXiv · show

Most modern convolutional neural networks (CNNs) used for object recognition are built using the same principles: Alternating convolution and max-pooling layers followed by a small number of fully connected layers. We re-evaluate the state of the art for object recognition from small images with convolutional networks, questioning the necessity of different components in the pipeline. We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks. Following this finding -- and building on other recent work for finding simple network structures -- we propose a new architecture that consists solely of convolutional layers and yields competitive or state of the art performance on several object recognition datasets (CIFAR-10, CIFAR-100, ImageNet). To analyze the network we introduce a new variant of the "deconvolution approach" for visualizing features learned by CNNs, which can be applied to a broader range of network structures than existing approaches.

1 INTRODUCTION AND RELATED WORK

The paper questions which components of conventional CNNs are necessary for state-of-the-art object recognition by studying a homogeneous architecture made solely of convolutional layers. It reports state-of-the-art performance without complicated activation functions, response normalization, or max-pooling, and evaluates the simplification across multiple datasets.

  • Conventional CNNs: Modern object-recognition CNNs typically alternate convolution and max-pooling layers before a small number of fully connected layers.These networks commonly use piecewise-linear activations, large parameterizations, and dropout regularization.
  • Simplified architecture: The paper investigates the simplest architecture it could conceive: a homogeneous network consisting solely of convolutional layers with occasional dimensionality reduction through strided convolutions.This design is used to examine which CNN components are actually necessary for state-of-the-art performance.
  • Results and evaluation: The simplified CNN reaches state-of-the-art performance without complicated activation functions, response normalization, or max-pooling.The authors empirically study the transition from a more standard architecture through an ablation study on CIFAR-10.
  • Results and evaluation: The model is compared with the state of the art on CIFAR-10, CIFAR-100, and the ILSVRC-2012 ImageNet dataset.The reported evaluation follows the CIFAR-10 ablation study and assesses the simplified CNN across several object-recognition benchmarks.

2 MODEL DESCRIPTION - THE ALL CONVOLUTIONAL NETWORK

The model replaces pooling with strided convolution and uses small convolutional kernels to build a simpler all-convolutional architecture. The section also distinguishes two pooling-replacement strategies and notes that ReLU convolutions do not exactly implement p-norm pooling.

  • Pooling replacement: The network replaces each pooling layer with a convolutional layer using stride two, removing pooling from the architecture.This is the section’s primary departure from standard CNNs.
  • Pooling replacement: Two alternatives are proposed: increase the preceding convolution’s stride or replace pooling with a normal convolution whose stride exceeds one.The second option preserves existing convolutional layers, whereas the first reduces their overlap.
  • Pooling replacement: Increasing the preceding layer’s stride can reduce overlap and hurt recognition accuracy, while replacing pooling with convolution increases overall network parameters.The first approach is equivalent to considering only the top-left feature response.
  • Network design: The architecture uses small convolutional kernels with k < 5, which can reduce network parameters and provide a form of regularization.This design follows recent high-performing models and further unifies the architecture.
  • Limitation: A convolution with rectified linear activation cannot by itself implement the p-norm computation required for proper pooling.Implementing proper pooling requires a special nonlinearity, such as a squaring operation.

3 EXPERIMENTS

Experiments across CIFAR-10, CIFAR-100, and ImageNet evaluate simplified convolutional architectures, including replacing max-pooling with strided or convolutional layers. All-convolutional models perform competitively on CIFAR and ImageNet, while a modified deconvolution method improves higher-layer feature reconstructions.

  • 3.1 Experimental setup: Experiments span CIFAR-10, CIFAR-100, and ILSVRC-2012 ImageNet, using CIFAR-10 for an in-depth architecture study.A large CIFAR-10 model requires approximately 10 hours to train on a modern GPU.
  • 3.1 Experimental setup: The study compares three base CNNs of increasing depth and parameter count, plus variants removing, replacing, or preceding max-pooling with convolutional operations.The variants are Strided-CNN, All-CNN, and ConvPool-CNN; the latter controls for effects from increased model size.
  • 3.2 CIFAR experiments: 12.5% error is achieved by the simplest model on unaugmented CIFAR-10, while merely removing max-pooling and increasing stride diminishes performance in all settings.The comparison trains all models with stochastic gradient descent and fixed momentum of 0.9.
  • 3.2 CIFAR experiments: The simple all-convolutional model derived from base model C outperforms the best previously reported CIFAR-10 result with and without data augmentation.The results suggest that stacks of convolutional layers with occasional stride-2 subsampling are sufficient for strong benchmark performance; fractional max-pooling remains better but uses roughly 50M parameters.
  • 3.3 ImageNet experiment: 41.2% Top-1 validation error is obtained on ILSVRC-2012 using a center 224 × 224 patch, compared with 40.7% for Krizhevsky et al., while using fewer than 10 million parameters.The up-scaled ALL-CNN-B took roughly 4 days to train on a Titan GPU; a larger 3 × 3-convolution network was not completed because training would take several weeks.

4 DISCUSSION

The experiments show that very simple convolutional architectures can achieve strong recognition performance, while explicit pooling is not always necessary. The paper also introduces a sharper visualization method that works without max-pooling switches, while positioning the results as a baseline rather than a rejection of more sophisticated models.

  • Key observations: Convolution-only networks with subsampling match or slightly outperform state-of-the-art results on CIFAR-10 and CIFAR-100 and achieve competitive results on ImageNet.These architectures use nothing but convolutions and subsampling.
  • Key observations: Explicit max-pooling does not always improve CNN performance, especially when a network is large enough to learn necessary invariances through convolutional layers.This contrasts with previous observations about the benefits of pooling.
  • Key observations: The proposed visualization method produces sharper representations of descriptive image regions than previous methods and works without max-pooling switches.The method is presented as simple and applicable when positions of maxima in pooling regions are unavailable.
  • Scope and limitations: The paper seeks the minimum necessary CNN ingredients for recognition and establishes a strong baseline rather than discouraging pooling or sophisticated activation functions.The authors explicitly frame the work as an attempt to identify necessary components, not as a universal prohibition of other design choices.
  • Scope and limitations: All evaluated models could potentially improve through increased overall model size or more thorough hyperparameter search.The passage ends with this qualification about the reported results.

APPENDIX · A LARGE ALL-CNN MODEL FOR CIFAR-10

The appendix presents a large All-CNN model for CIFAR-10 derived from Benjamin Graham’s spatially sparse network. The model uses leaky ReLU units, a larger centered input, and an architecture specified in Table 5.

  • A LARGE ALL-CNN MODEL FOR CIFAR-10: The large All-CNN model is derived from Benjamin Graham’s spatially sparse network.The passage refers readers to Graham (2015) for an explanation of the underlying network.
  • A LARGE ALL-CNN MODEL FOR CIFAR-10: The network uses leaky ReLU units instead of ReLUs.
  • A LARGE ALL-CNN MODEL FOR CIFAR-10: Leaky ReLU units were selected because they were found to speed up training.
  • A LARGE ALL-CNN MODEL FOR CIFAR-10: The model requires a much larger input size than the original 32 × 32 pixel image.
  • A LARGE ALL-CNN MODEL FOR CIFAR-10: The 32 × 32 pixel image is centered within the larger input.
  • A LARGE ALL-CNN MODEL FOR CIFAR-10: Table 5 gives the complete architecture of the Large All-CNN network for CIFAR-10.

B IMAGENET MODEL

The section presents the complete architecture of the network trained on the ILSVRC-2102 ImageNet dataset, referring readers to Table 6 for its specification.

  • B IMAGENET MODEL: The complete model architecture for the ILSVRC-2102 ImageNet network is provided in Table 6.The passage identifies Table 6 as the source for the architecture specification.
  • B IMAGENET MODEL: Table 6 is titled “Architecture of the ImageNet network.”The table is explicitly labeled as the ImageNet network architecture.

C ADDITIONAL VISUALIZATIONS

The section presents feature visualizations from the network’s last convolutional layer and pre-softmax layer, and adds a max-pooling model for fair comparison of deconvnet and guided backpropagation.

  • Feature visualizations: Figures 4 and 5 visualize features from conv12 and global pool, respectively, in the ImageNet-trained network.Figure 4 uses the single largest activation in conv12, while Figure 5 uses the single largest activation in global pool.
  • Method comparison: Figure 6 compares deconvnet and guided backpropagation using visualizations from fc8 of the max-pooling Caffenet reference network trained on ImageNet.The figure presents reconstructions for 4 different images.
  • Feature visualizations: Figure 4 shows descriptive image regions from the single largest conv12 activation using different methods across four images.The reconstructions are presented for 4 different images.
Loading 1412.6806v3…