PaperlayerView exact revision on arXiv

Source-linked AI summary

Very Deep Convolutional Networks for Large-Scale Image Recognition

Karen Simonyan, Andrew Zisserman

arXiv:1409.1556v6cs.CV

TL;DR

How convolutional network depth affects large-scale image-recognition accuracy remained an important architecture-design question. The paper evaluates increasingly deep ConvNets built primarily with 3×3 filters, finding that deeper models achieve state-of-the-art ImageNet performance and generalise well to other datasets.

  • Problem

    The paper examines whether increasing ConvNet depth improves accuracy, addressing limited evidence about depth as an architecture-design factor.

  • Method

    The authors evaluate ConvNets of increasing depth, using very small 3×3 convolution filters while fixing other architectural parameters.

  • Results

    Deeper configurations achieve state-of-the-art ImageNet classification and localisation performance, while the learned representations also perform strongly across other image-recognition datasets.

  • Takeaways & Limitations

    The results support depth as an important property of visual representations and show that a conventional ConvNet can reach state-of-the-art performance with substantially increased depth.

Abstract

from arXiv · show

In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

1 INTRODUCTION

The introduction motivates improving ConvNet accuracy through architectural and training changes, then presents more accurate models that achieve state-of-the-art performance on ILSVRC and generalise to other image-recognition datasets. The authors also release their two best-performing models for further research.

  • The paper targets more accurate ConvNet architectures for large-scale image recognition, building on prior improvements to receptive windows, strides, and dense training and testing.The introduction cites Krizhevsky et al. (2012), Zeiler & Fergus (2013), and Sermanet et al. (2014) as relevant prior work.
  • The resulting architectures achieve state-of-the-art accuracy on ILSVRC classification and localisation tasks and perform excellently on other image-recognition datasets.Their features can also work in relatively simple pipelines, such as deep features classified by a linear SVM without fine-tuning.
  • The authors released their two best-performing models to facilitate further research.
  • The paper describes ConvNet configurations, training and evaluation, ILSVRC classification comparisons, object localisation, and generalisation to other datasets.

2 CONVNET CONFIGURATIONS

The paper evaluates ConvNets built under common design principles, varying primarily in depth from 11 to 19 weight layers while using very small convolutional filters. The architecture combines 224 × 224 RGB inputs, ReLU nonlinearities, shared fully connected layers, and selective 1 × 1 convolutions to increase nonlinearity.

  • 2.1 Generic layout: The models process fixed-size 224 × 224 RGB images after mean-RGB subtraction and use predominantly 3 × 3 convolution filters throughout the network.The 3 × 3 filters are the smallest size described as capturing left/right, up/down, and center relationships; one configuration also uses 1 × 1 filters.
  • 2.1 Generic layout: Every configuration ends with two 4096-channel fully connected layers, a 1000-channel ILSVRC classifier, and a soft-max layer, with ReLU applied to hidden layers.The fully connected-layer configuration is shared across all networks, while nearly all networks omit Local Response Normalisation because it does not improve ILSVRC performance and increases computation and memory use.
  • 2.2 Specific configurations: Networks A–E follow the same generic layout while increasing depth from 11 to 19 weight layers by adding convolutional layers.Network A has 8 convolutional and 3 fully connected layers, whereas network E has 16 convolutional and 3 fully connected layers.
  • 2.2 Specific configurations: 144M weights are used by a cited shallower network with wider layers and larger receptive fields, whereas the deeper VGG configurations are reported as not exceeding that parameter count.The comparison emphasizes that increased depth does not require more weights than the referenced shallower design.
  • 2.3 Design choices: Three stacked 3 × 3 layers provide a 7 × 7 effective receptive field, add nonlinearities, and use fewer parameters than one 7 × 7 layer.The stack is also described as regularising larger filters by decomposing them through 3 × 3 filters with intervening nonlinearities.
  • 2.3 Design choices: Configuration C uses 1 × 1 convolutions to add a rectification nonlinearity without changing convolutional receptive fields.Although the 1 × 1 operation is a linear projection when input and output channel counts match, the following rectification introduces additional nonlinearity.

3 CLASSIFICATION FRAMEWORK

The classification framework trains ConvNets with momentum-based mini-batch optimisation, scale-aware crop augmentation, and dense fully-convolutional evaluation. Training uses fixed or randomly sampled image scales, while test-time scale selection can differ from training.

  • Training procedure: Training optimises multinomial logistic regression with mini-batch back-propagation and momentum, using batch size 256, momentum 0.9, and L2 weight decay of 5·10−4.
  • Training procedure: Deeper networks are initialised using selected layers from shallow configuration A to avoid stalled learning caused by unstable random initialisation.The first four convolutional layers and last three fully-connected layers reuse net A, while intermediate layers are random.
  • Training image size: Training crops fixed 224×224 inputs from rescaled images, with random horizontal flips, RGB colour shifts, and either fixed-scale or jittered-scale training.Multi-scale training samples S between 256 and 512 and fine-tunes a single-scale model pretrained at S = 384.
  • Testing: At test time, images are rescaled to a smallest side Q and processed densely by the fully-convolutional network, with Q not necessarily equal to training scale S.Dense evaluation avoids repeated crop recomputation, while multi-crop evaluation can provide finer image sampling and is complementary.
  • Implementation: Multi-GPU data-parallel training provides a 3.75 times speedup on an off-the-shelf 4-GPU system, while one network takes 2–3 weeks to train.

4 CLASSIFICATION EXPERIMENTS

The classification experiments show that deeper VGG configurations, scale jittering, and complementary model fusion progressively improve ILSVRC performance. The best two-model ensemble reaches 6.8% test error and compares favorably with prior state-of-the-art systems, including GoogLeNet’s 6.7% error and Clarifai’s 11.2–11.7%.

  • Single-scale evaluation: Classification error decreases as depth increases from 11-layer configuration A to 19-layer configuration E, while 3 × 3 convolutions outperform same-depth configurations containing 1 × 1 layers.Configuration C is better than B, indicating that added nonlinearity helps, but configuration D is better than C, indicating the importance of spatial context.
  • Single-scale evaluation: Scale jittering during training improves results over fixed training scales, even when testing uses a single scale, supporting augmentation for capturing multi-scale image statistics.Training with S ∈[256; 512] performs significantly better than training with S = 256 or S = 384.
  • Multi-scale evaluation: Test-time scale jittering improves performance over single-scale evaluation, with the best single-network validation result reaching 24.8%/7.5% top-1/top-5 error.The deepest configurations D and E perform best, and the E configuration achieves 7.3% test error on the test set.
  • Evaluation techniques: Multiple-crop evaluation slightly outperforms dense evaluation, while averaging their outputs performs best because the techniques are complementary.The experiments use training scales sampled from [256; 512] and three test scales, {256, 384, 512}.
  • Model fusion: 6.8% test error is achieved by an ensemble of two best-performing multi-scale models using combined dense and multi-crop evaluation.The submitted seven-network ensemble achieved 7.3% test error, while the later two-model ensemble achieved 7.0% with dense evaluation and 6.8% with combined dense and multi-crop evaluation.
  • State-of-the-art comparison: Very deep ConvNets significantly outperform previous-generation models, approach GoogLeNet’s 6.7% error, and outperform Clarifai’s 11.2% with outside data and 11.7% without it.The comparison reports the VGG team’s second-place result in the ILSVRC-2014 classification task.

5 CONCLUSION

The work evaluates convolutional networks up to 19 weight layers, showing that greater representation depth improves classification accuracy and enables state-of-the-art ImageNet performance with a conventional ConvNet architecture.

  • Very deep convolutional networks with up to 19 weight layers were evaluated for large-scale image classification.
  • Greater representation depth was shown to improve classification accuracy.
  • State-of-the-art performance on the ImageNet challenge dataset was achieved using a conventional ConvNet architecture with substantially increased depth.

A LOCALISATION

The section examines the ILSVRC 2014 localisation task, which the authors won with a 25.3% error rate, and defines it as predicting one bounding box for each top-5 class.

  • A LOCALISATION: 25.3% error: the authors won the ILSVRC 2014 localisation challenge.
  • A LOCALISATION: The localisation task requires predicting a single object bounding box for each of the top-5 classes, regardless of how many objects of each class appear.
  • A LOCALISATION: Unlike the paper’s earlier classification analysis, this section focuses on the challenge’s localisation task.

A.1 LOCALISATION CONVNET

The localisation ConvNet predicts bounding-box parameters instead of class scores, using either shared or class-specific regression. Training uses Euclidean loss, while testing ranges from ground-truth-class central-crop evaluation to dense whole-image prediction with greedy merging.

  • A.1 LOCALISATION CONVNET: The localisation ConvNet replaces the final class-score prediction with a bounding-box prediction encoding center coordinates, width, and height.The model supports single-class regression (SCR) with a shared prediction or per-class regression (PCR).
  • A.1 LOCALISATION CONVNET: Training uses Euclidean loss against ground-truth bounding-box parameters instead of the logistic regression objective.Two single-scale localisation models were trained at S = 256 and S = 384 without training-scale jittering for ILSVRC-2014.
  • A.1 LOCALISATION CONVNET: Validation comparisons use only the ground-truth-class prediction from the image’s central crop to factor out classification errors.This protocol is used to compare network modifications on the validation set.
  • A.1 LOCALISATION CONVNET: The full testing procedure densely applies the localisation ConvNet across the whole image and greedily merges spatially close bounding-box predictions.Unlike classification, the final fully connected output is a set of bounding-box predictions rather than a class score map.
  • A.1 LOCALISATION CONVNET: The approach omits Sermanet et al. (2014)’s multiple pooling offsets technique, which can increase prediction-map resolution and further improve results.This is an explicitly stated limitation of the submitted procedure.

A.2 LOCALISATION EXPERIMENTS

The localisation experiments found that per-class regression and fine-tuning all layers performed best in the simplified setting. The resulting VGG system achieved 25.3% test error and won the ILSVRC-2014 localisation challenge, outperforming Overfeat despite using fewer scales and no resolution enhancement.

  • Evaluation protocol: Localisation error followed the ILSVRC criterion, counting a bounding-box prediction as correct when its intersection-over-union with the ground truth exceeded 0.5.The experiments first selected the best setting under a simplified protocol and then evaluated it in a fully fledged scenario.
  • Settings comparison: Per-class regression (PCR) outperformed class-agnostic single-class regression (SCR), contrary to Sermanet et al. (2014), while fine-tuning all layers improved over fine-tuning only fully connected layers.These settings were evaluated with the smallest image side set to S = 384.
  • Comparison with the state of the art: 25.3% test error earned the VGG team the ILSVRC-2014 localisation challenge win and surpassed Overfeat despite fewer scales and no resolution enhancement.The comparison used the team’s best localisation result.

B GENERALISATION OF VERY DEEP FEATURES

ILSVRC-pre-trained very deep ConvNets generalise effectively as fixed image features on smaller recognition datasets. Using multi-scale feature aggregation and linear SVMs, the representations achieve state-of-the-art or competitive performance across VOC, Caltech, action classification, and other recognition tasks.

  • Feature Extraction: The feature pipeline removes the 1000-way classifier, uses 4096-D penultimate-layer activations, aggregates them across locations and scales, L2-normalises the descriptor, and trains a linear SVM with fixed weights.Aggregation uses dense convolutional application, global average pooling, and horizontal-flip averaging.
  • Feature Aggregation: Multi-scale averaging suits VOC, where objects span varied scales, whereas Caltech benefits from stacking because whole-object and object-part features carry scale-specific semantics.VOC uses Q ∈{256, 384, 512, 640, 768}; Caltech uses Q ∈{256, 384, 512}.
  • Image Classification on VOC-2007 and VOC-2012: The representations set a new state of the art on VOC image classification, outperforming the previous best ILSVRC-pre-trained representation by more than 6%.Net-D and Net-E perform identically, while combining them slightly improves results.
  • Image Classification on Caltech-101 and Caltech-256: On Caltech-256, the features outperform the state of the art by 8.6%, while deeper Net-E surpasses Net-D and their combination performs best across the Caltech evaluations.On Caltech-101, the representations are competitive with He et al. (2014).
  • Action Classification on VOC-2012: The best representation stacks Net-D and Net-E features and achieves state-of-the-art VOC-2012 action classification even without bounding boxes, improving further when boxes are included.The approach uses no task-specific heuristics and relies on the representation power of very deep convolutional features.
  • Other Recognition Tasks: The released models also improve recognition beyond these benchmarks, including object detection, semantic segmentation, image caption generation, and texture and material recognition.Girshick et al. (2014) achieved state-of-the-art object detection by replacing Krizhevsky et al. (2012)'s ConvNet with the 16-layer model.

C PAPER REVISIONS

The paper revisions progressed from the initial pre-ILSVRC experiments to added scale-jittering, generalisation, multi-crop, and benchmark-comparison experiments, alongside submission-format changes. The revised paper also made the models publicly available for further research.

  • v3 adds generalisation experiments on PASCAL VOC and Caltech image classification datasets and makes the evaluated models publicly available.
  • v4 converts the paper to ICLR-2015 submission format and adds multiple-crop classification experiments.
  • v6 adds a comparison of net B with a shallow network and results on the PASCAL VOC action-classification benchmark.
Loading 1409.1556v6…