PaperlayerView exact revision on arXiv

Source-linked AI summary

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Ken Chatfield, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman

arXiv:1405.3531v4cs.CV

TL;DR

It remains unclear how CNN architectures compare with one another and with shallow representations under common evaluation conditions. The paper rigorously evaluates these alternatives and finds that augmentation improves shallow methods, while deep architectures still outperform them by a large margin.

  • Problem

    It remains unclear how CNN architectures compare with one another and with shallow methods because systematic common-ground evaluation of design choices is largely missing.

  • Method

    The paper rigorously evaluates shallow, externally pretrained deep, and fine-tuned deep image representations across common experimental scenarios.

  • Results

    Deep architectures still outperform shallow methods by a large margin, even after data augmentation significantly improves shallow representations.

  • Takeaways & Limitations

    Data augmentation is useful for both shallow and deep representations, while fine-tuning can further improve already strong deep-representation results.

Abstract

from arXiv · show

The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in challenging benchmarks on image recognition and object detection, significantly raising the interest of the community in these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on a common ground, identifying and disclosing important implementation details. We identify several useful properties of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced significantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source code and models to reproduce the experiments in the paper is made publicly available.

1 INTRODUCTION

The paper rigorously evaluates modern CNN representations against different deep architectures and shallow encodings such as BoVW and IFV on common grounds. It also investigates implementation choices and identifies properties and techniques shared across deep and shallow methods.

  • Motivation: Visual feature design is a central determinant of progress in image classification and object recognition, from BoVW to IFV.The introduction attributes many quantitative improvements over the past dozen years to improved image representations.
  • Motivation: CNNs achieve excellent performance on ImageNet ILSVRC and PASCAL VOC, substantially exceeding standard image encodings.These networks contain many data-learned parameters despite their handcrafted structure.
  • Research gap: Comparisons among deep architectures and against shallow methods such as IFV remain unclear because extensive common-ground testing is largely missing.The paper frames systematic evaluation of design and implementation choices as an unresolved need.
  • Study scope: The study conducts rigorous experiments analyzing image representations as encoding functions φ that map an image I to a vector φ(I) ∈ R^d for linear classification.It examines shallow and deep representation scenarios.
  • Key findings: The evaluation finds that CNN output dimensionality can be reduced substantially without harming performance and that data augmentation can also improve shallow methods.The paper further identifies aspects of deep and shallow approaches that can be shared.

2 SCENARIOS

The section defines shallow IFV and deep CNN image representations across three scenarios, then describes shared augmentation, normalization, classification, and benchmark protocols. It evaluates representations on VOC2007, ImageNet, Caltech-101, and Caltech-256 using task-specific datasets and metrics.

  • Common methodology: All representations feed linear SVM predictors trained with hinge loss, quadratic regularization, and a validation-selected trade-off parameter C.The same learning framework is used across the three scenarios.
  • Benchmarks: The evaluation uses VOC2007, ImageNet, Caltech-101, and Caltech-256 with mean Average Precision, top-5 classification error, and dataset-specific train/test splits.VOC2007 contains about 10,000 images and twenty object classes; ImageNet contains 1,000 categories with roughly 1.2M training images and 50,000 validation images.
  • Shallow representation: The paper uses Improved Fisher Vector as its reference shallow representation because it usually outperforms BoVW, LLC, and VLAD.IFV is constructed from dense multi-scale patches and local descriptors, then improved through signed square-rooting and unit ℓ2 normalization.
  • Deep representation: Deep representations use penultimate-layer activities from CNNs trained on large datasets such as ImageNet as transferable image descriptors.The paper notes that selecting the best CNN architecture remains an open question.
  • Scenarios: Fine-tuning pretrained CNNs on target data is treated separately because it can significantly improve performance and makes features dataset-specific.This scenario concerns features trained on one large dataset and applied to another, usually smaller, dataset.
  • Common methodology: Data augmentation applies to both shallow and deep representations by perturbing images with class-preserving transformations such as cropping and flipping.Augmentation may occur during training, testing, or both, and augmented samples may be combined into a single feature.

3 DETAILS

The methods use detailed Fisher Vector and CNN implementations, with normalization, spatial encoding, architecture choices, and dataset-specific training procedures designed to balance performance, efficiency, and fair comparison. The section also examines compact CNN representations and multiple data-augmentation strategies.

  • IFV implementation: The IFV pipeline upsamples images, extracts SIFT at multiple scales, reduces features from 128D to 80D, and encodes them with a 256-component GMM.The resulting Fisher Vector has dimension 40,960 and is signed-square-rooted and l2-normalised before classification.
  • IFV implementation: Intra-normalisation applies l2 normalisation to individual Fisher Vector descriptor blocks to reduce local-feature burstiness.For these improved features, applying square-rooting only once to the final encoding produced the best results.
  • IFV implementation: Spatially extended descriptors append normalized image coordinates before GMM quantization, capturing spatial information directly while reducing memory compared with a spatial pyramid.The augmented descriptors are 82-dimensional and produce a 41,984-dimensional IFV.
  • CNN implementation: Three CNN architectures are trained with the same protocol and implementation for fair comparison, and l2-normalising CNN features before SVM classification is important for performance.The architectures represent different accuracy/speed trade-offs.
  • CNN implementation: Baseline CNNs use a 4096-dimensional full7 layer, while CNN-M variants reduce this layer to 2048, 1024, or 128 dimensions.These networks were trained on ILSVRC-2012.
  • Data augmentation: CNN experiments compare no augmentation, horizontal flipping, and combined cropping-plus-flipping, with images resized and cropped to 224 × 224.Without augmentation, the smallest image dimension is resized to 224 pixels and a centre crop is extracted.

4 ANALYSIS · 4.1 Data augmentation

The analysis evaluates feature representations and augmentation schemes, finding that augmentation improves both IFV and CNN performance by about 3%. For CNNs, additional training samples with sum-pooled testing perform best, while C+F sampling adds about 2–3% and denser crop sampling offers no benefit.

  • 4 ANALYSIS: The experiments compare different features and data augmentation schemes on VOC-2007 before moving to other datasets and state-of-the-art comparisons.
  • 4.1 Data augmentation: Augmentation consistently improves IFV and CNN performance by ∼3%.The passage gives IFV [d] versus [f] and CNN [o] versus [p] as examples.
  • 4.1 Data augmentation: Using additional training samples with sum-pooling at test time performs best, followed by sum-pooling, max pooling, and stacking.
  • 4.1 Data augmentation: Extracting a 224 × 224 centre crop from a 256 × 256 image results in worse performance.
  • 4.1 Data augmentation: C+F sampling improves performance by about 2 ∼3%, whereas flipping alone improves only marginally.
  • 4.1 Data augmentation: Sampling more transformations or taking a higher density of centre crops provides no benefit.

4.2 Colour · 4.3 Scenario 1: Shallow representation (IFV)

The paper finds that colour augmentation benefits IFV modestly in non-augmented settings but has little effect after augmentation, while several IFV implementation choices improve performance. In particular, intra-normalisation yields an approximately 1% gain, and the baseline surpasses prior reported results due to richer SIFT processing.

  • 4.2 Colour: Combining SIFT and colour descriptors in IFV improves non-augmented performance by around ∼1%.The combined representation stacks the corresponding IFVs, denoted COL+.
  • 4.2 Colour: Colour descriptors alone perform significantly worse than the SIFT-based IFV representation.Replacing SIFT with the colour descriptors is denoted COL in the method.
  • 4.2 Colour: The combined SIFT-and-colour representation has little impact when data augmentation is used.The passage contrasts the augmented comparison with the approximately 1% non-augmented improvement.
  • 4.3 Scenario 1: Shallow representation (IFV): The baseline IFV with a spatial pyramid slightly outperforms results reported by Chatfield et al.The difference is attributed primarily to using more spatial scales for SIFT extraction and square-rooting the resulting SIFT features.
  • 4.3 Scenario 1: Shallow representation (IFV): Intra-normalisation improves IFV performance by approximately ∼1%.Intra-normalisation is denoted as IN in the method column.
  • 4.3 Scenario 1: Shallow representation (IFV): The IFV evaluation compares spatial pooling with alternative spatial-pyramid implementation choices.Spatial pooling is denoted spm in the SPool column; the supplied passage introduces a switch away from it but truncates the resulting finding.

4.4 Scenario 2: Deep representation (CNN) with pre-training

Pre-trained CNN representations consistently outperform shallow encodings, with architecture depth and speed affecting performance. Their output dimensionality can be reduced substantially with little loss, while l2-normalisation provides an additional boost.

  • Scenario 2: Deep representation (CNN) with pre-training: CNN-based methods outperform shallow encodings by a large ∼10% mAP margin, even after the discussed improvements.
  • Scenario 2: Deep representation (CNN) with pre-training: Medium CNN-M and slow CNN-S outperform fast CNN-F by a significant 2 ∼3% margin, while CNN-M and CNN-S have nearly the same accuracy.
  • Scenario 2: Deep representation (CNN) with pre-training: Reducing the CNN output from 4096 to 2048 produces a marginal performance boost.
  • Scenario 2: Deep representation (CNN) with pre-training: CNN outputs can be reduced to 1024D and 128D with only a ∼2% drop, yielding codes 32× smaller and ∼650× smaller than the best-performing IFV.
  • Scenario 2: Deep representation (CNN) with pre-training: ℓ2-normalising the features accounts for up to ∼5% of performance.

4.5 Scenario 3: Deep representation (CNN) with pre-training and fine-tuning · 4.6 Combinations

Fine-tuning the CNN-S architecture on VOC2007 improved performance by 2.7%, showing that learned deep representations can adapt despite limited training data. Combining deep and shallow representations had little effect for CNN-M 2048 but modestly boosted the weaker CNN-F by approximately 0.8%.

  • 4.5 Scenario 3: Deep representation (CNN) with pre-training and fine-tuning: 2.7% improvement followed fine-tuning CNN-S on VOC2007 with the ranking hinge loss.The comparison is [γ] vs. [n].
  • 4.5 Scenario 3: Deep representation (CNN) with pre-training and fine-tuning: Fine-tuning adjusted the learned deep representation to better suit VOC2007.This result held despite VOC2007 providing only 5,011 training images.
  • 4.5 Scenario 3: Deep representation (CNN) with pre-training and fine-tuning: VOC2007 fine-tuning used only 5,011 training images.The paper presents this as a small amount of training data for adapting the representation.
  • 4.6 Combinations: For CNN-M 2048, stacking deep and shallow representations made little difference.The comparison is [x] vs. [β].
  • 4.6 Combinations: For the weaker CNN-F, stacking deep and shallow representations produced a small boost of ∼0.8%.The comparison is [m] vs. [α].
  • 4.6 Combinations: The effect of combining representations depended on the CNN representation used.CNN-M 2048 showed little difference, whereas CNN-F showed a small improvement.

4.7 Comparison with the state of the art

The CNN-S model is competitive with recent CNN approaches and establishes state-of-the-art performance on VOC-2007 and VOC-2012 among methods pretrained only on ILSVRC-2012. Fine-tuning improves performance on VOC, while data augmentation also substantially improves shallow IFV.

  • CNN comparison: CNN-F, CNN-M, and CNN-S achieve lower ILSVRC error rates than related configurations reported in,, and.The authors attribute the difference partly to sampling image crops from the uncropped image plane rather than only the center.
  • CNN comparison: Across other datasets, CNN-F performs worst, whereas CNN-M and CNN-S perform considerably better.This relative ordering generally follows the pattern observed on ILSVRC, where the networks are trained.
  • Fine-tuning: Fine-tuning CNN-S benefits VOC datasets, with ranking loss marginally outperforming classification loss on VOC-2012.Fine-tuning on Caltech-101 produces a small improvement, but no gain is observed over Caltech-256.
  • State-of-the-art comparison: CNN-S is competitive with recent CNN approaches and sets the state of the art on VOC-2007 and VOC-2012 among methods pretrained solely on ILSVRC-2012.Methods achieving 86.3% and 90.3% on VOC used extended ILSVRC datasets, while the 90.3% VOC-2012 result also used late fusion with a complex hand-crafted method; without fusion, it reached 84.2%.
  • Shallow-method comparison: Data augmentation and related modifications improve shallow IFV performance to 68.02%.This achieves performance comparable to the state of the art while retaining a simple approach based on shallow features.

4.8 Performance Evolution on VOC-2007

Performance on PASCAL VOC-2007 increased substantially across successive shallow and CNN-based methods, culminating in 82.42% for CNN-S with fine-tuning. This result was comparable to the most recent state-of-the-art.

  • Performance Evolution: 54.48% mAP was reported for the BoVW model in 2008, while IFV reached 61.7% in 2010.The comparison tracks classification accuracy over PASCAL VOC across the reported years.
  • Performance Evolution: 82.42% was achieved by the best-performing CNN-based method, CNN-S with fine-tuning, comparable to the most recent state-of-the-art.The result is reported for PASCAL VOC-2007.
  • Performance Evolution: 73.41% was obtained by DeCAF and similar CNN-based methods introduced in late 2013.These methods followed earlier shallow representations in the performance evolution.

4.9 Timings and dimensionality

CNN-M-2048 provides a substantially more compact representation than IFV, while CNN-M feature computation is faster with augmentation and remains competitive without it.

  • 4.9 Timings and dimensionality: CNN-M-2048 is ∼42× more compact than the best-performing IFV, at 2K versus 84K dimensions.The comparison is between CNN-M-2048 and IFV representations.
  • 4.9 Timings and dimensionality: ∼50× faster computation is reported for augmented CNN-M features than IFV, at ∼2.4s versus ∼120s per image on one CPU core.With augmentation disabled, CNN-M takes around 0.3s per image, compared with ∼0.4s for CNN-S and ∼0.13s for CNN-F.

5 CONCLUSION

The paper rigorously evaluates CNN and shallow representations, finding that data augmentation substantially improves shallow methods but deep architectures still outperform them by a large margin. Deep performance on ILSVRC predicts performance on other datasets, while fine-tuning further improves strong deep-representation-plus-linear-SVM results.

  • 5 CONCLUSION: Data augmentation significantly improves shallow representations, but deep architectures still outperform shallow methods by a large margin.The augmentation is commonly used in deep learning and yields a substantial improvement for shallow methods.
  • 5 CONCLUSION: Performance on ILSVRC is a good indicator of how deep representations perform on other datasets.
  • 5 CONCLUSION: Fine-tuning further improves already strong results from combining deep representations with a linear SVM.
  • 5 CONCLUSION: Source code and CNN models are available to reproduce the paper’s experiments.
Loading 1405.3531v4…