PaperlayerView exact revision on arXiv

Source-linked AI summary

CNN Features off-the-shelf: an Astounding Baseline for Recognition

Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, Stefan Carlsson

arXiv:1403.6382v3cs.CV

TL;DR

It was unclear whether CNN features trained for one task could generalize across diverse visual-recognition problems without task-specific retraining. The paper evaluates off-the-shelf OverFeat features with simple classifiers and finds strong performance across tasks and datasets, often outperforming sophisticated state-of-the-art systems.

  • Problem

    It was unclear whether CNN features trained for one task could serve as generic representations across diverse recognition problems without task-specific training.

  • Method

    The paper uses off-the-shelf OverFeat features with simple classifiers across object, scene, fine-grained, attribute, and retrieval tasks.

  • Results

    Across recognition tasks and datasets, the CNN representation consistently outperformed or matched sophisticated baselines, except against low-memory retrieval methods on the sculptures dataset.

  • Takeaways & Limitations

    Off-the-shelf CNN representations are effective and general across varied visual-recognition tasks and datasets.

  • Takeaways & Limitations

    The retrieval comparison reports only low-memory-footprint methods.

Abstract

from arXiv · show

Recent results indicate that the generic descriptors extracted from the convolutional neural networks are very powerful. This paper adds to the mounting evidence that this is indeed the case. We report on a series of experiments conducted for different recognition tasks using the publicly available code and model of the \overfeat network which was trained to perform object classification on ILSVRC13. We use features extracted from the \overfeat network as a generic image representation to tackle the diverse range of recognition tasks of object image classification, scene recognition, fine grained recognition, attribute detection and image retrieval applied to a diverse set of datasets. We selected these tasks and datasets as they gradually move further away from the original task and data the \overfeat network was trained to solve. Astonishingly, we report consistent superior results compared to the highly tuned state-of-the-art systems in all the visual classification tasks on various datasets. For instance retrieval it consistently outperforms low memory footprint methods except for sculptures dataset. The results are achieved using a linear SVM classifier (or $L2$ distance in case of retrieval) applied to a feature representation of size 4096 extracted from a layer in the net. The representations are further modified using simple augmentation techniques e.g. jittering. The results strongly suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks.

1. Introduction

The introduction asks whether publicly available OverFeat features, trained for ImageNet classification, can serve as generic representations across vision tasks. It motivates a simple feature-plus-linear-classifier approach and reports convincing performance against descriptor methods without 3D geometry constraints.

  • Results: Augmented CNN representations with a linear SVM consistently outperform state-of-the-art methods across multiple tasks.The comparison distinguishes these generic representations from specialized CNNs designed for particular tasks.
  • Motivation: Publicly available OverFeat enabled experiments on whether features from an ImageNet-trained classification network could transfer across diverse vision tasks.The paper shifts focus from training task-specific networks to exploiting features from an existing network.
  • Method: The proposed baseline extracts final-layer OverFeat responses and combines them with a simple linear classifier.This approach is presented as the simplest test of CNN feature effectiveness.
  • Results: CNN features achieve very convincing descriptor performance against engineered vectors when systems imposing 3D geometric constraints are excluded.The comparison tests newer CNN descriptors in the traditional domain of hand-engineered descriptors.

2. Background and Outline

The study uses the large publicly available OverFeat CNN, whose generic features are trained on ImageNet while task-specific classifiers are trained separately for each dataset. The authors note that optimizing CNN features for individual tasks could further improve performance.

  • CNN features: The experiments use the publicly available large OverFeat CNN, following the architecture of Krizhevsky et al.Its convolutional layers contain 96 to 1024 kernels sized 3×3 to 7×7, with half-wave rectification and max pooling for robustness to intra-class deformations.
  • Training setup: OverFeat features are trained only on ImageNet, whereas the simple classifiers are trained on images from each task’s dataset.This separates generic representation learning from task-specific classification.
  • Training setup: Task- and dataset-specific optimization of the CNN features could further improve the system when sufficient computational resources are available.The current approach deliberately retains simple classifiers while using fixed generic CNN features.

3. Visual Classification

This section surveys different visual-classification tasks, which are discussed in the following subsections.

  • The section introduces a set of visual-classification tasks covered in subsequent subsections.

3.1. Method

The method uses 4096-dimensional OverFeat features from layer 22, extracted after resizing images to 221×221 and L2-normalizing the resulting vectors. Classification uses linear SVMs, with optional data augmentation and task-dependent multiclass strategies.

  • Feature extraction: 4096-dimensional feature vectors are extracted from OverFeat’s first fully connected layer, layer 22, after resizing each image or cropped sub-window to 221×221.Max-pooling and rectification count as separate layers in OverFeat’s numbering.
  • Classification: Feature vectors are L2-normalized to unit length and combined with a Support Vector Machine for classification tasks.This configuration is referred to as CNN-SVM.
  • Data augmentation: Training data are optionally augmented with cropped and rotated samples, followed by component-wise power transformation, producing the CN-Naug+SVM setting.The paper reports separate results for this augmented configuration.
  • Classification strategy: One-against-all SVMs handle non-mutually-exclusive labels, while the remaining experiments use one-against-one linear SVMs with voting.The classification models are linear SVMs defined using equation 1.

3.2. Image Classification

Off-the-shelf OverFeat CNN features are evaluated for object and indoor-scene classification on datasets whose distributions differ from ILSVRC. With linear SVMs, the representation outperforms prior systems and most baselines, while performance generally improves in deeper network layers.

  • Datasets: The CNN representation is best aligned with object classification because it was trained for ILSVRC object classification, although both target datasets differ in image distribution.This experiment is therefore closer to the network’s original task than the paper’s other experiments.
  • Datasets: The experiments use Pascal VOC 2007 object classification and MIT-67 indoor-scene recognition, without using VOC bounding-box annotations.VOC contains approximately 10,000 images across 20 classes, with objects generally not centered.
  • Object classification: OverFeat features outperform all compared prior efforts on Pascal VOC 2007 when evaluated using average precision.The comparison uses methods trained with data outside the standard Pascal VOC 2007 dataset.
  • Layer analysis: Performance generally increases with deeper representations, except for the final two layers, while ReLU layers cause subtle drops by half-rectifying signals.A linear SVM is trained using the output of each network layer.
  • Object classification: 77.7 is achieved by adapting the representation to the VOC dataset, indicating further task- or dataset-specific adaptation can improve performance.This result is reported as better than the untuned VOC result.
  • Scene recognition: On MIT-67, off-the-shelf CNN features with linear SVMs significantly outperform most baselines in mean classification accuracy.The CNN-SVM confusion matrix has a strong diagonal, though close-up scene views remain difficult to distinguish.

3.3. Object Detection

The paper reports no experiments of its own on object detection, but cites Girshick et al.’s results showing that off-the-shelf CNN features achieved 46.2 mAP on PASCAL VOC 2007 and outperformed the state of the art by about 10%.

  • Object Detection: 46.2 mAP on PASCAL VOC 2007 was achieved using off-the-shelf CNN features, outperforming the state of the art by about 10%.These results were reported by Girshick et al.; the paper did not conduct its own object-detection experiments.

3.4. Fine grained Recognition

This section evaluates generic CNN features for fine-grained recognition, where subtle differences between subordinate classes require detailed representations. Experiments use the CUB 200-2011 and 102 Flowers datasets, comparing CNN-SVM with strong task-specific baselines under differing annotation conditions.

  • Motivation: Fine-grained recognition distinguishes subordinate classes within the same object category, such as bird species, dog breeds, and flower types.The subtle differences between subordinate classes require a fine-detailed representation.
  • Datasets: The evaluation uses CUB 200-2011 and 102 Flowers, including 11,788 images of 200 bird subordinates in CUB.CUB provides 5,994 training images and 5,794 evaluation images, with species differences sometimes difficult for humans to distinguish.
  • CUB 200-2011: On CUB 200-2011, CNN-SVM is compared with top-performing baselines that use bounding-box annotations or part annotations.The first two table entries use only bounding-box annotations, while the remaining baselines use part annotations during training and sometimes evaluation.
  • 102 Flowers: On 102 Flowers, CNN-SVM is compared with baselines that use flower segmentation from the background.All methods other than CNN-SVM use flower segmentation.

3.5. Attribute Detection

Attribute detection is evaluated on UIUC 64 object attributes and H3D human attributes, covering semantic properties such as shape, parts, materials, pose, gender, and glasses. CNN features perform favorably against state-of-the-art methods, matching DPD without requiring part-level annotations, while task-specific CNN adaptation reaches 78.98 mAP.

  • 3.5. Attribute Detection: The experiments use UIUC 64 object attributes and H3D human attributes, with attributes spanning shape, parts, materials, glasses, and gender.UIUC contains three attribute categories—shape, part, and material—whereas H3D defines nine attributes for a subset of Pascal VOC 2007 person images.
  • 3.5. Attribute Detection: On H3D, CNN features perform as well as DPD despite using only one feature from the person bounding box rather than part-level training annotations.Both poselets and DPD use part-level annotations during training, while the CNN representation is extracted from the bounding box around the person.
  • 3.5. Attribute Detection: CNN features perform very favorably on the UIUC 64 object attribute dataset compared with existing methods.Table 5 reports results for attribute detection both across and within categories.
  • 3.5. Attribute Detection: 78.98 mAP was achieved by adapting the CNN architecture specifically for attribute detection.This result is attributed to adapting the architecture for different tasks given enough computational resources.

3.6. Implementation Details

The CNN-SVM experiments used precomputed linear kernels with libsvm, while CNNaug-SVM used liblinear’s primal solver. Data augmentation created 16 representations per sample using crops, rotations, and mirrors.

  • Classifiers and solvers: CNN-SVM used precomputed linear kernels with libsvm, whereas CNNaug-SVM used liblinear with the primal solver because #samples ≫ #dim.The solver choice differed between the two experiment types.
  • Data augmentation: 16 representations were generated for each sample: the original image, 5 crops, 2 rotations, and their mirrors.This augmentation combined spatial crops, rotations, and reflection variants.
  • Data augmentation: Each crop contained 4/9 of the original image area and was taken from the 4 corners or the center.The cropping procedure used five subwindow locations.
  • Implementation resources: The system’s extracted features, scripts, and updated tables were available on the project webpage.The passage provides the URL: http://www.csc.kth.se/cvap/cvg/DL/ots/.

4. Visual Instance Retrieval

The CNN representation is evaluated for visual instance retrieval on five datasets against low-memory state-of-the-art pipelines. Using an L2-normalized first fully connected-layer representation with spatial search or jittering, the study reports consistently high results under fair comparison conditions.

  • Comparison setup: CNN features are compared with VLAD, BoW, IFV, Hamming Embedding, and BoB using representations with comparable dimensionality and without spatial re-ranking or query expansion.Unlike CNN features, the comparison methods use dictionaries trained on datasets similar or identical to their evaluation data.
  • Datasets: The retrieval evaluation covers five common datasets, including Oxford5k and Paris6k building benchmarks.Oxford5k contains 5063 reference photos and 55 building queries; Paris6k contains 6412 reference images and 55 queries.
  • Representation: The representation is the L2-normalized output of the first fully connected layer.This representation is used as the basis for retrieval comparisons.
  • Spatial search: Spatial search extracts multi-scale patches at multiple locations and averages each query patch’s minimum L2 distance to corresponding reference patches.Features are extracted from the smallest square containing the region of interest rather than resizing the region.
  • Results: The study reports that CNN features produce consistently high retrieval results compared with low-memory-footprint methods.Only low-memory-footprint methods are included in the reported results.
  • Evaluation procedure: Spatial search is used for the first three datasets, while the remaining two use the jittering procedure described in Section 3.1.The spatial-search settings reported for Oxford5k, Paris6k, and Sculpture6k are h_r = 4 and h_q = 3.

5. Conclusion

Off-the-shelf OverFeat features paired with simple classifiers addressed diverse recognition tasks and datasets, competing strongly with sophisticated state-of-the-art methods. The consistent results highlight the effectiveness and generality of this CNN representation.

  • 5. Conclusion: Off-the-shelf OverFeat representations with simple classifiers addressed different recognition tasks.OverFeat was originally optimized for object classification on ILSVRC 2013.
  • 5. Conclusion: The CNN representation remained a strong competitor to sophisticated, highly tuned state-of-the-art methods.This result held despite the model being trained for object classification.
  • 5. Conclusion: Consistent results across recognition tasks and datasets highlight the representation’s effectiveness and generality.The conclusion reports the same trend across varied tasks and datasets.
Loading 1403.6382v3…