Source-linked AI summary
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell
TL;DR
With limited data, fully supervised deep architectures can overfit, motivating features that transfer to new visual tasks. The paper evaluates frozen activations from an ImageNet-trained convolutional network as DeCAF features and finds they outperform conventional representations across object, adaptation, fine-grained, and scene-recognition benchmarks.
Problem
Conventional visual representations have likely plateaued, while limited-data recognition tasks make fully supervised deep architectures prone to overfitting.
Method
DeCAF uses frozen hidden-layer activations from a convolutional network trained on labeled object-recognition tasks as transferable features for new tasks.
Results
DeCAF outperforms conventional visual representations across Caltech-101, Office domain adaptation, Caltech-UCSD Birds, and SUN-397 benchmarks.
Takeaways & Limitations
Deep convolutional features provide general semantic representations that support strong performance with simple linear classifiers across diverse visual recognition tasks.
Takeaways & Limitations
The experiments do not fine-tune the convolutional network layers on the evaluated tasks.
Abstract
from arXiv · showhide
We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be re-purposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.
1. Introduction
The paper addresses the limits of conventional representations and overfitting in data-scarce visual recognition by reusing supervised deep convolutional representations for new tasks. It introduces DeCAF and reports that ImageNet-trained features outperform conventional representations across several recognition benchmarks.
- Motivation: Conventional flat visual representations have likely plateaued despite strong performance, motivating the search for representations that capture task-relevant semantics.
- Problem: Fully supervised deep architectures can dramatically overfit when recognition tasks provide limited training data, including fine-grained recognition, attributes, and domain adaptation.
- Approach: The paper learns deep convolutional representations on related problems and applies them to new tasks with too few examples to learn a full deep representation.
- Approach: DeCAF is a visual feature defined by convolutional network weights learned on predefined object recognition tasks, serving as a supervised-pretraining-based transfer representation.
- Main result: ImageNet-trained convolutional features outperform conventional visual representations on Caltech-101, Office domain adaptation, Caltech-UCSD Birds, and SUN-397 benchmarks.
- Analysis: Convolutional features appear to cluster semantic topics more readily than conventional features, which the paper examines through visualizations of semantic clustering properties.
2. Related work
Deep convolutional networks progressed from supervised digit recognition to competition-winning performance on million-image benchmarks. Related-task and transfer-learning research motivates representations that preserve object-category information while reducing irrelevant variation, but prior deep transfer successes were limited on larger datasets.
- Deep convolutional networks: Deep convolutional networks evolved from successful supervised digit recognition to competition-winning results on benchmarks containing more than one million images.The passage cites LeCun et al. (1989) and Krizhevsky et al. (2012), with ImageNet as an example.
- Learning from related tasks: Learning from related tasks has been studied through multitask learning, efficient representation optimization, parameter-manifold transfer, and sets of classifiers trained on related tasks.The cited work spans Caruana, Thrun, Argyriou et al., Ando & Zhang, and related computer-vision approaches.
- Representation goals: A central representation-learning challenge is retaining object-category information while discarding nuisance variation such as illumination.This passage frames the desired feature representation in terms of category-relevant information versus irrelevant noise.
- Transfer learning: Prior deep transfer-learning studies, especially in unsupervised settings, reported success mainly on small datasets such as CIFAR and MNIST, with only modest success on larger datasets.The passage contrasts relatively small-dataset results with larger-dataset efforts.
- Evaluation setting: The paper evaluates whether deep convolutional features trained on generic recognition tasks generalize across ImageNet-biased domains, including SUN-397 scenes and domain-adaptation datasets.These evaluations test whether learned features can undo domain bias by capturing real semantic information.
3. Deep Convolutional Activation Features
The section presents activations from a supervised deep convolutional network as features for generic vision tasks, examining their cross-dataset generalization and behavior across network depth. It evaluates these properties through semantic visualizations and comparisons with existing baselines.
- Approach and evaluation: A CNN is trained fully supervised, after which features from different network levels are extracted and evaluated on generic vision tasks.The section explicitly asks whether CNN features generalize to other datasets and how performance varies with depth.
- Semantic visualization: t-SNE visualizations compare GIST, LLC, DeCAF1, and DeCAF6 features on ILSVRC-2012 validation data using semantic label groupings.DeCAF1 is taken from the first pooling layer, while DeCAF6 is taken from the second-to-last hidden layer.
- Implementation and release: The publicly available decaf framework and released network parameters enable out-of-the-box feature extraction without retraining the large network.The implementation processes about 40 images per second on an 8core commodity machine when executing the CNN model.
- Network architecture: The adopted architecture propagates 224 × 224 images through five convolutional and three fully connected layers to classify 1000 object categories.The single trained instance achieves a 42.9% error rate on ILSVRC-2012 validation, versus 40.7% for Krizhevsky et al. (2012).
- Cross-dataset generalization: DeCAF6 features trained on ILSVRC-2012 show strong semantic clustering on SUN-397, including separation of indoor and outdoor classes.This cross-dataset clustering is presented as evidence that DeCAF can support generic object recognition when target classes are absent from the original object pool.
4. Experiments
Experiments show that fixed DeCAF activations transfer across basic object recognition, domain adaptation, fine-grained recognition, and other visual recognition tasks without CNN fine-tuning. Later-layer features, especially DeCAF6 and DeCAF7, consistently provide strong off-the-shelf representations.
- Experimental setup: Experiments evaluate activations from hidden layers as fixed DeCAF features across object recognition, domain adaptation, fine-grained recognition, and scene recognition benchmarks.DeCAF5 is the first fully convolutionally propagated feature, while DeCAF7 is taken from the final hidden layer.
- Basic object recognition: 86.9% test accuracy is achieved by a linear SVM on DeCAF6 with dropout for Caltech-101 recognition.DeCAF5 performs substantially worse, DeCAF7 is generally 1–2% lower than DeCAF6, and dropout improves each combination by 0–2%.
- Basic object recognition: 2.6% improvement over Yang et al. is obtained by a linear SVM using a single DeCAF feature on Caltech-101.The same method outperforms Jarrett et al.’s two-layer convolutional network by over 20%.
- Basic object recognition: 33.0% one-shot accuracy for SVM indicates that DeCAF can support useful category models from a single positive example.This result uses fixed parameters and the same metric as the Caltech-101 evaluation.
- Domain adaptation: DeCAF qualitatively reduces domain bias between Webcam and Dslr, and its domain-adaptation performance is evaluated across Amazon→Webcam and Dslr→Webcam shifts.The quantitative comparison averages multi-class accuracy over five train/test splits using SURF, DeCAF6, and DeCAF7 with SVM and logistic regression.
- Fine-grained and other recognition tasks: 40.94% test performance is obtained by DeCAF7 with SVM, improving 2.9% over Xiao et al.’s state-of-the-art method using only DeCAF.For fine-grained recognition, DeCAF with logistic regression already significantly improves existing approaches, while structured part information improves performance further.
5. Discussion
The discussion concludes that deep features learned from a large labeled object database provide strong representational power and generalization for semantic visual discrimination. These features support simple linear classifiers, outperform sophisticated hand-engineered approaches, and reveal semantic organization beyond the network’s explicit training tasks.
- An auxiliary large labeled object database enables deep convolutional features with sufficient representational power and generalization for semantic visual discrimination.The features are analyzed within a semi-supervised multi-task framework.
- Deep features with simple linear classifiers reliably outperform state-of-the-art approaches using sophisticated multi-kernel learning and traditional hand-engineered features.
- The features cluster images into interesting semantic categories despite those categories never being explicit network training targets.The visual results demonstrate the generality and implicit semantic knowledge of the learned features.