PaperlayerView exact revision on arXiv

Source-linked AI summary

What makes ImageNet good for transfer learning?

Minyoung Huh, Pulkit Agrawal, Alexei A. Efros

arXiv:1608.08614v2cs.CVcs.AIcs.LG

TL;DR

The paper asks which properties of ImageNet make its learned CNN features transferable, given competing beliefs about dataset size, class count, and fine-grained labels. It varies ImageNet subsets and evaluates fine-tuned features on PASCAL detection, action classification, and SUN scene classification. The results show that large reductions in pre-training classes or images usually have only modest effects on transfer performance, although the definitive explanation remains unresolved.

  • Problem

    The paper addresses the lack of empirical evidence about whether ImageNet’s transferability comes from its size, number of classes, or fine-grained visual distinctions.

  • Method

    The authors pre-train CNN features on varied ImageNet subsets and evaluate them after fine-tuning on PASCAL detection, action classification, and SUN scene classification.

  • Results

    Large reductions in ImageNet pre-training images or classes have only modest effects on transfer-task performance.

  • Takeaways & Limitations

    Common assumptions that full ImageNet size, all classes, or fine-grained recognition are essential for transferable features are not supported by these experiments.

  • Takeaways & Limitations

    The experiments use only AlexNet, so whether the findings hold for deeper architectures remains open.

Abstract

from arXiv · show

The tremendous success of ImageNet-trained deep features on a wide range of transfer tasks begs the question: what are the properties of the ImageNet dataset that are critical for learning good, general-purpose features? This work provides an empirical investigation of various facets of this question: Is more pre-training data always better? How does feature quality depend on the number of training examples per class? Does adding more object classes improve performance? For the same data budget, how should the data be split into classes? Is fine-grained recognition necessary for learning good features? Given the same number of training classes, is it better to have coarse classes or fine-grained classes? Which is better: more classes or more examples per class? To answer these and related questions, we pre-trained CNN features on various subsets of the ImageNet dataset and evaluated transfer performance on PASCAL detection, PASCAL action classification, and SUN scene classification tasks. Our overall findings suggest that most changes in the choice of pre-training data long thought to be critical do not significantly affect transfer performance.? Given the same number of training classes, is it better to have coarse classes or fine-grained classes? Which is better: more classes or more examples per class?

1. Introduction

The paper investigates which ImageNet properties produce general-purpose transfer features, challenging assumptions about dataset size, class count, fine-grained recognition, and data allocation. Across transfer experiments, substantial reductions in images or classes generally cause only modest performance changes.

  • Motivation: ImageNet’s success motivates testing whether dataset size, class diversity, or fine-grained recognition is responsible for general-purpose features.The paper evaluates these competing explanations empirically rather than relying on prevailing assumptions.
  • Approach: Features are evaluated after fine-tuning on PASCAL detection, PASCAL action classification, and SUN scene classification.These three transfer tasks provide the paper’s evaluation framework.
  • Pre-training data: 500 images per class cause only a 1.5 mAP drop on PASCAL-DET compared with 1000 images per class.The transfer-performance decline is much smaller than the corresponding decline on ImageNet classification.
  • Class count: 127 pre-training classes cause only a 2.8 mAP drop on PASCAL-DET compared with 1000 classes.Some transfer tasks perform better with fewer classes.
  • Class granularity: Coarse-class pre-training still produces good transfer features, indicating that fine-grained recognition is not required.The paper also reports that features trained on coarse classes can distinguish unseen fine-grained classes, and vice versa.
  • Data allocation: With the same pre-training-image budget, fewer classes with more images per class perform slightly better than more classes with fewer images per class.This finding directly addresses how to allocate a fixed data budget across classes.
  • Additional data: 771 ImageNet classes excluding all PASCAL VOC classes achieve nearly the same PASCAL-DET performance as complete ImageNet, while blindly adding data can sometimes hurt.The result challenges the assumption that all ImageNet data is necessary or uniformly beneficial.

2. Related Work

Prior work established ImageNet pre-training as the dominant route to transferable CNN features, while alternative supervised, weakly supervised, and unsupervised approaches had not surpassed it. This paper studies why ImageNet remains effective.

  • Prior transfer-learning studies: Earlier studies examined pre-training duration, transfer-layer selection, architecture, and fine-tuning without resolving ImageNet’s broader advantage.The related work situates this paper as an investigation of the dataset rather than only optimization or architecture.
  • Unsupervised alternatives: Unsupervised feature-learning criteria such as reconstruction and feature slowness had not produced features competitive with supervised ImageNet pre-training.The limitation concerns transfer-task performance relative to ImageNet-supervised representations.
  • Weak supervision: YFCC100M’s nearly two orders of magnitude larger image collection did not provide a substantial boost over ImageNet pre-training when using noisy user tags.The comparison motivates examining properties beyond raw dataset scale.
  • Open question: Alternative methods for learning general-purpose deep features had not outperformed ImageNet-supervised pre-training on transfer tasks.The paper’s stated goal is to understand ImageNet’s continuing success.

3. Experimental Setup

The experiments use AlexNet features initialized through ImageNet classification and adapted to transfer tasks under a standardized evaluation setup. ImageNet subsets vary in images per class and label taxonomy while keeping the relevant comparison controlled.

  • Model and training: All experiments use AlexNet, with ImageNet classification providing supervised CNN initialization before target-task adaptation.The paper calls this initialization pre-training and the subsequent target-task adaptation fine-tuning.
  • Transfer tasks: Transfer evaluation covers PASCAL VOC 2007 object detection, PASCAL VOC 2012 action recognition, and SUN scene classification.The setup closely follows an established experimental protocol.
  • PASCAL-DET: PASCAL-DET fine-tuning adapts AlexNet convolutional layers and reports test-set performance after training on VOC 2007 train/val.The implementation uses Faster-RCNN code and a 70K-iteration SGD schedule.
  • PASCAL-ACT-CLS: PASCAL-ACT-CLS uses VOC 2012 train/val and follows the PASCAL-DET fine-tuning procedure.The implementation uses the R*CNN experimental setup and code.
  • SUN-CLS: SUN-CLS replaces AlexNet’s FC-8 layer with a randomly initialized 397-unit layer before fine-tuning.The SUN setup uses the specified train/val/test splits and SGD optimization.
  • Run variance: PASCAL transfer results average three runs because Faster-RCNN and R*CNN vary across training runs, whereas SUN-CLS uses one run.The paper reports mean ± standard deviation for the variable PASCAL procedures.
  • ImageNet subsets: Reduced-image pre-training models sample N images randomly from each ImageNet class when N < 1000.The 1000-images-per-class model uses the original ILSVRC 2012 training set.

4. How does the amount of pre-training data affect transfer performance?

Using half the ImageNet training images per class causes only a small transfer-performance drop, despite continued gains on the ImageNet classification task itself.

  • The study trained five AlexNet models from scratch using 50, 125, 250, 500, or 1000 images per ImageNet class.These models were fine-tuned for PASCAL-DET, PASCAL-ACT-CLS, and SUN-CLS.
  • 58.3, 57.0 and 54.6 mAP were obtained on PASCAL-DET with 1000, 500 and 250 images per class, respectively.The corresponding transfer tasks showed a similar trend.
  • 500 images per class produce only a 1.5 mAP drop on PASCAL-DET relative to 1000 images per class.The transfer-performance decline is much smaller than the effect of reducing data on ImageNet classification.
  • Additional pre-training images steadily improve ImageNet classification, but produce significantly slower gains on transfer tasks.The results indicate diminishing returns from adding examples to existing ImageNet classes for transfer performance.

5. How does the taxonomy of the pre-training task affect transfer performance?

The experiments vary ImageNet’s class taxonomy while preserving or reallocating data, testing whether class count, granularity, and class composition determine transferable feature quality. Across tasks, coarse or reduced class sets often retain near-full transfer performance, while visually coherent subclasses support broader representations.

  • 5.1. The effect of number of pre-training classes on transfer performance: WordNet-derived taxonomies preserve the total image budget while varying the label sets from 918, 753, 486, 79, 9, 127, 10, or 2 classes.The authors trained separate AlexNet CNNs from scratch on these alternative label sets.
  • 5.1. The effect of number of pre-training classes on transfer performance: Using 486 classes changes transfer performance by -1.7 mAP on PASCAL-DET, -0.8% accuracy on SUN-CLS, and +0.6 mAP on PASCAL-ACT-CLS.These changes are measured relative to the 1000-class setting.
  • 5.1. The effect of number of pre-training classes on transfer performance: Beyond 127 pre-training classes, transfer performance shows only diminishing returns, including on SUN-CLS, which requires discrimination among 397 classes.For PASCAL-ACT-CLS and SUN-CLS, some smaller class sets outperform all 1000 classes.
  • 5.2. Is fine-grain recognition necessary for learning transferable features?: Coarse 127-class pre-training causes only a small transfer-performance drop, indicating that fine-grained recognition is marginally helpful rather than critical for transferable features.The 127 classes represent coarse semantic concepts, although 65 are also present among the original 1000 classes.
  • 5.3. Does training with coarse classes induce features relevant for fine-grained recognition?: 127-class coarse training yields fine-grained nearest-neighbor performance only about 15% below direct fine-class training on unseen ImageNet classes.The induction-accuracy evaluation uses top-1 and top-5 nearest neighbors in FC7.
  • 5.4. Does training with fine-grained classes induce features relevant for coarse recognition?: Fine-grained subclasses support coarse recognition most when they share visual structure; visually dissimilar subclasses, such as home appliances, show less of this effect.Mammal, fruit, and bird are examples of visually consistent coarse classes.
  • 5.5. More Classes or More Examples Per Class?: With a fixed pre-training-image budget, fewer classes with more images per class perform slightly better on transfer tasks than more classes with fewer images per class.Table 2 compares 500, 250, or 125 classes with 1000 images each against 1000 classes with 500, 250, or 125 images each.
  • 5.6. PASCAL overlap: PASCAL-DET reaches 57.8 mAP after pre-training on PASCAL-removed-ImageNet, compared with 58.3 mAP using full ImageNet.This indicates that ImageNet classes absent from PASCAL can still produce features effective for PASCAL classes.

6. Does data augmentation from non-target classes always improve performance?

Adding unrelated pre-training classes does not consistently improve transfer performance: random splits benefit from added data, whereas disjoint minimal splits can perform better alone under last-layer fine-tuning.

  • Random versus minimal splits: Randomly adding classes improves transfer performance because semantically similar classes, such as dog breeds, can occur across splits.The experiment compared a CNN trained on split A with one trained on splits A and B, then fine-tuned on split A.
  • Random versus minimal splits: Minimal splits deliberately separate classes with no common WordNet ancestor through depth 4, creating sufficiently disjoint object groups.Minimal split A contains 522 classes and split B contains 478 classes.
  • Random versus minimal splits: Both minimal split A and B outperform the full dataset when only the last layer is fine-tuned.Thus, adding unrelated classes did not match the performance of networks trained on a single minimal split.
  • Training regime: With extensive full-network training, minimal split A benefits from split B, but minimal split B does not.The authors speculate that images in split B may be contained in images from split A, but not vice versa.
  • Interpretation: The results suggest that unrelated training data may push optimization toward a local minimum that is difficult to escape through fine-tuning.The authors note that clever learning-rate adjustments might recover performance, but this possibility was not established.

7. Discussion

The discussion argues that ImageNet transfer features remain strong despite substantial reductions in pre-training data or classes, challenging common assumptions about why ImageNet works. However, the conclusions are bounded by architecture and task choices, and the paper leaves the underlying resilience unexplained.

  • Contribution: The study analyzes which ImageNet training-data factors affect transfer features rather than comparing alternative neural-network architectures.Its goal is to establish facts about the importance of training-data aspects for feature learning.
  • Challenge to folk wisdom: The prevailing view attributes generalizable features to large amounts of training data and many classes, but the paper quantitatively questions whether full ImageNet is necessary.The discussion contrasts the field’s consensus with the paper’s empirical findings.
  • Open question: The cause of this resilience remains unexplained, with the authors offering only speculative possibilities for future study.The paper does not establish why reduced data or class counts preserve transfer performance.
  • Limitations: Because experiments used only AlexNet, it remains unknown whether the findings hold for deeper architectures such as VGG, ResNet, and GoogLeNet.The authors suggest that different outcomes could indicate AlexNet has less capacity than previously thought.
  • Limitations: PASCAL and SUN may be too similar to ImageNet to adequately test feature generalization, motivating evaluation with less or no fine-tuning.The authors mention one-shot learning and nearest-neighbor evaluation as possible alternatives.
  • Conclusion: The paper concludes that its results undermine much of the folk wisdom surrounding ImageNet’s transfer-learning success, without providing a definitive explanation.The authors hope this motivates further research.
Loading 1608.08614v2…