PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Better ImageNet Models Transfer Better?

Simon Kornblith, Jonathon Shlens, Quoc V. Le

arXiv:1805.08974v3cs.CVcs.LGstat.ML

TL;DR

The paper asks whether better ImageNet models necessarily transfer better, a relationship that had not been systematically tested across architectures. It compares 16 networks across 12 datasets and finds strong architecture-level transfer correlations, while regularization and small fine-grained tasks reveal limits of ImageNet features.

  • Problem

    The study addresses the untested hypothesis that architectures performing better on ImageNet also perform better on other vision tasks and learn more transferable features.

  • Method

    The authors compare 16 classification networks across 12 image classification datasets using fixed features, ImageNet fine-tuning, and random initialization.

  • Results

    Better ImageNet networks transfer better, with r = 0.99 for fixed features and r = 0.96 for fine-tuning, but regularizers can damage feature transfer and ImageNet pretraining offers minimal benefits on two small fine-grained datasets.

  • Takeaways & Limitations

    ImageNet architectures generalize well across datasets, whereas ImageNet features are less general than previously suggested.

  • Takeaways & Limitations

    The value of ImageNet pretraining diminishes with more training time, more data, and greater divergence from ImageNet labels, with some fine-grained datasets favoring competitive training from scratch.

Abstract

from arXiv · show

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy ($r = 0.99$ and $0.96$, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from ImageNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.

1. Introduction

This study tests whether stronger ImageNet architectures and features transfer better across vision tasks. Across 16 networks and 12 datasets, architecture quality transfers strongly, but ImageNet training choices and fine-grained tasks expose limits to feature transfer.

  • The motivating assumption is that better ImageNet architectures learn better features and therefore perform better across other vision tasks, but this had not been systematically tested across architectures.
  • The study evaluates transferability across 16 modern convolutional networks, 12 image classification datasets, and three settings: fixed features, fine-tuning, and random initialization.
  • r = 0.99 for fixed-feature transfer and r = 0.96 for fine-tuning, linking higher ImageNet accuracy with better transfer performance.
  • Regularizers that improve ImageNet performance can substantially harm transfer based on penultimate-layer features.
  • On two small fine-grained datasets, fine-tuning provides little benefit over random initialization, although better ImageNet architectures still achieve higher accuracy.

2. Related work

Prior work established that ImageNet features can transfer broadly and that fine-tuning often improves accuracy, but evidence about architecture and dataset effects remained mixed or incomplete.

  • Early ImageNet CNN transfer methods used fixed features with SVM or logistic-regression classifiers, outperforming hand-engineered features on tasks distinct from ImageNet.
  • Fine-tuning typically achieves higher accuracy than fixed-feature classification, especially for larger datasets or greater domain mismatch.
  • Studies of ImageNet subsets reached conflicting conclusions about whether transfer depends more on the number of classes or images per class.
  • Freezing later AlexNet layers during transfer between natural and manmade ImageNet subsets substantially reduces accuracy, unlike freezing the first layer.
  • Research also shows transfer can remain useful from extremely large datasets to ImageNet and when labeled data span multiple domains.

3. Statistical methods

The analysis compares accuracies across datasets of different difficulty using logit-transformed values and model-adjusted error bars. Correlations are computed on transformed ImageNet and transfer accuracies, with statistical tests assessing network differences.

  • The logit transformation makes additive accuracy changes comparable across datasets and interprets a change Δ as multiplying correct-classification odds by exp Δ.
  • Accuracy values are plotted on logit-scaled axes for comparisons across datasets.
  • Adjusted accuracy subtracts each dataset’s mean transformed accuracy across models before model means and standard errors are computed.
  • The reported r correlates logit-transformed ImageNet accuracy with logit-transformed transfer accuracy averaged across datasets; Spearman’s ρ appears in Appendix A.1.2.
  • Permutation or equivalent binomial tests compare paired network predictions, while t-tests compare average performance across datasets.

4. Results

Across transfer settings, stronger ImageNet architectures generally transferred better, but feature quality depended sharply on training configuration and task similarity. Fine-tuning usually helped, while ImageNet pretraining offered little accuracy benefit on two fine-grained datasets despite substantial speedups.

  • Transfer performance: r = 0.99 between ImageNet top-1 accuracy and fixed-feature transfer accuracy when all models used identical ImageNet training settings.Inception-ResNet v2 and NASNet Large were statistically tied for first place.
  • Transfer performance: Regularization choices caused low or nonsignificant fixed-feature correlations in public checkpoints despite strong correlations under matched training settings.The discrepancy arose from differences in regularization across checkpoints.
  • Training settings: Four Inception training choices harmed penultimate-layer transfer features while changing ImageNet top-1 accuracy by less than 1%.The choices were omitting batch-normalization scale, enabling label smoothing, enabling dropout, and using an auxiliary classifier head.
  • Training settings: Differences between optimal and suboptimal training settings were approximately as large as the gap between the worst and best optimally trained ImageNet models.Transfer-accuracy differences appeared earlier than ImageNet-accuracy differences and were consistent across datasets.
  • Fine-tuning: r = 0.96 between ImageNet top-1 accuracy and transfer accuracy when networks were fine-tuned.Fine-tuning generally reduced the impact of regularization settings compared with fixed-feature transfer.
  • Fine-tuning: Fine-tuning outperformed fixed-feature logistic regression in 179 of 192 dataset-model combinations and significantly improved every dataset except Caltech-101 when averaged across architectures.The improvement was generally larger for larger datasets, but substantial gains also occurred on Oxford 102 Flowers.
  • Random initialization: For networks trained from random initialization, the ImageNet-to-transfer correlation was r = 0.55 overall, r = 0.29 on datasets with fewer than 10,000 examples, and r = 0.86 on larger datasets.This correlation was more variable than in the ImageNet-pretrained settings.
  • Comparison with prior methods: Fine-tuning achieved state-of-the-art performance on half of the 12 datasets using the same image sizes as baseline methods.The results identify pretrained-model ImageNet performance as a critical factor in transfer performance.

5. Discussion

The discussion finds that ImageNet architectures transfer broadly, but ImageNet features have clear limits, especially under regularization and on fine-grained tasks. Pretraining helps most when data are scarce, while architecture quality remains valuable.

  • Strong ImageNet-to-transfer correlations suggest better ImageNet architectures learn more transferable representations.The discussion frames this as evidence against broad overfitting to ImageNet as a benchmark.
  • Regularizers that improve ImageNet performance can harm penultimate-layer features and have mixed effects during fine-tuning.
  • ImageNet pretraining accelerates convergence and improves many datasets, but its value diminishes with more training time, data, and label divergence.
  • On some fine-grained datasets, a few thousand labeled examples can make training from scratch competitive with fine-tuning.
  • The paper identifies a need for more adaptable visual features because ImageNet weights do not always transfer profitably to smaller, different datasets.

Supplementary Material for “Do Better ImageNet Models Transfer Better?"

The supplementary material documents statistical comparison procedures, dataset characteristics, and validation-set construction used to evaluate transfer across datasets.

  • Permutation tests compare two networks by randomly exchanging per-example predictions and measuring how often accuracy differences are at least as extreme as observed.
  • Table A.1 reports Pearson and Spearman correlations between ImageNet accuracy and average transfer accuracy across three transfer settings.
  • Pearson correlation is treated as more appropriate because effects are approximately linear and less dependent on the particular CNNs selected.
  • Most datasets have median shortest-side image sizes of at least 331 pixels, while Caltech-101 and CIFAR datasets are smaller.
  • Provided validation sets were used where available; otherwise, the authors constructed validation subsets and generally used the first supplied split.

A.3. Networks and ImageNet training procedure

The authors retrained a diverse set of classification networks under controlled ImageNet procedures, then specified transfer-training protocols for fixed features, fine-tuning, and random initialization.

  • The study generally uses retrained networks rather than publicly available checkpoints.
  • Table A.2 catalogs each examined network’s parameter count, penultimate-feature dimension, and input image size.
  • ImageNet training used batch size 4096, Nesterov momentum, weight decay, warmup, exponential learning-rate decay, and data augmentation.
  • The networks were not individually hyperparameter-optimized because retraining usually matched public-checkpoint ImageNet performance closely.
  • Fixed-feature transfer used multinomial logistic regression with L2 regularization selected over 45 values from 10^-6 to 10^5.
  • Fine-tuning initialized from ImageNet weights and trained for 20,000 steps with validation-selected learning rate and weight decay.
  • Random-initialization training used a similar protocol but broader learning-rate and weight-decay ranges, with longer training for dataset-size experiments.

B. Logistic regression performance of public checkpoints

Public checkpoints yield weaker and less reliable fixed-feature transfer than retrained models. Across training analyses, reduced regularization generally improves fixed features, while fine-tuning effects are smaller and dataset-dependent.

  • Public checkpoints: Public-checkpoint ImageNet accuracy correlates weakly and nonsignificantly with transfer accuracy, unlike retrained models.The cited passage reports r = 0.37 and p = 0.16 for public checkpoints; the retrained-model value is truncated in the supplied passage.
  • Public checkpoints: Retrained models outperform public checkpoints on 11 of 12 datasets and on 162 of 192 dataset/model pairs.
  • Fixed features: Across nearly all datasets and models, least-regularized models achieve the best penultimate-feature transfer despite lower ImageNet top-1 accuracy.
  • Fixed features: Regularization affects transfer performance earlier in training than ImageNet top-1 accuracy.
  • Fine-tuning: Regularization benefits fine-tuning clearly only when the same settings are used during pretraining and fine-tuning.
  • Fine-tuning: Fine-tuning is less sensitive to regularization than fixed-feature transfer, with dropout and auxiliary-head effects varying across models and datasets.

D. Relationship between dataset size and predictive power of ImageNet accuracy

ImageNet accuracy predicts performance on larger datasets more reliably than on smaller datasets when models are trained from random initialization. This dataset-size relationship was significant for scratch training but not for transfer settings.

  • p = 0.0002: dataset size significantly related to the correlation between ImageNet accuracy and accuracy on other datasets for random initialization.No significant relationship appeared in the transfer-learning settings.
  • Larger datasets make ImageNet accuracy a better predictor of models trained from scratch.Figure D.1 evaluates this relationship across 12 datasets using Pearson correlations based on 16 ImageNet networks.
  • One proposed explanation is that ImageNet performance reflects both inductive bias and capacity, with inductive bias potentially more important on smaller datasets.

E. Additional comparisons of logistic regression, fine-tuning, and training from random initialization

Additional comparisons show that fine-tuning usually outperforms fixed-feature logistic regression and scratch training, although the gaps vary by dataset. The analyses also include alternative classifier settings and published-performance comparisons.

  • Fine-tuning usually achieved higher accuracy than fixed-feature logistic regression or randomly initialized training, though some dataset gaps were small.
  • The relative performance of fixed-feature logistic regression versus scratch training depended heavily on the dataset.
  • State-of-the-art performance was reached on 4 datasets at native image sizes, or 6 when networks were retrained at 448 × 448.For CIFAR-10, CIFAR-100, and Stanford Cars, the best result was trained from scratch; other baselines used ImageNet pretraining.
  • The study additionally evaluated support vector machines and logistic regression with data augmentation on fixed ImageNet features.These settings were examined alongside the primary logistic-regression comparison.

G.2. Logistic regression with data augmentation

Logistic regression with data augmentation was compared with alternative fixed-feature classifiers and evaluated alongside duplicate-image controls. Its average improvement was slight, and it did not consistently produce the best dataset-level results.

  • 100/192 dataset/model pairs performed better with logistic regression using data augmentation.
  • Average log odds improved slightly from 1.32 without augmentation to 1.35 with augmentation.The best model without augmentation still outperformed the best augmented model on half of the 12 datasets.
  • The analysis compared ImageNet accuracy with transfer accuracy across 12 datasets and contrasted logistic regression with SVMs and augmentation.
  • The duplicate detector used a conservative threshold, producing some false positives but very few false negatives and therefore a worst-case overlap estimate.
  • 98.02% accuracy was achieved on the 9,863-example CIFAR-10 test set disjoint from ImageNet, versus 98.04% on the full test set.

I. Numerical performance results

The paper reports numerical performance for logistic regression, fine-tuning, and random initialization. Bold values identify the best or statistically indistinguishable models within each training setting.

  • Numerical results cover logistic regression, fine-tuning, and training from random initialization.
  • Bold-faced numbers mark models that were best or insignificantly different from the best in each training setting.
Loading 1805.08974v3…