PaperlayerView exact revision on arXiv

Source-linked AI summary

Do ImageNet Classifiers Generalize to ImageNet?

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

arXiv:1902.10811v2cs.CVcs.LGstat.ML

TL;DR

The paper asks whether classifiers trained and evaluated on heavily reused CIFAR-10 and ImageNet benchmarks generalize to new data from the same sources. It replicates the dataset-creation processes and finds substantial accuracy drops, while higher original-test accuracy predicts larger gains on the new tests; the evidence points away from adaptivity and toward harder images.

  • Problem

    CIFAR-10 and ImageNet have been intensely researched for almost a decade, raising concern that repeatedly reused test sets may provide limited evidence about generalization.

  • Method

    The authors replicate the dataset-creation processes for CIFAR-10 and ImageNet and evaluate a broad range of classification models on the resulting new test sets.

  • Results

    3% to 15% accuracy drops occur on CIFAR-10 and 11% to 14% on ImageNet, while model rankings remain nearly preserved and original-test improvements translate into larger new-test improvements.

  • Takeaways & Limitations

    The results suggest that accuracy drops are unlikely to arise from adaptivity and instead reflect difficulty differences between the original and new test sets.

  • Takeaways & Limitations

    The selection-frequency analysis does not identify which image characteristics make the new-test images harder for models.

Abstract

from arXiv · show

We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

1 Introduction

The paper tests whether classifiers generalize from heavily reused CIFAR-10 and ImageNet test sets to newly collected sets from the same sources. Models show substantial accuracy drops, but improvements on the original tests translate into even larger improvements on the new tests, making adaptivity an unlikely explanation.

  • Findings: 3% to 15% accuracy drops occur on CIFAR-10 and 11% to 14% on ImageNet when models are evaluated on newly replicated test sets.The ImageNet loss corresponds to approximately five years of progress during a highly active period of machine learning research.
  • Findings: Models retain almost the same relative accuracy ordering on the new test sets as on the original test sets.Models with the highest original-test accuracy remain the highest-performing models on the new test sets.
  • Interpretation: The results provide evidence that exhaustive test-set evaluations remain effective for improving image classification models, making adaptivity an unlikely explanation for the observed drops.Later models could have been more adapted to the original test sets, yet they experience smaller accuracy drops.
  • Interpretation: The authors instead attribute the drops to differences in relative test-set difficulty and show that selecting only the easiest candidate images can nearly recover original ImageNet accuracies.This indicates sensitivity to minutiae of the data-cleaning process and unreliable generalization even in a controlled reproducibility experiment.
  • Findings: Every percentage point of improvement on the original test set translates into a larger improvement on the new test set.Figure 1 reports linear-fit slopes of 1.7 for CIFAR-10 and 1.1 for ImageNet.

2 Potential Causes of Accuracy Drops

The paper separates accuracy drops between original and newly collected test sets into sampling, adaptivity, and distribution effects. Its evidence indicates that systematic distribution differences, rather than random chance or adaptive overfitting alone, mainly account for the observed drops.

  • Test-set error is used as a proxy for population loss because the underlying data distribution is unknown.This assumption supports claims that classifiers generalize to future examples from the same distribution.
  • The accuracy difference between original and new test sets is decomposed into generalization, adaptivity, and distribution gaps.The decomposition provides a framework for assessing which mechanism contributes to the observed accuracy drop.
  • Random sampling error is unlikely to explain the observed drops for the new ImageNet test set with 10,000 data points.The 95% confidence interval has size at most ±1%, while increasing confidence to 99.99% yields at most ±2%.
  • The adaptivity gap measures how adapting a model to the original test set makes test error underestimate population loss.Hyperparameter tuning on the test set can create dependencies that artificially inflate accuracy beyond random error.
  • The distribution gap quantifies the systematic effect of changing from the original distribution D to the new distribution D′.The paper notes that high-dimensional distributions are difficult to define and match exactly because dataset construction involves many design choices.
  • Across multiple model generations, later models show increased advantages on new test sets rather than diminishing returns.This pattern contrasts with adaptive overfitting and is interpreted as evidence that accuracy drops mainly stem from a large distribution gap.

3 Summary of Our Experiments

The experiments replicate CIFAR-10 and ImageNet dataset creation to test generalization, finding substantial accuracy drops on new test sets while higher-accuracy models remain comparatively more robust.

  • Experimental Design: The study recreates CIFAR-10 and ImageNet test-set construction by drawing from the original sources and replicating their manual annotation procedures.ImageNet replication uses Flickr images and MTurk workers; CIFAR-10 replication uses Tiny Images and student labelers.
  • A Significant Drop in Accuracy: The ImageNet top-5 drop for the 2018 state-of-the-art model is 6%, corresponding to approximately five years of progress.The model falls from 96% to 90% top-5 accuracy, while earlier ILSVRC models achieved 89% in 2013 and 93% in 2014.
  • Few Changes in the Relative Order: Models preserve nearly the same relative ordering on the new test sets, and original accuracy strongly predicts new accuracy through a linear relationship.Bootstrap confidence intervals for the fitted slopes are [1.63, 1.76] on CIFAR-10 and [1.07, 1.19] on ImageNet.
  • Few Changes in the Relative Order: Because fitted slopes exceed 1, each percentage point of original accuracy improvement yields a larger improvement on the new test sets, with no diminishing returns.The effect is stronger on CIFAR-10, whose slope is 1.7, than on ImageNet, whose slope is 1.1.

4 Understanding the Impact of Data Cleaning on ImageNet

The ImageNet replication varies MTurk-based image-selection strategies while holding the source and labeling requirements broadly consistent. These choices produce large accuracy differences, showing that data-cleaning details materially affect benchmark performance.

  • Sampling Strategies: Three ImageNet test sets use the same Flickr candidate pool and correctly labeled images, yet their model accuracies vary by 14%.All images are selected by more than 70% of MTurk workers on average.
  • MTurk Tasks: MTurk workers evaluate 48 candidate images per class, with tasks modeled on the original ImageNet instructions and quality-control images embedded.At least 20 annotated validation images are collected for each class.
  • MTurk Tasks: Selection frequency records the fraction of MTurk workers choosing each image, with at least ten workers recruited per task.Original validation images included in the tasks provide a reference for estimating selection frequencies.
  • Sampling Strategies: The three sampling strategies are MatchedFrequency, Threshold0.7, and TopImages, differing in how image selection frequencies determine each class’s ten-image sample.MatchedFrequency matches estimated class-specific distributions; Threshold0.7 requires frequency at least 0.7; TopImages chooses the ten highest-frequency images.
  • Results: TopImages raises average top-1 and top-5 accuracy by about 2%, whereas MatchedFrequency lowers them by 12% and 8%, respectively.Threshold0.7 produces smaller decreases of 3% in top-1 and 1% in top-5 accuracy.
  • Results: Changing from TopImages to MatchedFrequency decreases accuracy by about 14% in top-1 and 10% in top-5, exceeding variation among post-AlexNet convolutional networks.The corresponding across-model ranges are roughly 14% top-1 and 8% top-5 after excluding AlexNet-related models.
  • Results: Original and new ImageNet accuracies closely follow linear relationships for Threshold0.7 and TopImages.Each plotted model includes 95% Clopper-Pearson confidence intervals, and the fit uses a 95% bootstrap confidence region.

5 Discussion

The experiments find no diminishing returns consistent with adaptive overfitting, while distribution differences remain the leading explanation for accuracy drops. A simple difficulty model explains the observed linear relationship between original and new accuracies, but the results expose the brittleness of current classifiers to dataset-construction choices.

  • 5.1 Adaptivity Gap: The models show no diminishing returns: each accuracy point gained on the original test set translates to more than 1% on the new test set.This pattern is opposite to the standard adaptive-overfitting scenario.
  • 5.1 Adaptivity Gap: The experiments do not exclude adaptive-overfitting variants that produce a roughly constant accuracy drop across models.Testing that possibility likely requires a truly i.i.d. test set rather than a separately collected one.
  • 5.2 Distribution Gap: The distribution gap remains the leading hypothesis because sampling-strategy changes can substantially alter ImageNet accuracy while leaving other dataset-creation components unchanged.The authors identify the subjective human-annotation step as a likely source of distribution differences.
  • 5.3 A Model for the Linear Fit: A Gaussian difficulty model yields a linear relationship between original and new test accuracy on the probit scale.The model assigns each image a scalar difficulty and each classifier a skill parameter, with accuracy determined by their relationship.
  • 5.3 A Model for the Linear Fit: The observed ImageNet accuracies from 21% to 83% fit better under probit scaling than under an ordinary linear scale.On this scale, the relationship is expressed using the Gaussian CDF and scalars u and v.

6 Related Work

The paper differs from prior dataset-bias and transfer-learning studies by reproducing benchmark data collection and testing on previously unseen images. It also distinguishes its benign sampling shifts from adversarial or explicitly engineered image perturbations.

  • Dataset Biases: Unlike cross-dataset bias studies, this work attempts to reproduce the original data distribution and shows that small collection differences can cause significant accuracy drops.The comparison also emphasizes that the prior experiments did not test previously unseen data and therefore could not rule out adaptive overfitting.
  • Transfer Learning From ImageNet: Unlike ImageNet transfer-learning studies, this work evaluates models on new data without retraining them on target datasets.The cited transfer-learning experiments also cannot rule out adaptive overfitting because they do not use new data.
  • Adversarial Examples: The reported accuracy drops arise from benign data-sampling variations rather than adversarial perturbations finely tuned against models.The authors contrast their new images with adversarial examples and emphasize that no adversary is used.
  • Adversarial Examples: The tested ℓ∞-robustness baseline offers no apparent benefit against the distribution shift studied on CIFAR-10.Its accuracies fall almost exactly on the linear fit defined by the other models.
  • Non-Adversarial Image Perturbations: Compared with perturbation studies that modify ImageNet images using specified transformations, this work uses new images designed to match the existing validation-set distribution closely.The properties of the new images responsible for the accuracy drops remain unclear.

7 Conclusion & Future Work

The paper frames reliable generalization as an unresolved challenge and proposes future dataset practices and test-set designs to measure it more accurately. It also argues that CIFAR-10 and ImageNet remain valuable benchmarks, while their creation and use should be revisited for current reliability expectations.

  • Current models still face significant generalization and reliability hurdles even in benign environments.
  • Future work should characterize the distribution gap, develop more robust models, and determine whether humans also find the new test sets harder.
  • A single test set is insufficient to comprehensively characterize model reliability, motivating new test data from varied sources.
  • Future dataset practices should release creation code, assess annotator quality, and consider hidden long-term holdout sets.
  • Alternative designs include simpler human tasks with broader visual variation and expert-annotated test sets, although the latter violate train-test i.i.d. assumptions.
  • CIFAR-10 and ImageNet were valuable benchmarks for an earlier accuracy regime, but rising model accuracy and reliability expectations now justify revisiting dataset construction and use.

B.2.1 Statistical Error

The authors rule out ordinary statistical fluctuation and near-duplicate effects as sufficient explanations for the CIFAR-10 accuracy drop. Hyperparameter retuning also provides little recovery, while hard images remain recognizable to humans.

  • A 95% confidence interval of about ±1.2% for the new test set is too small to explain the observed accuracy drop.The original test set intervals are smaller still: roughly ±0.6% at 90% accuracy and ±0.3% at 97%.
  • Near-duplicate removal explains only a 0.6% reduction in estimated accuracy, versus the observed 8%–9% drops for VGG and ResNet models.For a 93% model with 8% near-duplicates, the estimated non-near-duplicate accuracy is approximately 92.4%.
  • The near-duplicate analysis counted images with the same object and pose as duplicates, while allowing changes in zoom, color, scale, stretching, or small shifts.
  • Hyperparameter grid search did not find a significantly better new-test accuracy; the largest improvement was from 85.3% to 85.8%.
  • Visual inspection found that images no model correctly classified were valid and recognizable to humans.

B.2.5 Human Accuracy Comparison

A preliminary human study compared the hardest original and new CIFAR-10 images, selected by model errors. Humans showed no meaningful average accuracy drop, but the study remains limited and requires further experiments.

  • The study focused on approximately the hardest 5% of images, defined by how many evaluated models classified correctly.It included 500 original-test images and 115 new-test images after ties were rounded into the selection.
  • Nine UC Berkeley graduate students labeled randomly ordered images from the ten CIFAR-10 classes without knowing their dataset origin.
  • The average human accuracy difference was -0.8%, indicating no drop on this subset between the original and new test images.
  • The authors interpret the result as suggesting that the new test set is not significantly harder for humans.
  • The human-accuracy experiment is preliminary, and a fuller CIFAR-10 human baseline requires additional experiments.

B.2.8 Training a Discriminator for Original vs. New Test Set

The discriminator experiment tested whether a convolutional network could distinguish original from new CIFAR-10 test images. It achieved only modest discrimination, supporting the view that the datasets differ subtly rather than through an easily detected visual signature.

  • The experiment trained binary classifiers to distinguish original from new, class-balanced CIFAR-10 images using 3,200 training and 800 test examples.
  • The discriminator models used ResNet-32 and ResNet-110 architectures trained for 160 epochs, with either random or original-task pre-trained initialization.
  • The best discriminator accuracy was 53.1%, only slightly above chance.
  • The confidence intervals for discriminator accuracies overlapped 50%, so the models did not discriminate well between the test sets.
  • The new dataset had 8% ship images rather than the original 10%, prompting construction of an exactly class-balanced 2,000-image variant for additional accuracy analysis.

B.3.5 Hard Images

The new test set exposes hard images that current models systematically misclassify, while its construction follows ImageNet’s original collection and annotation process. This section also situates the test set within ImageNet’s benchmark design and distribution-matching choices.

  • Hard Images: Figure 4 identifies new CIFAR-10 images that every evaluated model misclassified, despite the images having correct class labels.Examples include automobiles predicted as trucks or airplanes, and birds or horses predicted as frogs.
  • ImageNet Background: ImageNet’s classification benchmark uses 1.2 million training images, 50,000 validation images, and 100,000 test images across 1,000 categories.The test labels were withheld and evaluated through an evaluation server.
  • Dataset Construction: The new ImageNet test set replicates the validation-set distribution because Flickr provides a controllable source and the validation set has labels for inspecting class-specific image distributions.Validation accuracy is also the most relevant comparison because most papers report it and models are developed using the validation set.
  • Dataset Construction: The new test set targets 10,000 images while limiting the search period to reduce potential distribution shift from expanding the candidate pool.The design prioritizes confidence intervals that are already small without searching for substantially more images.
  • Dataset Construction: The dataset pipeline has two stages: creating a candidate image pool and sampling the final clean dataset from that pool.Candidate images were collected from Flickr using WordNet synonyms as separate class-specific queries.

C.1.2 Sampling a Clean Dataset

The clean-dataset procedure matches class-specific annotation-frequency distributions, then manually reviews and resamples images to remove incorrect or near-duplicate examples. This improves label quality but may introduce distribution shift.

  • Candidate Pool: The candidate pool contains MTurk annotations and image selection frequencies used to sample a test set resembling the existing validation distribution.Selection frequency is the central signal used in the subsequent sampling step.
  • Reproducibility Constraints: The original ImageNet class-specific agreement thresholds are unavailable, and exactly reproducing the 2010–2012 MTurk worker population is impossible.The authors therefore explored alternative sampling strategies using selection frequencies measured for their own MTurk population.
  • Sampling Strategy: Sampling matches the per-class selection-frequency distribution rather than only its overall mean, accounting for class-specific differences in image ambiguity and quality.The frequency bins range from [0.0, 0.2) for the worst images to [0.8, 1.0] for the best images.
  • Sampling Strategy: Ten images per class are sampled from the corresponding frequency bins, with uniform sampling within each bin and fallback sampling when a class lacks enough candidates.This procedure produced a 10,000-image test set with average sampling frequency 0.73 versus 0.71 for annotated validation images.
  • Final Reviewing: Manual review removes incorrect or near-duplicate images and triggers repeated resampling, but it can create distribution shift relative to the original validation set.The authors accepted this possibility because correct labels were considered more important than minimizing distribution shift.

C.2 Model Performance Results

The evaluation compares broad model families on the original and newly constructed ImageNet sets, while testing whether cross-validation explains the observed gap. The new cross-validation splits show no comparably large accuracy difference.

  • Evaluation: The evaluation reports top-1 and top-5 accuracies for a broad range of models on the original validation set and three ImageNet test sets.The main text emphasizes convolutional neural networks because they are the most widely used ImageNet model family.
  • Accuracy Comparison: The accuracy relationship is better represented in the probit domain than by a linear model when ImageNet accuracies span a wide range.Figures 6 and 13 exhibit linear fits after applying the inverse Gaussian CDF to accuracy scores.
  • Accuracy Comparison: Figure 6 compares original validation accuracy with MatchedFrequency accuracy for each tested model using 95% Clopper-Pearson confidence intervals.It presents both linear-axis and probit-scale plots, with accuracy α transformed to Φ^-1(α).
  • Cross-Validation: The cross-validation experiment addresses whether adaptive overfitting or the non-i.i.d. relationship between training and validation data explains the new-test-set accuracy gap.The study creates five additional class-balanced folds from 250,000 training images while treating the validation set as one fold.
  • Cross-Validation: Cross-validation on ImageNet’s training and validation data produces no large accuracy differences across new splits.All differences fall within the 95% confidence intervals, unlike the significantly larger drops on the new test sets.

C.3.2 Impact of Dataset Revisions

Dataset revisions remove images judged incorrect, nonrepresentative, or otherwise unsuitable, and the authors measure whether these interventions alter the main accuracy findings. Reviewing improves accuracy but does not explain the accuracy drop or the accuracy relationship.

  • Reviewing Criteria: Subjective reviewing choices can introduce distribution shift, although the authors regard dataset correctness as more important than minimizing that shift.They note that an easily identifiable shift, such as more black-and-white images, could make the resulting accuracy drop less informative.
  • Reviewing Criteria: Reviewing excludes images that are not representative, are artificial or heavily altered, differ substantially in distribution, or are unclear, blurry, occluded, or overly edited.Reviewers compare candidate images with random validation examples from the same class to detect distribution shifts.
  • Revision Effects: About 4% accuracy improvement for resnet50 followed reviewing on the MatchedFrequency dataset, providing evidence that reviewing did not cause the accuracy drop.The authors tracked dataset revisions and measured model accuracy across them.
  • Revision Effects: The first dataset revision still showed a similarly good linear fit between original and new test accuracies, indicating that the relationship was not caused by reviewing.This evaluation used the model testbed before the later reviewing passes.
  • Revision Effects: The review interface blacklisted incorrect images and replaced them with new candidates, while displaying original validation images for distribution comparison.The process was repeated until the dataset converged.

C.4.4 Full Results Tables

The full results section documents model accuracies across the original ImageNet validation set and three new test sets, using figures, tables, and image-selection analyses. It also examines dataset review, selection frequency, and ambiguous class boundaries.

  • Dataset review: The first MatchedFrequency revision already shows a linear relationship between original and new test accuracy before the final author review step.The review interface was used only during final dataset review, not for MTurk labeling.
  • Image examples: The section presents randomly selected images from the original and new test sets, plus examples of three ambiguous ImageNet class pairs.The highlighted pairs are illustrative; additional ImageNet classes also have ill-defined boundaries.
  • Selection frequency: Model accuracies are stratified into five selection-frequency bins to assess how often an image was selected during dataset construction.This analysis isolates the relationship between image selection frequency and model accuracy.
  • Accuracy comparisons: The figures compare model accuracy on the original validation set with accuracy on the new test sets, including confidence intervals and fitted relationships.Figure 13 additionally includes Fisher Vector models and uses a probit-transformed accuracy scale.
Loading 1902.10811v2…