Do ImageNet Classifiers Generalize to ImageNet?

• Adaptivity is unlikely to explain the drops: models that scored highest on the original test sets remained highest on the new sets, with no diminishing returns in accuracy.Later models saw smaller drops despite potentially greater exposure to the original test set, suggesting exhaustive test-set evaluation can improve models.Source: §1 · pp. 1–2
• Including only the easiest candidate images nearly recovered original ImageNet accuracy, suggesting that classifiers remain sensitive to small differences in data cleaning.This brittleness puts claims about human-level performance in context [20] [48] and shows unreliable generalization even in a controlled reproducibility experiment.Source: §1 · p. 2
• The paper releases its new test sets and corresponding code to support future research.Source: §1 · p. 2
• Accuracy fell 3%–15% on CIFAR-10 and 11%–14% on ImageNet’s replicated test sets, with the ImageNet loss equaling about five years of progress.The study recreated the dataset-creation process for both benchmarks.Source: §1 · p. 1, Figure 1 · p. 2
• Accuracy gains on the original test sets translated into larger gains on the new sets, with fitted slopes of 1.7 for CIFAR-10 and 1.1 for ImageNet.Model rankings were almost exactly preserved across the original and new test sets.Source: §1 · p. 1, Figure 1 · p. 2

2 Potential Causes of Accuracy Drops

• With 10,000 test examples, 95% accuracy intervals are at most ±1% and 99.99% intervals at most ±2%, so random chance alone cannot explain the drops.The intervals are smaller at higher accuracies, the regime of the best-performing models.Source: §2 · p. 3

Adaptivity Gap.

• The adaptivity gap captures how tuning or otherwise adapting a model to the original test set can make its test error underestimate population loss.If the model were independent of the test set, the gap would follow the same concentration laws as the standard generalization gap; test-set hyperparameter tuning undermines that assumption.Source: Adaptivity Gap · p. 4
• The distribution gap measures systematic effects of changing from the original distribution D to the new distribution D′, which are difficult to eliminate or precisely characterize.The authors note that high-dimensional distributions are hard to define precisely and that real-data collection involves many design choices.Source: §2 · p. 3, Distribution Gap · p. 4

2.1 Distinguishing Between the Two Mechanisms

• Later models gain an increased advantage over earlier ones, supporting the conclusion that accuracy drops mainly stem from a large distribution gap.The authors contrast this with the possibility that successive models improve only through increasing adaptive overfitting to the original test set.Source: §2.1 · p. 4

3.1 Choice of Datasets

• CIFAR-10 and ImageNet were chosen for their widespread use and documented curation; contrary to expectations, CIFAR-10 did not show more adaptive overfitting and ImageNet was not significantly more robust.CIFAR-10’s smaller scale suggested greater hyperparameter tuning and sensitivity to distribution changes, while ImageNet’s broader image coverage suggested stronger generalization.Source: §3.1 · p. 5

3.2 Dataset Creation Methodology

• New test sets mimicked the original data sources and subclass distributions, replicated manual labeling, and sampled 2,000 CIFAR-10 and 10,000 ImageNet images.The authors drew from Tiny Images for CIFAR-10 and Flickr for ImageNet; the sample sizes aimed for 95% confidence intervals of about ±1%.Source: §3.2 · pp. 5–6

3.3 Results on the New Test Sets

• All models lost accuracy: VGG and ResNet dropped 8% on CIFAR-10 and 11% on ImageNet; the best ImageNet model’s top-1 accuracy fell 11%, from 83% to 72%.The CIFAR-10 state of the art dropped 3%, from 98.4% to 95.5%; the best ImageNet model’s top-5 accuracy fell 6%, from 96% to 90%.Source: Table 1 · p. 7, §3.3 · p. 7
• Original accuracy strongly predicted new accuracy, with few ranking changes; fitted slopes above 1 mean more accurate models lost less accuracy on both datasets.The slope was 1.7 on CIFAR-10 and 1.1 on ImageNet; 95% bootstrap confidence intervals were [1.63, 1.76] and [1.07, 1.19], respectively.Source: §3.3 · p. 8

3.4 Experiments to Test Follow-Up Hypotheses

• Follow-up experiments, including hyperparameter retuning and training on part of the new test set, did not explain the accuracy drops; the authors conjectured that small annotation-process variations caused them.Source: §3.4 · p. 8

Sampling Strategies

• All three new test sets drew from the same Flickr pool, contained correctly labeled images, and had average worker selection above 70%, yet accuracies varied by 14%.Source: §4 · p. 8
• Average worker selection frequency was 0.93 for TopImages, 0.85 for Threshold0.7, and 0.73 for MatchedFrequency, versus 0.71 for the original validation set.MatchedFrequency was used in preceding sections because its selection frequencies were closest to the original validation set.Source: §4 · p. 9

Results

• About 14% (top-1) and 10% (top-5) separate TopImages and MatchedFrequency; TopImages is about 2% higher on both metrics, while MatchedFrequency is 12% and 8% lower, respectively.Threshold0.7 is 3% lower in top-1 and 1% lower in top-5 than the original validation set.Source: §4 · p. 9, Table 2 · p. 10
• The sampling-strategy accuracy variation exceeds the variation among tested post-AlexNet convolutional networks, spanning roughly 14% in top-1 and 8% in top-5.Source: §4 · pp. 9–10
• Accuracies on Threshold0.7 and TopImages closely followed a linear fit against original validation accuracy.Source: Figure 2 · p. 10, §4 · p. 10

5.1 Adaptivity Gap

• On CIFAR-10 and ImageNet, each accuracy point gained on the original test set corresponds to more than one point on the new set, with no diminishing returns.The authors say this contradicts the standard adaptive-overfitting scenario, though it does not exclude variants that cause a roughly constant accuracy drop across models.Source: §5.1 · p. 11
• The authors say their experiments cannot distinguish two possible reasons for the lack of adaptive overfitting: constrained test-set interaction and a limited set of evaluated models.Shared errors among CIFAR-10 convolutional networks may indicate model similarity, but the authors call for more benchmark data, especially where adaptive overfitting occurs naturally.Source: §5.1 · p. 11

5.2 Distribution Gap

• The authors identify the distribution gap as the leading explanation for accuracy drops, noting that ImageNet sampling changes can substantially affect accuracy even with the same data source.They suggest subjective annotation conditions and the lack of exact definitions for some ImageNet classes make the original distribution difficult to replicate.Source: §5.2 · p. 12
• Current ImageNet models struggle to generalize from easy to hard images, and a hypothetical test set of correct Flickr images selected by over half of MTurk labelers could yield even lower accuracies.The paper links this concern to the difficulty of defining the data distribution and to the tested models’ brittle behavior.Source: §5.2 · p. 12

5.3 A Model for the Linear Fit

• Across ImageNet testbed models with original accuracies from 21% to 83%, a probit scaling fits the original-to-new accuracy relationship better, and a simple Gaussian difficulty model yields a linear relationship on that scale.The model illustrates how distribution shifts could preserve model ordering while changing accuracy; the authors do not present it as the true explanation.Source: §5.3 · pp. 12–13

Dataset Biases

• Dataset-bias work tests cross-dataset generalization, where larger distribution changes and accuracy drops are expected; unlike this work, it does not test unseen data and cannot rule out adaptive overfitting.This work instead aims to reproduce the original distribution and finds that small sampling variations can cause significant accuracy drops.Source: §6 · p. 14

Transfer Learning From ImageNet

• Kornblith et al. [33] retrain ImageNet models on target datasets and find better ImageNet models usually perform better there, but their adapted-task experiments use no new data.They also report an approximately linear relationship between ImageNet and transfer accuracy; their setup does not measure drops caused by small data-generating variations.Source: §6 · p. 14

Non-Adversarial Image Perturbations

• Unlike adversarial examples [2] [50], which require finely tuned perturbations, benign variations in data sampling can cause significant accuracy drops without an adversary.Studies of specified perturbations [17] [23] alter existing validation images; this work instead evaluates new images matched as closely as possible to the validation-set distribution, leaving the cause of the drops unclear.Source: §6 · p. 14

Adversarial Examples

• The tested CIFAR-10 ℓ∞-robustness baseline [41] falls almost exactly on the other models’ linear fit, suggesting ℓ∞ robustness offers no benefit against this distribution shift.The authors identify testing other robustness forms, including spatial transformations and color-space changes, as future work.Source: §6 · p. 14

7 Conclusion & Future Work

• The authors report no diminishing returns from test-set reuse on CIFAR-10 and ImageNet, and propose testing whether other tasks resist adaptive overfitting.Possible follow-ups include reproducibility experiments in natural language processing and analyses of Kaggle or CodaLab competition data.Source: §7 · p. 15
• The selection-frequency analysis suggests less frequently selected images are harder for models, but their challenging visual attributes remain unexplained.The authors propose investigating object size, image filters, and unusual poses, and consider data augmentation, robust optimization, added hard examples, or new classification methods to improve robustness.Source: §7 · p. 15, §B.2 · p. 26
• Human accuracy on CIFAR-10 was not significantly affected by the shift, suggesting the images are harder for trained models than for humans.The authors call for additional human-accuracy experiments on both CIFAR-10 and ImageNet to better establish the human baseline.Source: §7 · p. 15
• Evaluating models on a single test set may not fully characterize their reliability, motivating new test data from varied sources.The authors also ask whether accuracy on other test sets will follow a linear function of original test accuracy.Source: §7 · p. 16
• Future dataset practices could include releasing creation code, measuring annotator quality, retaining a long-term hidden hold-out, and using simpler human tasks or expert test labels.The authors suggest standardized annotator tests and variants with different annotation processes; simpler tasks could retain fewer, clearer classes while expanding pose, lighting, occlusion, and corruption variety. Expert test annotations may improve label correctness and example diversity, despite violating the train-test i.i.d. assumption.Source: §7 · p. 16
• The authors emphasize that their recommendations do not negate CIFAR-10 or ImageNet’s value, but argue that dataset design should evolve with model accuracy and reliability expectations.They note that the datasets were assembled for a different accuracy regime and that ImageNet helped guide progress toward better models.Source: §7 · p. 16

B Details of the CIFAR-10 Experiments

• CIFAR-10 offers a useful reproducibility test because it remains actively researched, spans models with varied accuracies, and has publicly available model code.Its smaller size and accessibility also make follow-up experiments easier, while potentially increasing exposure to test-set adaptivity relative to ImageNet. The appendix opens with a contents guide to detailed CIFAR-10 and ImageNet experiments.Source: A Overview of the Appendix · p. 23, B Details of the CIFAR-10 Experiments · p. 24

B.1 Dataset Creation Methodology

• The new CIFAR-10 test set was designed to match the original keyword distribution, using reviewed selections and near-duplicate filtering to preserve unseen examples.The authors selected common keywords covering more than 95% of CIFAR-10, followed the original labeling guidance, and kept collection separate from classifier evaluation.Source: §B.1 · pp. 25–26

B.2.1 Statistical Error

• The new-set 95% confidence interval is about ±1.2%, versus ±0.6% at 90% and ±0.3% at 97% original-set accuracy, ruling out statistical error as the main explanation.Source: §B.2.1 · p. 26

B.2.2 Differences in Near-Duplicate Removal

• For a model with 93% original accuracy, 8% near-duplicates at 100% accuracy imply about 92.4% accuracy on non-duplicates—a 0.6% effect, versus the observed 8%–9% drop.Source: §B.2.2 · p. 27
• Near-duplicates were identified by visually inspecting top-10 neighbors under ℓ2 distance and SSIM, counting same-object, same-pose images despite changes such as zoom or shifts.Rotated images or images with the object in a different pose were not counted as near-duplicates.Source: §B.2.2 · pp. 27–28

B.2.3 Hyperparameter Tuning

• Grid-searching VGG learning rate, dropout, and weight decay found no significantly better new-set accuracy; the largest improvement was 85.3% to 85.8%.The best performance was never at an extreme of a tested hyperparameter range.Source: §B.2.3 · p. 28

B.2.4 Visually Inspecting Hard Images

• Visual inspection found that the hard new-set images that no model classified correctly were valid images recognizable to humans.Source: §B.2.4 · p. 28

B.2.5 Human Accuracy Comparison

• On the top 5% model-hard images (500 original, 115 new), humans averaged a -0.8% accuracy difference; four of nine favored the original set and five the new set, a preliminary result showing no average drop.The images were ranked by the number of model errors.Source: §B.2.5 · pp. 28–29, Table 3 · p. 29

B.2.6 Training on Part of Our New Test Set

• If the new test set differed substantially from the original CIFAR-10 distribution, training on part of it could improve accuracy on its held-out fraction.Source: §B.2.6 · p. 29
• The experiment added about 1,000 class-balanced new-test images to the full CIFAR-10 training set, then tested on the remaining half.The vgg16_keras model was retrained, and the experiment was repeated with two different random splits.Source: §B.2.6 · p. 29
• 85.1% and 85.4% accuracy after retraining compared with 84.9% without extra data suggest no large distribution shift or inability to learn the modified distribution.The experiment used two randomly selected splits.Source: §B.2.6 · pp. 29–30

B.2.7 Cross-validation

• Cross-validation used five class-balanced training-set splits, combining four with the original test set and reserving the remaining split as the new test set.Source: §B.2.7 · p. 30
• The vgg_15_BN_64, wide_resnet_28_10, and shake_shake_64d_cutout models were retrained on each of the five cross-validation datasets.Their accuracies are reported in Table 4.Source: §B.2.7 · p. 30
• Cross-validation accuracy varied significantly less than the drop to the new test set, while split accuracies stayed close to the original test-set accuracy.The reported accuracies are in Table 4.Source: §B.2.7 · p. 30, Table 4 · p. 30

B.2.8 Training a Discriminator for Original vs. New Test Set

• The best discriminator reached 53.1% accuracy, and the 95% confidence intervals overlapped 50% chance, indicating weak separation of the original and new CIFAR-10 test sets.The class-balanced binary-classification experiment compared models trained from scratch with models initialized from CIFAR-10 pretraining.Source: §B.2.8 · pp. 30–31, Table 5 · p. 31

B.2.9 An Exactly Class-balanced Test Set

• The new test set contained 8% ship images versus 10% in the original; exact balancing yielded around 0.3% gains for lower-accuracy models, while the best model’s drop remained unchanged.The exactly balanced version contained 2,000 images, with 200 per class.Source: §B.2.9 · p. 31

B.3 Additional Figures, Tables, and Lists

• Appendix B.3 provides supplementary figures, tables, and lists that did not fit in the preceding CIFAR-10 experiment sections.Source: §B.3 · p. 31

B.3.1 Keyword Distribution in CIFAR-10

• Table 6 documents the keyword distributions for each of the ten classes in the original and new CIFAR-10 test sets.Source: §B.3.1 · p. 31, Table 6 · p. 32

B.3.2 Full List of Models Evaluated on CIFAR-10

• Appendix B.3.2 lists the evaluated CIFAR-10 models with references and links to their corresponding source code.Source: §B.3.2 · p. 35

B.3.3 Full Results Table

• Table 11 provides detailed accuracy scores for the original and new CIFAR-10 test sets.Source: §B.3.3 · p. 36

B.3.4 Full Results Table for the Exactly Class-Balanced Test Set

• Table 12 provides detailed accuracy scores for the original CIFAR-10 test set and the exactly class-balanced version of the new test set.Source: §B.3.4 · p. 36

B.3.5 Hard Images

• Figure 4 shows correctly labeled CIFAR-10 images that every model in the testbed misclassified.Examples include automobiles predicted as airplanes or trucks.Source: §B.3.5 · pp. 36, 39, Figure 4 · p. 39

C Details of the ImageNet Experiments

• CIFAR-10 models may fail to generalize under small distribution variations, motivating a test of whether ImageNet models fare better on its broader range of natural images.ImageNet has about 24× more training images and roughly 100× more pixels per image than CIFAR-10.Source: C Details of the ImageNet Experiments · p. 40
• The study focuses on ILSVRC2012 classification, whose benchmark comprises 1.2 million training images, 50,000 validation images, and 100,000 test images across 1,000 categories.The test labels were withheld and evaluation was limited through a server, while validation labels and scores were available without similar restrictions; most papers reported validation accuracy.Source: C Details of the ImageNet Experiments · p. 40

C.1 Dataset Creation Methodology

• The authors chose to replicate the validation-set distribution because Flickr searches are controllable, labels facilitate inspection, and validation accuracy is the most relevant comparison and may be adaptively overfit.They targeted 10,000 images to keep confidence intervals small while limiting temporal distribution shift that a longer search period could introduce.Source: §C.1 · p. 41

C.1.1 Creating a Candidate Pool

• For each class, the Flickr candidate search used WordNet synonyms, a post-release date range, and upload-date ordering to reduce temporal shift and avoid classifier-dependent ranking bias.The initial date range was July 11, 2012 to July 11, 2013, and the initial target was up to 100 images per class; the collection yielded 208,145 candidates.Source: §C.1.1 · pp. 42–43
• When classes needed more candidates, the search was expanded in stages, including parent-class terms, higher image limits, a longer date range, compound queries, and alternative or translated terms.The stages proceeded only as needed for each class.Source: §C.1.1 · pp. 42–43
• MTurk workers filtered candidate images for class membership and suitability, with at least ten workers evaluating each image and original validation images embedded for quality control.Workers saw grids of 48 images and were instructed to accept target-class images despite clutter or occlusion but reject drawings and paintings.Source: §C.1.1 · p. 43
• To prevent near-duplicates from undermining generalization tests, the authors compared candidates with one another and ImageNet using pixel and VGG-feature distances and SSIM, then manually reviewed close pairs and removed duplicates.Near-duplicates could test memorization rather than generalization, or reproduce images potentially exposed to adaptive overfitting.Source: §C.1.1 · pp. 43–44

C.1.2 Sampling a Clean Dataset

• Because the original agreement thresholds were incompletely specified and the 2010–2012 MTurk population could not be replicated, the authors explored three sampling strategies and calibrated them using original validation images.The original images included in MTurk tasks let the authors estimate selection frequencies for their own worker population.Source: §C.1.2 · pp. 44–45
• Sampling matched each class’s selection-frequency distribution rather than only the overall mean, using five frequency bins to represent worker-assessed image quality.For each class, ten candidates were sampled according to its annotated validation-image histogram; when a bin ran out, candidates were drawn from the next higher bin.Source: §C.1.2 · p. 45
• The sampled test set contained 10,000 images with an average selection frequency of 0.73, close to 0.71 for annotated validation images.When candidate images were unavailable in the target frequency bin, sampling from a higher bin affected 0.8% of the set and made it slightly easier.Source: §C.1.2 · p. 45
• Manual review removed incorrect or near-duplicate images, followed by resampling, until the dataset converged after 33 iterations.Reviewers accepted the possibility of distribution shift because correct labels were prioritized; the process was intended to minimize dependence on existing models.Source: §C.1.2 · pp. 45–46
• The main accuracy drop and approximately linear relationship between original and new accuracy also held for the first dataset iteration without additional review.The authors noted that removing incorrect images makes the dataset easier, opposite to the observed accuracy-drop trend.Source: §C.1.2 · p. 46

C.2 Model Performance Results

• The evaluation covered a broad range of models on the original validation set and new test sets, with Tables 14 and 15 reporting MatchedFrequency top-1 and top-5 accuracy.Figure 12 visualizes top-1 and top-5 accuracy on all three test sets.Source: §C.2 · p. 46
• The main-text plots focus on convnets, excluding lower-accuracy Fisher Vector models to preserve resolution and emphasize the widely used ImageNet model family.The exclusion applies to Figure 12 and the main text.Source: §C.2 · p. 46
• Figure 13 presents top-1 and top-5 accuracies for all three datasets in the probit domain.The paper points to Figures 6 and 13 as exhibiting a linear fit in the probit domain.Source: §C.2 · pp. 46–47
• On ImageNet, a probit-scale linear fit accurately described original-to-new test accuracy across both model families over a 60-percentage-point range.Figure 6 compares linear- and probit-scale fits; the probit fit has a smaller 95% confidence region.Source: §C.2 · p. 46, Figure 6 · p. 47

C.3 Follow-up Hypotheses

• The large gap between original and new accuracy motivated investigation of multiple hypotheses.The passage describes the gap as concerningly large, as it also did for CIFAR-10.Source: §C.3 · p. 47

C.3.1 Cross Validation

• Cross-validation tested whether adaptive overfitting or the validation set’s non-i.i.d. relationship to training data could explain the accuracy drop.The experiment treated the existing validation set as one fold, formed five class-balanced 50,000-image folds from a 250,000-image training sample, and trained one resnet50 model [21] per split with unchanged training-set size.Source: §C.3.1 · pp. 47–48
• All cross-validation accuracy differences fell within the 95% confidence intervals, unlike the significantly larger drops on the new test sets.Table 13 reports resnet50 accuracy on the cross-validation splits; the passage does not provide the individual accuracy values.Source: §C.3.1 · p. 48, Table 13 · p. 48

C.3.2 Impact of Dataset Revisions

• Reviewers compared ten original validation images with ten candidate images per class, while making subjective choices to remove visually identifiable distribution shifts and measuring their effects.The authors give artificial great white shark images as an example of a mismatch they corrected.Source: §C.3.2 · pp. 48–49
• About 4%: reviewing improved ResNet-50 top-1 accuracy on the new MatchedFrequency test set, indicating review did not cause the original-to-new accuracy drop.The authors report this as evidence that manual reviewing did not cause the accuracy gap.Source: §C.3.2 · p. 49
• The first MatchedFrequency revision still showed a good linear fit between original- and new-test accuracies, so the relationship was not a result of final reviewing.The fit was described as similar in quality to the one for the final dataset.Source: §C.3.2 · p. 49

C.4 Additional Figures, Tables, and Lists

• Appendix C.4 collects supplemental figures, tables, and lists that did not fit in the preceding ImageNet experiment sections.Source: §C.4 · p. 49

C.4.1 MTurk User Interfaces

• Both the original ImageNet and experimental MTurk tasks used 48 images and similar directions, while the experimental interface required scrolling and provided more image detail.Source: §C.4.1 · p. 49

C.4.2 User Interface for our Final Reviewing Process

• The reviewer interface paired candidate images with original validation examples and supported blacklisting incorrect images and marking near-duplicates for replacement or removal.It displayed 20 images in four rows, with candidates above a random sample of target-class validation images.Source: §C.4.2 · p. 50

C.4.3 Full List of Models Evaluated on ImageNet

• The appendix lists all ImageNet models evaluated, with references and links to their source code.Source: §C.4.3 · p. 50

C.4.4 Full Results Tables

• Tables 14–19 report top-1 and top-5 accuracy scores for the original validation set and the MatchedFrequency, Threshold0.7, and TopImages test sets.The MatchedFrequency tables also report relative changes in model ranking.Source: §C.4.4 · p. 52, Table 14 · p. 57
• The linear relationship between original and new test accuracy also appears on the first MatchedFrequency revision, before the final dataset-review step.Figure 8 shows models with 95% Clopper–Pearson confidence intervals and a 95% confidence region for the linear fit.Source: Figure 8 · p. 54

C.4.8 Ambiguous Class Examples

• Three class pairs illustrate ambiguous ImageNet class boundaries, which the authors note are ill-defined for other classes as well.The pairs are presented as illustrative examples only.Source: §C.4.8 · p. 69, Figure 18 · p. 76

C.4.5 Accuracy Plots for All ImageNet Test Sets

• MatchedFrequency and Threshold0.7 accuracies are below the original validation-set accuracies, while TopImages accuracies are higher.The plots show approximately linear relationships between original and new test accuracy; for top-5 on TopImages, the slope is 0.88 (95% confidence interval [0.81, 0.91]).Source: Figure 12 · p. 70
• In probit-scaled plots, all three datasets and both top-1 and top-5 accuracies fit well to linear relationships spanning about 60 percentage points.These plots include Fisher Vector models as well as the other models in the testbed.Source: Figure 13 · p. 71

C.4.6 Example Images

• Randomly selected images from three classes illustrate the original validation set and each of the three new ImageNet test sets.The displayed classes are Cypripedium calceolus, gyromitra, and mongoose.Source: Figure 14 · p. 72

C.4.7 Effect of Selection Frequency on Model Accuracy

• In the [0.4, 0.6) frequency bin, MatchedFrequency accuracy is about 20% below its full-set average and 30% below original validation accuracy.On the original validation set, that bin is 10–15% below the full-set accuracy; the new-versus-original gap is about 20% in middle bins and 5–10% in the lowest and highest bins.Source: Figure 15 · p. 73, Figure 16 · p. 74, Figure 17 · p. 75