PaperlayerView exact revision on arXiv

Source-linked AI summary

Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

arXiv:1806.00451v1cs.LGstat.ML

TL;DR

Benchmark-based progress leaves open whether CIFAR-10 classifiers generalize to unseen data. The authors create a new test set approximating the original distribution and report accuracy drops across models.

  • Problem

    Machine-learning progress relies on standard benchmarks, but there is limited explanation of whether benchmark improvements reliably generalize to unseen data.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The authors conduct a CIFAR-10 reproducibility study, repeating the original dataset-creation process to build a new test set that approximates the original data distribution.

    Source: §1.1 · p. 2, §2 · p. 3

  • Results

    Accuracy drops by 4–15% across all models and relative error increases by up to 3×; the results show no signs of adaptivity-based overfitting.

    Source: §6 · pp. 12–13

  • Takeaways & Limitations

    The findings indicate that current CIFAR-10 classifiers have difficulty generalizing to natural variations in image data.

    Source: §6 · p. 13

    The generalization argument assumes that the classifier does not depend on the test set.

    Source: §2 · p. 3

Abstract

from arXiv · show

Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images. Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even minute natural variations in the data distribution.

1 Introduction

Repeated use of standard test sets can undermine the assumption that classifiers are independent of test data. In a CIFAR-10 reproducibility study, accuracy fell on a new test set, although relative improvements often transferred.

1 Introduction

  • Repeated test-set use for tuning and comparison undermines classifier independence from the test set, risking models that perform well there but fail to generalize to new data [1].Because truly new data from the same distribution are limited, researchers commonly reuse test sets across model development and publications.Source: §1 · pp. 1–2

1.1 A Reproducibility Study on CIFAR-10

  • The study evaluates 30 image-classification models on about 2,000 newly collected images, matching the new test set’s subclass distribution to CIFAR-10.CIFAR-10 was chosen because its transparent creation process makes it well suited to a reproducibility study after nearly 10 years of intense research.Source: §1.1 · p. 2
  • VGG and ResNet architectures [7] [18] fall from 93% accuracy on the original test set to about 85% on the new test set, while original-test performance remains predictive.Even small incremental improvements on CIFAR-10 often transfer to truly held-out data.Source: §1.1 · p. 2
  • Retuning standard hyperparameters recovers only about 0.6% accuracy, and further experiments explain some—but not all—of the gap.A significant gap remains between performance on the original and new test sets.Source: §1.1 · p. 2
  • The recent Shake-Shake network with Cutout [3] [4] remains the top model, and its advantage over a standard ResNet rises from 4% to 8% on the new test set.The authors interpret this as evidence that extended adaptation to the CIFAR-10 test set has not caused stagnation.Source: §1.1 · p. 2
  • Despite only a minute, benign distribution shift—not adversarial or from a different data source—widely used models lose accuracy, with VGG and ResNet’s drop corresponding to multiple years of CIFAR-10 progress.The authors argue that even benign distribution shift challenges claims that current classifiers generalize.Source: §1.1 · p. 2

2 Formal Setup

The paper formalizes generalization as minimizing population loss under an unknown true data distribution, using test-set performance as a proxy when the classifier does not depend on the test set. Because exact distribution matching is difficult, it approximates the original distribution by repeating the dataset creation process.

2 Formal Setup

  • The goal is to find a classifier that minimizes population loss under the true distribution D over labeled examples.Because D is generally unknown, classifier performance is measured using a test set drawn from D.Source: §2 · p. 3
  • A sufficiently large test set approximates population loss when the classifier does not depend on that test set.This independence condition is the assumption that supports interpreting test performance as generalization rather than memorization.Source: §2 · p. 3
  • A new test set drawn from the same distribution should yield accuracy within confidence intervals of the original test set.The expected variation is due to inherent sampling error.Source: §2 · p. 3
  • Because the exact data distribution is hard to define, the paper approximates it by repeating the original dataset creation process.This may not produce an i.i.d. draw from the original distribution, but is presented as a close approximation; an ideal design would randomly split a larger dataset into training and test sets.Source: §2 · p. 3

3 Dataset Creation Methodology

The authors construct a new CIFAR-10 test set from Tiny Images to assess generalization on unseen images, aiming to minimize distribution shift by matching the original set’s keyword composition.

3 Dataset Creation Methodology

  • The authors chose CIFAR-10 because its documented creation from Tiny Images enables distribution-shift minimization, and its varied models have public code independent of the new test set.CIFAR-10 was also widely used and remained an active research benchmark.Source: §3 · pp. 3–4

3.1 Background

  • CIFAR-10 is a cleanly labeled, class-balanced subset of Tiny Images, created through student annotation, label verification, and ℓ2-nearest-neighbor duplicate removal.It contains 10 classes with 6,000 images per class, split into 50,000 training and 10,000 test images.Source: §3.1 · p. 4

3.2 Building the New Test Set

  • Matching Tiny Images keywords to CIFAR-10 preserves its sub-class distribution; the 25 most common keywords per class account for more than 95% of the dataset.The authors selected frequent keywords to avoid introducing infrequent sub-classes that classifiers had little incentive to learn.Source: §3.2 · pp. 4–5
  • Two authors divided image selection and review to mirror CIFAR-10’s collection process, applying its labeling guidelines and removing candidates deemed unclear or nonconforming.The target of 2,000 images was considered sufficient for an accuracy confidence interval of about ±1% at 90% accuracy, while smaller size also reduced pressure to include questionable images.Source: §3.2 · p. 5
  • The authors filtered near-duplicates within the new set and against CIFAR-10, manually reviewing each image’s top 10 neighbors, and kept collection separate from classifier evaluation.They re-sampled keywords until duplicate removal converged to the final dataset, and did not run classifiers during data collection.Source: §3.2 · p. 6

4 Model Performance Results

The evaluated classifiers span original-test accuracies of about 80% to 97%. All lose accuracy on the new test set, while their relative ranking changes little and original and new accuracy are linearly related.

4.1 A Significant Drop in Accuracy

  • About 8%: VGG and ResNet accuracy falls from around 93% to around 85%, while shake_shake_64d_cutout falls about 4%, from 97% to 93%.All models lose accuracy, with larger absolute gaps for models that perform worse on the original test set; no model is a clear outlier.Source: §4.1 · p. 7, Table 1 · p. 8
  • 3×: DARC, shake_shake_32d, and resnext_29_4x64d show this relative error increase, while VGG, AlexNet, and ResNet increase by 1.7× to 2.3×.Models with higher original accuracy tend to have larger relative error increases.Source: §4.1 · p. 7

4.2 Few Changes in the Relative Order

  • Few models change overall ranking, and a least-squares linear function explains the relationship between original and new accuracy well.Adding Cutout [3] to shake_shake_64d adds 0.12% on the original test set and about 1.5% on the new set; for wide_resnet_28_10, the gains are about 1% and 2.2%, respectively. Increasing ResNet width rather than depth also provides larger benefits on the new test set.Source: §4.2 · p. 7, Figure 2 · p. 9

4.3 A Model for the Linear Fit

  • A mixture of easy and harder subpopulations, with different proportions in the two test sets, produces a linear relationship between original and new accuracy.The authors present this as an illustration, not a ground-truth explanation: the test sets have more complex subpopulation compositions, and small distribution shifts can still produce accuracy sensitivities even when classifier ordering remains constant.Source: §4.3 · pp. 7, 9, 10

5 Explaining the Gap

The section tests statistical error, near-duplicates, hyperparameter tuning, hard images, distribution shift, and cross-validation as explanations for the accuracy gap. The reported evidence rules out statistical error as its sole cause and limits the contribution of near-duplicates, while the other experiments find only small tuning gains, recognizable hard images, little benefit from added training data, and cross-validation accuracies similar to the original test set.

5.1 Statistical error

  • The new test set’s 95% confidence interval is about ±1.2%, versus roughly ±0.6% at 90% and ±0.3% at 97% original-test accuracy, ruling out statistical error as the sole explanation.Source: §5.1 · p. 10

5.2 Differences in near-duplicate removal

  • Near-duplicates could explain at most 1% of the observed accuracy difference: about 800 original test images qualified, and most classifiers scored 99%–100% on them.The authors identified near-duplicates by visually inspecting top-10 neighbors under ℓ2 distance and SSIM, counting same-object, same-pose images despite small scale, color, stretch, or position changes.Source: §5.2 · p. 10

5.3 Hyperparameter tuning

  • Hyperparameter grid search over learning rate, dropout, and weight decay found no significantly better new-test accuracy; the largest improvement was from 85.25% to 85.84%.Dropout was tuned by scaling its amount across layers while preserving the layer-to-layer ratios.Source: §5.3 · pp. 10–11

5.4 Inspecting hard images

  • Inspection of images that most models misclassified found that all new images were natural and recognizable to humans.Figure 3 in Appendix B shows examples of hard images that no model correctly classified.Source: §5.4 · p. 11

5.5 Training on part of our new test set

  • Retraining VGG16 with 1,010 new-test images yielded 85.06% and 85.36% on held-out images, versus 84.9% without extra data, providing evidence against a large distribution shift.The experiment used class-balanced splits and 1,011 held-out images, and was repeated with two different splits.Source: §5.5 · p. 11

5.6 Cross-validation

  • Accuracies across the five cross-validation splits did not vary much from original-test accuracies for the three retrained models.The held-out splits were class-balanced training-set partitions, while each corresponding training set combined the other four partitions with the original test set; accuracies are reported in Table 2.Source: §5.6 · pp. 11–12, Table 2 · p. 12

6 Discussion

The discussion argues that the results show no test-set-adaptivity overfitting, while a substantial accuracy drop under a small distribution shift reveals difficulty generalizing to natural image variations. It calls for broader evaluation on truly new data.

Overfitting

  • The paper distinguishes training-set overfitting, a train–test accuracy gap already present with usually 100% training accuracy, from test-set overfitting, a test-to-distribution accuracy gap linked to adaptivity.The authors consider test-set overfitting more important because machine learning aims to generalize to unseen data.Source: §6 · p. 12
  • Despite years of competitive adaptivity, the best models gained advantage on the new test set, with no signs of test-set overfitting; the authors support competition-based accuracy gains.They note that further replication is needed for a conclusive picture. They relate this to the Ladder analysis [1] and report no adaptivity overfitting even without its modifications.Source: §6 · p. 12

Distribution shift

  • A small distribution shift coincided with accuracy drops of 4–15% across all models and relative error-rate increases up to 3×, despite efforts to replicate CIFAR-10’s creation process.The authors characterize the shift as benign, not a specific data-generation change or an adversarial challenge, and conclude that classifiers struggle with natural image variations.Source: §6 · pp. 12–13

Future work

  • Future experiments should test competition’s resilience on other datasets and tasks, keep new test sets close to the original distribution, and investigate naturally occurring shifts and difficult sub-populations.The authors suggest a mixture model based on sub-population shifts, described in Section 4.3, as a starting point.Source: §6 · p. 13

Broader evaluation

  • The authors call for studies that collect insightful new data to evaluate existing algorithms, distinguishing statistical reproducibility on truly new data from computational reproducibility on the same test data.They note that open-source repositories contain many essentially pre-registered classifiers, enabling studies under established standards of statistically valid research.Source: §6 · p. 13

A Model Descriptions

This section inventories code repositories and default configurations for a broad set of CIFAR-10 deep models, including residual architectures, randomized features, architecture search, and shake-based models.

A Model Descriptions

  • The repository list covers AlexNet without augmentation and with augmentation; configurations are defaults, and models used one GPU unless otherwise noted.The repository table notes that repositories are hosted on GitHub except darc.Source: Table 3 · p. 16, A Model Descriptions · p. 16
  • Other configurations include a regularized depth-29 ResNeXt, DenseNet-BC-100-12, a CIFAR-10 neural architecture search model, and depth-110 PyramidNets [11] [20] [10] [23].The DenseNet uses batch size 32, initial learning rate 0.05, bottleneck blocks, growth rate 12, and compression rate 0.5; PyramidNet variants use α = 270 or 84.Source: A Model Descriptions · pp. 16–17
  • Four random-feature models use one-layer convolutional networks with 32k or 256k patch-sampled filters, with and without horizontal-flip augmentation.Each uses patch size 6, pool size 15, and pool stride 16.Source: A Model Descriptions · p. 17
  • The ResNet configurations use basic-block depths 32, 44, 56, and 110, while ResNet-preact configurations use basic depth 110 and bottleneck depth 164 [7] [8].A TensorFlow implementation of a basic-block ResNet-preact at depth 32 is also listed [8].Source: A Model Descriptions · p. 17
  • Shake-drop uses PyramidNet, while Shake-shake variants use depth 26 and base channels 32, 64, or 96, including a 64-channel cutout configuration [21] [4] [3].The Shake-drop model is trained for 300 epochs on four GPUs; the cutout variant uses cutout size 16.Source: A Model Descriptions · p. 17
  • The listed VGG models use batch normalization, and wide residual networks include widening-factor-10 variants, including a depth-28 cutout model [18] [22] [3].The VGG configurations have depths 16 and 15, with the latter using 64 channels; the cutout wide residual network uses cutout size 16.Source: A Model Descriptions · pp. 17–18

B Details for “Explaining the Gap" Experiments · Inspecting Hard Images

Figure 3 presents examples of new-test-set images that no model correctly classified, with each image’s true label and the majority of models’ predicted label. The examples show several kinds of class confusion.

Inspecting Hard Images

C Error ratios

The section describes a comparison of model error on the original and new CIFAR-10 test sets using the ratio between them.

C Error ratios

  • Table 4 covers model error on both test sets and the ratio of new-test-set error to original-test-set error.Source: Table 4 · p. 20

D Class-balanced Test Set

The new test set was not precisely class-balanced because keyword coverage missed ship-skewed images; the authors created a 2,000-image class-balanced version that achieved slightly higher model accuracy.

D Class-balanced Test Set

  • The top 25 keywords captured approximately 95% of the dataset, while the uncaptured 5% was heavily ship-skewed, leaving ships at only 8%.Source: D Class-balanced Test Set · p. 21
  • The authors constructed an exactly 2,000-image class-balanced test set by allocating images across the top 50 keywords in each class using fractional counts and rounding-error adjustments.They removed images from keywords with the largest rounding gaps until reaching 2,000 images.Source: D Class-balanced Test Set · p. 21
  • The class-balanced test set produced slightly higher model accuracy than the new test set described in the main paper.Table 5 compares accuracy on the original CIFAR-10 test set and the class-balanced new test set.Source: D Class-balanced Test Set · p. 21, Table 5 · p. 22

E Keywords

Table 6 presents the distribution of the top 25 keywords in each class for the new and original test sets.

E Keywords

  • Table 6 reports the top 25 keywords in each class for the new and original test sets.Source: Table 6 · p. 23

Related papers

Loading 1806.00451v1…