Source-linked AI summary
Do CIFAR-10 Classifiers Generalize to CIFAR-10?
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar
TL;DR
The paper asks how reliably benchmark accuracy measures machine-learning progress when the same test sets are reused for model selection. It constructs a new, closely matched CIFAR-10 test set and evaluates existing classifiers, finding substantial accuracy drops while newer models retain better performance. The results point to brittleness under small natural distribution shifts rather than adaptivity-based overfitting, though the constructed test set is not guaranteed to be i.i.d.
Problem
Repeated reuse of standard test sets makes it unclear how reliably benchmark accuracy measures generalization and progress in machine learning.
Method
The study constructs a new CIFAR-10 test set by repeating the dataset creation process and evaluates classifiers that are independent of this new test set.
Results
Accuracy drops by 4 - 15% across models, while newer high-performing models retain an increased advantage on the new test set.
Takeaways & Limitations
Current CIFAR-10 classifiers have difficulty generalizing to natural variations in image data, despite no observed adaptivity-based overfitting.
Takeaways & Limitations
The reproduced test set is a close approximation to the original data-generating distribution and is not necessarily an i.i.d. draw.
Abstract
from arXiv · showhide
Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images. Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even minute natural variations in the data distribution.
1 Introduction
The paper questions whether benchmark-based progress reliably measures generalization, because models and hyperparameters are repeatedly selected using the same test sets. It addresses this by constructing a new CIFAR-10 test set and evaluating existing classifiers on truly unseen images.
- 1 Introduction: Repeatedly reusing standard test sets can undermine classifier independence from the test data and threaten generalization to new data.The practice includes tuning within publications and building on prior results across publications.
- 1 Introduction: The study creates a new reproducibility test for contemporary CIFAR-10 classifiers, whose transparent creation process supports approximating the original data distribution.CIFAR-10 was selected because it has a transparent creation process and has been intensively studied.
- 1 Introduction: The new test set matches the original subclass distribution and contains about 2000 new images used to evaluate 30 image classification models.This design aims to test performance on truly unseen data while remaining close to the original dataset.
- 1 Introduction: VGG and ResNet accuracies fall from 93% on the original test set to about 85% on the new test set.The study reports this drop as one of two overarching phenomena observed across the evaluated models.
- 1 Introduction: Retuning standard hyperparameters improves new-test accuracy by only about 0.6%, leaving a significant accuracy gap unexplained.The hyperparameter experiment was motivated by the discrepancy between original and new accuracies.
- 1 Introduction: Despite years of CIFAR-10 test-set adaptation, the top model’s advantage over a standard ResNet increases from 4% to 8% on the new test set.The authors interpret this pattern as showing no stagnation and resilience to overfitting through test-set adaptation.
2 Formal Setup
The formal setup distinguishes population loss from test-set loss when evaluating a classifier on unseen data. Its key assumption is that the classifier does not depend on the test set, while the study approximates the data-generating distribution by repeating dataset creation.
- 2 Formal Setup: The setup posits a true distribution D over labeled examples and seeks a model f̂ that minimizes population loss.The distribution is generally unknown, motivating evaluation through sampled test data.
- 2 Formal Setup: Because D is unknown, classifier performance is measured on a test set Dtest drawn from D.This provides an empirical substitute for directly measuring population loss.
- 2 Formal Setup: Test-set loss approximates population loss when Dtest is sufficiently large and the classifier f̂ does not depend on Dtest.This independence condition is the core assumption supporting claims of generalization rather than memorization.
- 2 Formal Setup: A truly i.i.d. new test set would ideally come from randomly splitting a larger initial dataset into training and test subsets.The paper notes that this exact setup is usually unavailable for reproducing accuracy numbers.
- 2 Formal Setup: The study instead repeats the original dataset creation process to mimic D, producing a close approximation rather than necessarily an i.i.d. draw.This procedure is used because the exact original data-generating distribution is difficult to define and reproduce.
3 Dataset Creation Methodology
The study constructs a new CIFAR-10 test set designed to closely match the original dataset while containing truly unseen images. Its methodology matches subclass distributions, filters candidates for quality and duplicates, and separates collection from classifier evaluation.
- 3 Dataset Creation Methodology: CIFAR-10 is selected because its transparent creation process and fine-grained Tiny Images labels enable minimized distribution shift.The dataset is a cleanly labeled subset of Tiny Images, whose keyword structure supports matching the original data more closely.
- 3.1 Background: The dataset contains ten balanced classes with 50,000 training images and 10,000 original test images.The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
- 3.2 Building the New Test Set: The new test set is designed to match CIFAR-10’s distribution while using images unseen during evaluation.The authors collect images from the same Tiny Images keywords, remove near-duplicates, and keep data collection separate from classifier evaluation.
- 3.2 Building the New Test Set: The authors identify the 25 most common Tiny Images keywords per class, covering more than 95% of CIFAR-10.This avoids bias toward infrequent keywords and preserves distinctive subclass composition, such as stealth_bomber and stealth_fighter within airplanes.
- 3.2 Building the New Test Set: A two-author curation process selects and reviews roughly 9,000 candidates before assembling the new test set.Author A selects suitable images, while Author B removes unclear or instruction-inconsistent images; one keyword remained below the target because it comprised less than 0.3% of CIFAR-10.
- 3.2 Building the New Test Set: The final sample matches CIFAR-10’s keyword-level distribution and removes near-duplicates within the new set and against the original train and test sets.The new test set contains 2,000 images, chosen because this size gives approximately a ±1% confidence interval for 90% accuracy at 95% confidence.
4 Model Performance Results
Across a broad range of classifiers, accuracy drops on the new CIFAR-10 test set while model rankings largely persist. The original-to-new relationship is approximately linear, with some techniques gaining more on the new test set.
- 4.1 A Significant Drop in Accuracy: About 8%: VGG and ResNet accuracy falls from around 93% on the original test set to around 85% on the new set.
- 4.1 A Significant Drop in Accuracy: Roughly 4%: shake_shake_64d_cutout drops from 97% original accuracy to 93% on the new test set.
- 4.1 A Significant Drop in Accuracy: Higher-original-accuracy models generally experience smaller absolute gaps, while relative error increases reach 3× for some models and 1.7×–2.3× for simpler architectures.
- 4.2 Few Changes in the Relative Order: Model rankings change little, and a least-squares linear fit explains the relationship between original and new accuracy.
- 4.2 Few Changes in the Relative Order: Cutout improves shake_shake_64d accuracy by 0.12% originally versus about 1.5% on the new test set, showing technique-specific gains under distribution shift.
- 4.3 A Model for the Linear Fit: The two-subpopulation mixture is illustrative rather than ground truth, but it demonstrates how small distribution shifts can produce linear accuracy changes while preserving classifier ordering.
5 Explaining the Gap
The authors test several explanations for the accuracy gap between the original and new CIFAR-10 test sets. Statistical error, near-duplicate removal, hyperparameter retuning, unusually hard images, and major distribution shift explain little of the discrepancy.
- Statistical fluctuations: Statistical error cannot explain the gap because the new test set’s 95% confidence interval is about ±1.2%.The original test set’s interval is roughly ±0.6% at 90% accuracy and ±0.3% at 97% accuracy.
- Near-duplicate removal: Near-duplicate removal explains at most 1% of the observed accuracy difference.About 800 original test images were identified as near-duplicates, representing 8% of the test set; classifiers achieved 99%–100% accuracy on them.
- Hyperparameter tuning: Hyperparameter retuning produced no significant improvement on the new test set.A VGG grid search over learning rate, dropout, and weight decay increased accuracy from 85.25% to 85.84% at most.
- Hard images: The new test set includes natural images recognizable to humans, despite some images being hard for the models.Figure 3 shows images that no model correctly classified, with majority-model predictions differing from the true labels.
- Distribution shift: Training on 1010 new images improved held-out accuracy only from 84.9% to 85.06% and 85.36%, providing evidence against a large distribution shift.The experiment used class-balanced splits and retrained vgg16_keras on the original training set plus sampled new-test images.
- Cross-validation: Cross-validation accuracies did not vary much from original-test accuracies for the evaluated models.The authors created five class-balanced validation datasets and retrained vgg_15_BN_64, wide_resnet_28_10, and shake_shake_64d_cutout.
6 Discussion
The discussion distinguishes training-set overfitting from test-set overfitting and finds no evidence that years of CIFAR-10 adaptivity caused the observed gap. Instead, the results point to a small but consequential distribution shift and motivate evaluation on new data.
- Overfitting: Training-set overfitting is measured by the difference between training and test accuracy, and the models usually reach 100% training accuracy.
- Overfitting: Test-set overfitting is the gap between test accuracy and accuracy on the underlying distribution caused by adapting design choices to the test set.Under this definition, test accuracy may lose validity as a measure of performance on truly unseen data.
- Overfitting: The experiments show no signs of adaptivity-based overfitting on CIFAR-10 despite multiple years of competitive model development.The best models on the new test set increase their advantage over established baselines.
- Distribution shift: The authors attribute the accuracy gap to a small distribution shift between the original CIFAR-10 dataset and the new test set.All models lose 4–15% accuracy, and relative error increases by up to 3×, despite the benign, non-adversarial shift.
- Future work: The authors recommend collecting insightful new data to evaluate whether improvements on existing benchmarks generalize broadly.
A Model Descriptions
The appendix catalogs the deep models and their implementation configurations used in the study. It includes architectures spanning random features, convolutional networks, residual networks, and architecture-search models.
- Repository information: Table 3 lists code repositories for the deep models, with darc as the exception to the GitHub-hosting convention.
- Other models: Other listed models include AlexNet, DenseNet, NAS, PyramidNet, VGG, and DARC configurations.
- Random-feature models: The catalog includes random-feature convolutional networks with 32k or 256k filters, with and without horizontal-flip augmentation.
- Residual networks: Residual-model entries cover ResNet, ResNet-preact, ResNeXt, and wide residual network variants with differing depths, widths, and cardinalities.
- Regularized architectures: The catalog includes Shake-Drop, Shake-Shake, and Cutout variants with specified depths, channel counts, schedulers, and augmentation settings.
B Details for “Explaining the Gap" Experiments
The appendix illustrates images that no model correctly classified in the new test set. True labels and majority-model predictions show recurring confusions among visually related CIFAR-10 classes.
- Figure 3: Several automobile images are predicted as trucks or airplanes, while multiple cat images are predicted as deer.
- Figure 3: The examples include automobile–truck, ship–automobile, bird–frog, horse–frog, cat–deer, and airplane–ship confusions.
- Figure 3: Figure 3 pairs each hard image’s true class with the label predicted by the majority of models.
C Error ratios
Table 4 compares model error on the original CIFAR-10 test set with error on the new test set, including their ratio.
- Table 4 reports model error on the original CIFAR-10 test set and the new test set.
- The table also gives the error ratio between the new and original test sets.
D Class-balanced Test Set
The class-balanced new test set is evaluated against the original CIFAR-10 test set, with accuracy gaps and differences from the main-paper new test set reported.
- The new test set was not precisely class-balanced and contained only 8% ships.
- A class-balanced version of the new test set was created with exactly 2000 images.
- The class-balanced set uses keyword-based image counts adjusted by rounding errors to reach 2000 images.
- Table 5 compares accuracy on the original and class-balanced test sets, reporting the gap and ΔAcc. relative to the main-paper new test set.
E Keywords
Table 6 reports the distribution of the top 25 keywords in each class for the new and original test sets.
- Table 6 compares top-25 keyword distributions across classes and between the new and original test sets.