Source-linked AI summary
Are we done with ImageNet?
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, Aäron van den Oord
TL;DR
The paper asks whether recent ImageNet gains reflect meaningful generalization or overfitting to labeling artifacts. It collects ReaL annotations and uses them to reassess classifiers, finding weaker recent gains and that original labels no longer best predict human judgments. The procedure removes many labeling errors while preserving ImageNet’s usefulness for visual-recognition evaluation.
Problem
The paper investigates whether recent ImageNet progress still represents meaningful generalization or increasingly reflects artifacts in the benchmark’s labeling procedure.
Method
The authors collect independently reassessed human annotations for ImageNet validation images and evaluate classifiers with ReaL accuracy, which accepts multiple plausible labels.
Results
Recent models’ ImageNet-to-ReaL progress association weakened, and recent models surpassed the original ImageNet labels in ReaL accuracy.
Takeaways & Limitations
ImageNet’s original labels may be nearing the end of their usefulness, while ImageNet and ReaL can benchmark visual-recognition progress for the foreseeable future.
Takeaways & Limitations
ReaL accuracy evaluates only whether the single most confident prediction is among acceptable labels, not whether all top predictions are acceptable.
Abstract
from arXiv · showhide
Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.
1 Introduction
The paper asks whether recent ImageNet gains still reflect generalization, introducing ReaL annotations to reassess progress. Early ImageNet improvements translated strongly to ReaL, but this association weakened for newer models, which could outperform the original labels against human preferences.
- ImageNet has served as a central, difficult benchmark whose advances often generalized to other tasks and domains.
- The paper develops ReaL annotations from human judgments to test whether recent ImageNet progress remains general.
- Early ImageNet progress translated into corresponding ReaL gains, but this association became significantly weaker for recent models.
- Recent high-performing models systematically outperformed the original ImageNet labels as predictors of human preferences.
- The paper links some original-metric progress to overfitting the idiosyncrasies of ImageNet’s labeling pipeline.
2 What is wrong with ImageNet labels?
The original ImageNet labeling procedure can misrepresent image content because it assigns one label, introduces systematic inaccuracies, and preserves arbitrary class distinctions. These shortcomings motivate labels that let annotators compare multiple plausible descriptions.
- Many images contain multiple equally prominent objects, but ImageNet assigns only one label and can penalize other correct descriptions.
- ImageNet’s image-collection procedure can produce systematic inaccuracies when labels are judged in isolation rather than against other classes.
- Some ImageNet classes distinguish essentially duplicate, visually indistinguishable categories such as “sunglasses” and “sunglass”.
- The proposed approach allows multiple annotations from simultaneously presented label proposals to reduce ambiguity and produce a more meaningful performance metric.
3 Relabeling the ImageNet validation set
The relabeling procedure generates a compact, high-coverage set of candidate labels from diverse models and has human raters evaluate them together. Dawid–Skene aggregation then converts these assessments into ReaL labels and supports a metric that accepts multiple plausible answers.
- The procedure seeks to let annotators compare diverse candidate labels while keeping proposal sets small enough for robust annotation.
- Six selected models generated proposals with 97.1% recall and 28.3% precision, reducing the average proposals from 13 to 7.4 per image.
- The system retains agreed original labels and reduces the validation images requiring annotation from 50 000 to 24 889.
- Five annotators assess each image and candidate label as present, absent, or uncertain using a 95%-confidence response rule.
- Dawid–Skene combines rater assessments into binary label decisions, with the original animal label added as a virtual sixth rater.
- ReaL accuracy counts a model’s top-1 prediction as correct when it belongs to the image’s set of acceptable labels.
4 Re-evaluating the state of the art
ReaL accuracy reveals that ImageNet gains have weakened as a measure of progress, while recent models increasingly exploit biases in the original labeling procedure. ReaL remains a useful default metric, with secondary predictions offering a stricter extension.
- The slope relating ImageNet accuracy to ReaL accuracy fell from 0.86 for earlier models to 0.51 for recent models.The difference in slopes was significant (p < 0.001).
- Several recent models surpass the ReaL accuracy achieved by the original ImageNet labels, suggesting that ImageNet accuracy may be nearing the end of its evaluative utility.
- The three-model ensemble reaches 89.0% original ImageNet accuracy and 91.20% ReaL accuracy.It further outperforms ImageNet labels according to human preferences.
- 4.1 Beyond single label predictions: Models’ second- and third-prediction accuracies are significantly below their first, yet remain far above chance and strongly correlate with primary accuracy.ReaL accuracy can therefore serve as a default metric, with secondary predictions added if top-1 performance becomes near-perfect.
- 4.2 An analysis of co-occuring classes: For the desktop computer category, the unbiased oracle’s approximately 30% ceiling contrasts with 71.1% accuracy from Noisy Student and Assemble-ResNet152.More than 75% of desktop-computer images also contain a monitor, computer keyboard, and desk.
- 4.2 An analysis of co-occuring classes: Recent top-performing models dominate the unbiased-oracle curve, indicating that ImageNet progress partly exploits biases in its labeling procedure.The authors suggest this may explain why gains only partially transfer to ReaL accuracy.
5 Analyzing the remaining mistakes
Human review shows genuine progress on both benchmarks, but ImageNet labels produce substantially more false negatives than ReaL labels. Fine-grained classes remain difficult to assess and may require expert annotation.
- The highest-performing models have approximately 11% error under ImageNet labels and 9% under ReaL labels.
- The follow-up study shows raters images, model predictions, correct labels, and example class images before asking them to classify the prediction as a mistake, correct, or undecidable.
- As model ImageNet accuracy increases, the number of human-rated “clear mistakes” decreases, indicating genuine progress on both benchmarks.
- Human raters deem a significant fraction of ImageNet-defined mistakes correct, and this occurs much more often than for ReaL-defined mistakes.This indicates that ReaL labels remedied a large portion of false negatives in the original labels.
- A non-negligible fraction of mistakes is judged “undecidable,” especially for fine-grained classes that non-experts find difficult or impossible to distinguish.ImageNet and ReaL give similar assessments for these classes, suggesting expert annotation may be necessary.
6 Improving ImageNet training
The authors address ImageNet label noise with a multi-label sigmoid objective and cleaned training data, then evaluate both under longer schedules. Each modification improves accuracy, their combination performs best, and the gains are especially large for longer training.
- Motivation: Single-label annotations can misrepresent images containing multiple prominent objects, motivating methods that address ImageNet label noise.The section identifies this as a shortcoming of the labeling procedure and expects related noise in training labels.
- Method: Sigmoid cross-entropy treats multi-way classification as independent binary problems, allowing multiple non-exclusive predictions per image.Unlike softmax, the objective does not enforce mutually exclusive predictions.
- Results: Clean labels and sigmoid loss each consistently improve accuracy, while combining them produces further improvements across ResNet architectures and training settings.The evaluation compares softmax versus sigmoid loss, cleaned versus full training data, and standard versus longer schedules.
- Longer training: Longer training with the standard softmax loss and full ImageNet data overfits, whereas clean labels with sigmoid loss produce large gains.The authors connect the benefit of the proposed modifications to the presence of noisy data during extended training.
- Results: 78.5% top-1 accuracy is achieved by a 900-epoch ResNet-50 trained with sigmoid loss on clean data, improving 2.5% over the standard ResNet-50 baseline (76.0%).The corresponding ReaL accuracy gain is 1.6%, while the 900-epoch baseline loses 0.9% in ReaL accuracy.
7 Conclusion
The conclusion finds that ImageNet progress has become less closely associated with progress on independently collected ReaL labels, while ReaL corrects many original labeling mistakes. Two training modifications yield gains and support continued use of ImageNet with ReaL labels.
- Conclusion: The association between ImageNet progress and ReaL progress has weakened over time, although it remains significant.
- Conclusion: Recent models surpass the original ImageNet labels in ReaL accuracy, indicating that the original labels may be nearing the end of their usefulness.
- Conclusion: ReaL labels remove more than half of the ImageNet labeling mistakes and provide a superior estimate of model accuracy.
- Conclusion: The two proposed modifications produce relatively large empirical gains and suggest label noise may limit longer training schedules.
- Conclusion: ImageNet and ReaL labels can benchmark progress in visual recognition for the foreseeable future.
A Details on proposal generation
The proposal-generation procedure pools predictions from multiple models to assemble candidate labels, then uses expert annotations and subset selection to improve proposal precision while retaining high recall.
- Proposal generation: The algorithm collects high-logit or high-probability image-label pairs across models, removes singly proposed pairs, and adds every model’s top-1 prediction.
- Expert filtering: Five computer vision experts label a subset using the proposals, enabling selection of models with 97.08% recall and precision improved from 14.55% to 28.33%.The selected models reduce the labeling burden before proposals are generated for the full validation set.
B Improving ImageNet training
The training study compares sigmoid loss and clean labels against the standard ImageNet setup across training epochs. Both changes improve accuracy, with their combination performing best and showing stronger benefits under longer schedules.
- Training comparison: Using sigmoid loss and clean labels together achieves the best performance in the reported ImageNet accuracy comparisons.
- Longer training: The improvement from the proposed method is more pronounced with longer training schedules.
- Training comparison: Either sigmoid loss or a clean label set consistently improves top-1 accuracy over the standard softmax baseline.Figure 8 reports median results across three runs, with additional runs shown using lighter curves.
C ReaL accuracies of all models in Fig. 4
Table 5 reports ReaL and original ILSVRC-2012 top-1 accuracies for all models used in Fig. 4. The models are ordered by ReaL accuracy.
- Table 5 compares ReaL and original ILSVRC-2012 top-1 accuracies, both reported in percent.
- The table includes all models used for Fig. 4 in the main paper.
- Models are sorted by their ReaL accuracies.