PaperlayerView exact revision on arXiv

Source-linked AI summary

From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, Aleksander Madry

arXiv:2005.11295v1cs.CVcs.LGstat.ML

TL;DR

The paper asks how scalable, crowd-sourced dataset construction affects alignment between ImageNet and the real-world object-recognition task it represents. Through refined human annotations and studies of model predictions, it finds pervasive annotation deviations that alter both what models learn and how progress is measured.

  • Problem

    The paper examines whether widely used benchmarks align with the real-world tasks they approximate, despite scalable collection methods that can reduce annotation fidelity.

  • Method

    The authors bootstrap candidate labels from model predictions, use human CONTAINS and CLASSIFY studies, and aggregate responses into fine-grained ImageNet annotations.

  • Results

    ImageNet contains pervasive multi-label and ambiguous annotations, while top-1 accuracy can underestimate valid predictions and models can exploit ImageNet-specific features that may not generalize.

  • Takeaways & Limitations

    Accuracy against a single dataset label gives an imperfect view of object-recognition performance, motivating evaluation that accounts for benchmark-task misalignment.

  • Takeaways & Limitations

    Scalable annotation conflicts with faithfully capturing real-world complexity, and the refined annotations still rely on non-expert annotators who make occasional errors.

Abstract

from arXiv · show

Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the consequences of employing such a pipeline, focusing on the popular ImageNet dataset. We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset---including the introduction of biases that state-of-the-art models exploit. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for. Finally, our findings emphasize the need to augment our current model training and evaluation toolkit to take such misalignments into account. To facilitate further research, we release our refined ImageNet annotations at https://github.com/MadryLab/ImageNetMultiLabel.

1 Introduction

This paper asks how closely widely used vision benchmarks align with the real-world tasks they approximate, focusing on ImageNet. It develops finer-grained human annotations and shows that scalable collection choices can misalign labels, model learning, and benchmark-based progress.

  • The paper investigates whether existing benchmarks align with their motivating real-world tasks, using ImageNet as its focus.
  • Scalable collection methods can sacrifice faithfulness, producing annotations that are ambiguous, incorrect, or otherwise misaligned with the modeled task.
  • The authors develop fine-grained annotations through large-scale human studies to quantify how object-recognition benchmarks diverge from underlying ground truth.
  • More than one fifth of ImageNet images contain objects from multiple classes, and labels often do not identify the human-judged main object.
  • Annotators often validate mutually exclusive labels, leaving automated image retrieval to influence which label an image receives.
  • More accurate ImageNet models tend to be more human-aligned, yet non-experts cannot reliably distinguish their predictions from ImageNet labels.

2 A Closer Look at the ImageNet Dataset

ImageNet combines automated image retrieval with crowd-sourced filtering, but its validation task checks proposed labels in isolation rather than asking annotators to choose among possible labels. This design can admit multi-object images and systematic label biases.

  • ImageNet is a large-scale dataset built through automated collection and crowd-sourced filtering, using 1,000 classes and more than a million images.
  • Image retrieval assigns each image a proposed label from the WordNet node used in the search query.
  • The CONTAINS task asks annotators to select images containing a specified class, without comparing that proposed label against other possible classes.
  • Validating labels in isolation can include multiple objects or assign a label that does not match the image's most prominent object.
  • Annotators lacking knowledge of the full class set may accept visually similar objects, allowing automated retrieval errors to persist.
  • The paper uses a refined annotation framework to investigate these discrepancies and their effects on ImageNet-trained models.

3 From Label Validation to Image Classification

The authors replace single-label validation with a two-phase pipeline that narrows candidate labels and then asks annotators to classify every distinct object. Aggregated responses yield finer-grained annotations while retaining practical task size.

  • Because choosing among all 1,000 ImageNet classes is infeasible for untrained annotators, the proposed pipeline first narrows the candidate labels.
  • The two phases obtain potentially relevant candidates and then use CLASSIFY to select labels for each distinct object.
  • The analysis examines 10,000 validation images, using 10 randomly selected images per class.
  • 3.1 Obtaining candidate labels: Candidate labels combine top-5 predictions from 10 models with the existing label, then retain labels with high CONTAINS selection frequency.
  • 3.2 Image classification via the CLASSIFY task: CLASSIFY asks annotators to identify all object labels and the main-object label, selecting one label per distinct object before responses are aggregated.
  • 3.2 Image classification via the CLASSIFY task: The aggregation estimates object counts and main labels, partitions differing selections into distinct objects, and assigns each object its most frequently selected label.
  • 3.2 Image classification via the CLASSIFY task: The resulting annotations are finer-grained than ImageNet's original labels but can still contain errors from non-expert annotators and inherently ambiguous images.

4 Quantifying the Benchmark-Task Alignment of ImageNet

Human studies reveal that ImageNet’s single-label benchmark is systematically misaligned with the underlying object-recognition task. Multi-object images, ambiguous validation, and pipeline-sensitive annotation choices affect both measured accuracy and the features models exploit.

  • 4.1 Multi-object images: More than one fifth of ImageNet images contain objects from multiple classes, reflecting systematic overlap in class occurrences rather than only irrelevant clutter.Some class pairs consistently co-occur, indicating that nominally distinct ImageNet classes overlap in real-world contexts.
  • 4.1 Multi-object images: Top-1 accuracy drops by more than 10% on multi-object images across all models when evaluated against the single ImageNet label.Using multi-label accuracy, which counts a prediction correct when it matches any object label, makes the single- versus multi-object performance gap essentially disappear.
  • 4.1 Multi-object images: For nearly a third of multi-object images, annotators identify a main object different from the ImageNet label, yet models still predict the ImageNet label accurately.This pattern indicates that models use features humans do not regard as salient, including ImageNet-specific object associations.
  • 4.2 Bias in label validation: Under the CONTAINS task, nearly 40% of images receive another label at least as often as the ImageNet label, even when annotators perceive a single object.Annotators may select as many as 10 classes for a single-object image, so validation can fail to elicit an available single ground-truth label.
  • 4.2 Bias in label validation: Choosing the main object among multiple labels substantially reduces annotator confusion compared with validating labels in isolation.This shows that annotation outcomes depend on the task setup, not only on annotator expertise.
  • 4.2 Bias in label validation: Ambiguous class pairs can limit state-of-the-art accuracy below 40% because models struggle to distinguish distributions that may have been mixed during automated retrieval.The authors also identify semantic similarity and occasional class-name or link overlaps as sources of confusion, motivating class selection that accounts for human confusion.

5 Beyond Test Accuracy: Human-In-The-Loop Model Evaluation

Human assessment reveals that ImageNet top-1 accuracy can misrepresent model quality because labels may be ambiguous or incomplete. More accurate models are increasingly human-aligned, but crowd-sourcing cannot easily distinguish meaningful progress from better matching ImageNet’s idiosyncrasies.

  • Human assessment: Human-based metrics assess whether predicted labels are present and whether they denote the image’s main object.Selection frequency accommodates multiple valid objects, while main-label accuracy tests agreement with human main-object annotations.
  • Human assessment: State-of-the-art predictions are nearly as likely as ImageNet labels to be judged present or to denote the main object.Annotators are almost equally likely to select model predictions and ImageNet labels along both axes.
  • Interpretation: Further accuracy gains may reflect either real object-recognition improvements or closer matching to the ImageNet distribution.The remaining top-1 gap is therefore not wholly inconsequential, but its real-world meaning is difficult to identify through crowd-sourcing alone.
  • Incorrect predictions: More accurate models make fewer mistakes that humans judge implausible, while many ImageNet-mismatching predictions remain valid labels.This makes single-label evaluation pessimistic for multi-object images and ambiguous classes.
  • Dataset collection: Using selection frequency alone to filter collected images can retain examples that remain challenging for models.High selection frequency with respect to the dataset label does not guarantee easy model recognition.
  • Model-human similarity: Human and model confusion patterns can be compared across 11 ImageNet superclasses and against superclass co-occurrence in images.The comparison uses main-object annotations and examines whether cross-superclass confusions correspond to objects that co-occur.

6 Related Work

Related work has examined ambiguous ImageNet classes, cluttered images, human performance, cross-dataset generalization, and worst-case robustness. These lines of work frame benchmark fidelity and generalization beyond narrow test distributions as established concerns.

  • Identifying ImageNet issues: Prior studies identify inherently ambiguous ImageNet classes and discuss cluttered images as a feature of real-world conditions.These works motivate examining annotation ambiguity and multi-object content in ImageNet.
  • Human performance on ImageNet: Human ImageNet accuracy is difficult to study rigorously because annotators must consider all 1000 potentially present classes.Earlier work includes trained-annotator, self-evaluation, and subset-based MTurk studies.
  • Generalization beyond the test set: Cross-dataset evaluation has been proposed to test whether models generalize beyond a narrow benchmark task.Prior work evaluates models on different datasets with similar class structures and studies reproduction of ImageNet validation.
  • Adversarial testing: Adversarial testing measures model performance under worst-case perturbations, including imperceptible pixel changes and natural corruptions.This literature complements benign distribution-shift evaluations with robustness-focused testing.

7 Conclusion

The paper finds that ImageNet annotations systematically diverge from real-world object-recognition ground truth, affecting both what models learn and how progress is measured. Human evaluation reveals greater alignment in model errors but cannot readily separate meaningful progress from benchmark-specific overfitting.

  • Conclusion: ImageNet contains pervasive annotation deviations, including images with multiple valid labels and ambiguous classes.These issues arise in a benchmark intended to approximate real-world object recognition.
  • Conclusion: Top-1 accuracy can underestimate models that predict a different but valid object, while models may exploit ImageNet-specific features humans do not recognize.Thus accuracy alone provides an imperfect view of underlying object-recognition performance.
  • Conclusion: More accurate models tend to be more human-aligned in their errors, and annotators largely cannot distinguish state-of-the-art predictions from ImageNet labels.This includes predictions that do not match the dataset label but are judged valid by annotators.
  • Conclusion: Scalable annotation choices can conflict with capturing real-world complexity, motivating pipelines that better capture ground truth while remaining scalable.The paper identifies non-expert annotation and the CONTAINS task as central examples of this tension.

A.1 Datasets

The analysis uses a balanced 10,000-image ImageNet validation subset and standard ImageNet-trained models. Human annotations are made tractable by narrowing candidate labels from model predictions before applying further human selection.

  • Datasets: The study samples 10,000 validation images by selecting 10 random images from each of ImageNet’s 1,000 classes.Performance on this subset closely mirrors overall test accuracy.
  • Datasets: The analysis groups the 1,000 ImageNet classes into 11 manually defined semantic superclasses.The groups range from Dogs and Birds to Devices, Structures, and Vehicles.
  • Models: The evaluation uses standard open-source pretrained ImageNet models spanning multiple architectures.The full model list is provided in the paper’s appendix and implementations come from established repositories.
  • Models: The paper reports corresponding ImageNet top-1 and top-5 accuracies for the evaluated models.These values are summarized in the model table.
  • Annotation procedure: Candidate labels are narrowed by combining top-5 predictions from 10 models with the existing ImageNet label, yielding about 14 labels per image on average.Annotators then reuse the CONTAINS task to prune this candidate set further.
  • Annotation procedure: The candidate-label distributions are presented to annotators through CONTAINS to identify relevant labels for the later classification task.The resulting labels are shown in separate grids for human assessment.

B.1.2 Multi-label validation task

The study builds candidate-label sets from CONTAINS responses, then asks annotators to identify all valid labels and the image’s main object in CLASSIFY. Quality controls and selection-frequency rules constrain which images and labels enter this multi-label validation.

  • Candidate-label generation: CONTAINS collects image-level selection frequencies by showing annotators 48-image grids for each query label.Each grid includes at least five control images and label descriptions based on WordNet synsets and Wikipedia.
  • Quality control: Quality control removes low-control-performance annotators and tasks, while CLASSIFY tasks are filtered using basic validity and main-label consistency checks.The CONTAINS filtering omitted 3,617 of 35,406 tasks.
  • Candidate-label generation: Candidate sets retain the ImageNet label, add labels selected by at least half of annotators, and include semantically distant labels with non-zero selection frequency.Labels more than 5 WordNet nodes from the ImageNet label are included to capture different objects.
  • Candidate-label generation: Candidate sets are expanded when fewer than 5 labels exist and truncated to 6 when excess labels are weakly selected, while always retaining the ImageNet label.Truncation also applies when the ImageNet label’s selection frequency is ≤1/8.
  • Classification task: In CLASSIFY, annotators select all valid non-mutually-exclusive labels and designate the image’s main object from the candidate set.Images whose ImageNet label was clearly dominant were omitted from this experiment.

C.1 Multi-object Images

Human annotations show that many ImageNet images contain multiple objects, sometimes with a main object different from the ImageNet label. Models can exploit dataset-specific cues, while top-5 accuracy may obscure errors on individual objects.

  • Multi-object prevalence: More than one fifth of ImageNet images contain objects from multiple classes despite having a single dataset label.Annotators generally agree on the number of objects present.
  • Main-object mismatch: ImageNet labels often differ from the object annotators judge to be the image’s main object, yet models can still predict the ImageNet label accurately.The paper reports that models may rely on distinctive features associated with the dataset label.
  • Co-occurring objects: Frequent co-occurrence or part-whole relationships between classes are associated with poor model performance on one or both classes.Examples include objects such as suits and ties or space bars and keyboards.
  • Evaluation: Top-5 accuracy is much higher than top-1 or multi-label accuracy on multi-object images, but only about 20% of top-5 corrections for recent models are multi-object confusions.The remaining corrections suggest top-5 evaluation can mask errors involving single objects.
  • Evaluation: These findings motivate performance metrics that reflect the underlying multi-object structure of the dataset.Accounting for multi-object confusions should avoid treating every top-5 correction as a resolved labeling ambiguity.

C.2 Bias in label validation

The paper identifies bias and ambiguity in ImageNet label validation, including potentially mislabeled images and imperfect links between human selection frequency and model accuracy. Human–model confusion patterns are compared across classes and superclasses.

  • Selection-frequency bias: Selection-frequency estimates are biased because the ImageNet images were previously filtered using another annotator pool and high ImageNet-label agreement.Subsampling workers slightly changes the relevant plots but does not change the conclusions.
  • Selection frequency and accuracy: Higher human selection frequency generally corresponds to better model accuracy, but the relationship is not uniform, especially for fine-grained or multi-object classes.Selection frequency is therefore an imperfect proxy for image-recognition difficulty.
  • Potentially mislabeled images: The study finds 150 of 10k images for which no annotator selected the ImageNet label as present.These are identified as possibly mislabeled images.
  • Potentially mislabeled images: A separate detailed annotation process finds 119 of 10k images where the ImageNet label was never selected for any object.The most frequently selected main-object label is also reported for these examples.
  • Confusion matrices: Confusion-matrix entries count images with ImageNet label i predicted as class j, using either top CONTAINS frequency or the CLASSIFY main label for humans.Model predictions use the top-1 label.
  • Human–model comparison: As models become more accurate, their inter-superclass confusion patterns align more closely with humans, while human intra-superclass confusions are more prominent.Prominent superclass confusion blocks appear related to frequent object co-occurrences.
Loading 2005.11295v1…