PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageNet Large Scale Visual Recognition Challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, Li Fei-Fei

arXiv:1409.0575v3cs.CV

TL;DR

Scaling object recognition to thousands of categories and millions of images creates annotation and evaluation challenges. This paper presents the ILSVRC benchmark, reviews resulting advances in classification and detection, and reports that trained humans outperformed GoogLeNet by approximately 1.7%.

  • Problem

    Scaling from small datasets to millions of images and thousands of object classes makes conventional annotation approaches infeasible and large-scale evaluation ambiguous.

  • Method

    The paper constructs ILSVRC with image- and object-level annotations, defines standardized evaluation procedures, and analyzes recognition algorithms and their development.

  • Results

    The challenge documented advances in classification and detection, while trained human annotators outperformed GoogLeNet by approximately 1.7%.

  • Takeaways & Limitations

    ILSVRC provides a large-scale basis for evaluating recognition algorithms and analyzing their success and failure modes.

  • Takeaways & Limitations

    Annotating images across 1000 categories is extremely challenging for untrained annotators, who may overlook relevant classes because they are unaware of them.

Abstract

from arXiv · show

The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.

1 Introduction

ILSVRC established a large-scale benchmark for object recognition by expanding standardized evaluation to 1,000 categories and over 1.4 million images. This paper examines the dataset’s construction challenges, resulting advances in classification and detection, and the field’s current state.

  • Overview: Since 2010, ILSVRC has run annually and become the standard benchmark for large-scale object recognition.It follows PASCAL VOC’s model of yearly competitions with standardized evaluation.
  • Overview: ILSVRC provides image-level presence labels and object-level bounding boxes with class labels.Image-level annotations indicate whether a class is present; object-level annotations localize individual instances with tight boxes.
  • Large-scale challenges and innovations: Scaling from 19,737 images and 20 classes in PASCAL VOC 2010 to 1,461,406 images and 1,000 classes in ILSVRC 2010 created major annotation challenges.The scale made small-group annotation infeasible, while some categories were difficult to delineate and exhaustive object localization was impractical.
  • Large-scale challenges and innovations: The dataset’s scale enabled new recognition techniques, evaluation opportunities, fine-grained class discrimination, and comparisons with human-level accuracy.Its broad category spectrum also supported analysis of object statistics and their effects on recognition algorithms.
  • Goals: The paper aims to explain benchmark-creation challenges, highlight advances in object classification and detection, and assess categorical object recognition.These are identified as the paper’s three key goals.

2 Challenge tasks

ILSVRC evaluated image classification, single-object localization, and object detection, progressively requiring category labels, instance localization, and localization of multiple object categories. The tasks used standardized image data and annotations, with evaluation criteria increasing from label matching to localization accuracy and detection precision and recall.

  • Task overview: ILSVRC included image classification (2010-2014), single-object localization (2011-2014), and object detection (2013-2014).These tasks produced category labels, category labels with an axis-aligned bounding box for one instance, and bounding boxes for multiple object categories, respectively.
  • Image classification: 1000 object categories and one ground truth label defined the image classification data, with quality evaluated by the label best matching the ground truth.The photographs came from Flickr and other search engines and were manually labeled for category presence.
  • Single-object localization: Single-object localization added an axis-aligned bounding box for every instance of the labeled category and required both category-label and predicted-location accuracy.The task used the same photographs and 1000 categories as image classification, while evaluating appearance of the target object rather than image context.
  • Object detection: Object detection extended single-object localization to multiple object categories, evaluating detections by recall of target instances and precision against spurious detections.Scaling the task to an order of magnitude more categories and images created substantial dataset collection and annotation challenges.

3 Dataset construction at large scale

The ILSVRC dataset-construction process defines target categories, collects diverse candidate images, and annotates millions of images with task-specific crowdsourcing strategies. It produces large classification and detection datasets while emphasizing scalable annotation quality and precise bounding boxes.

  • Three-step construction: Dataset construction has three steps: defining target categories, collecting diverse candidate images, and annotating the millions of collected images.Category selection combines automatic heuristics with manual post-processing; image collection uses automatic and manual strategies across multiple search engines; annotation uses task-specific crowdsourcing.
  • Image classification dataset: 1000 object classes and approximately 1.2 million training, 50 thousand validation, and 100 thousand test images support image classification.The categories are selected from ImageNet synsets without ancestor-descendant overlap, and classification training images come directly from ImageNet while additional images form validation and test sets.
  • Annotation procedure: Bounding boxes are required to be as small as possible while including all visible object parts, avoiding ambiguity from estimating occluded extents.This procedure defines a visible-object annotation standard rather than an inferred full-object extent.
  • Annotation quality: 99.2% of bounding boxes are accurate and visibly tight, while none have less than 50% intersection over union overlap with ground truth.The remaining 0.8% are somewhat off, but no boxes fall below the stated overlap threshold.
  • Object detection dataset: 200 object classes and approximately 450K training, 20K validation, and 40K test images comprise the ILSVRC object-detection dataset.In ILSVRC2012, 40% of training images were annotated, yielding more than 520 thousand annotated images and more than 590 thousand annotated object instances; the benchmark substantially exceeds PASCAL VOC in scale.

4 Evaluation at large scale

ILSVRC defines standardized evaluation criteria for classification, single-object localization, and object detection while adapting established benchmarks to large-scale annotation and evaluation challenges. Classification uses top-5 error, localization requires both correct class identification and bounding-box overlap, and detection is evaluated with average precision.

  • Evaluation framework: ILSVRC evaluates three tasks—image classification, single-object localization, and object detection—using standardized criteria adapted to the large-scale setting.The procedures address challenges arising from the scale of the dataset and annotations.
  • Image classification: Top-5 classification is correct when any of five returned labels matches the single ground-truth class, and all three error measures produced the same result ordering.The measures were top-5, top-1, and hierarchical error; ILSVRC has used top-5 exclusively since ILSVRC2012.
  • Single-object localization: Single-object localization is correct only when one of five predictions both identifies the target class and localizes an instance with intersection-over-union greater than 0.5.Localization evaluation can require discarding inherently ambiguous images, such as images containing many fruit instances.
  • Object detection: Object detection penalizes missed instances, duplicate detections, and false positives, then scores each class by average precision across recall levels.Predictions are greedily matched to ground-truth boxes, and the challenge winner is the team winning the most object classes.
  • Object detection: ILSVRC includes very small instances in evaluation and adjusts the localization threshold to better handle low-resolution objects rather than ignoring them as difficult.This differs from PASCAL VOC, which ignored small difficult instances; ILSVRC uses a threshold computed from the ground-truth box dimensions.

5 Methods

The methods evolved from hand-crafted feature representations and SVMs in 2010–2011 to large-scale convolutional neural networks in 2012, which dominated subsequent ILSVRC submissions. This shift coincided with rapid participation growth and major gains in classification and detection performance by 2014.

  • ILSVRC2010: In 2010, NEC won classification using SIFT and LBP features, nonlinear coding representations, and a stochastic SVM, while XRCE used Fisher vectors, PCA, compression, and a linear SVM.Fisher vector-based methods continued performing strongly in every ILSVRC from 2010 to 2014.
  • ILSVRC2011: In 2011, XRCE won classification with compressed high-dimensional image signatures and one-vs-all linear SVMs, while UvA won the first localization competition using selective search regions.The 2011 localization competition had two entries.
  • ILSVRC2012: In 2012, SuperVision won both classification and localization with a 60 million-parameter deep convolutional neural network trained on RGB values using efficient GPU implementation and dropout.The passage identifies 2012 as a turning point when large-scale deep neural networks entered the challenge.
  • ILSVRC2013: In 2013, 24 teams participated, most using deep convolutional neural networks; Clarifai won classification with averaged deep networks, and OverFeat won localization with an integrated multiscale framework.OverFeat’s framework addressed classification, localization, and detection, making it the only team tackling all three tasks.
  • ILSVRC2014: In 2014, 36 teams submitted 123 entries, a 1.5x increase in participation, while nearly all submissions used convolutional neural networks.Image classification error was almost halved since ILSVRC2013, and object detection mean average precision almost doubled.

ILSVRC 2011

The ILSVRC 2011 results table lists participating teams, institutions, contributors, and reported scores. Entries include NII at 50.5, UvA at 31.0 and 42.5, and XRCE at 25.8 and 56.5.

  • ILSVRC 2011: NII, from Japan’s National Institute of Informatics, is reported at 50.5.The listed contributors are Duy-Dinh Le and Shin´ıchi Satoh.
  • ILSVRC 2011: UvA, representing the University of Amsterdam and University of Trento, is reported at 31.0 and 42.5.The listed contributors include Koen E. A. van de Sande, Jasper R. R. Uijlings, Arnold W. M. Smeulders, Theo Gevers, and Nicu Sebe.
  • ILSVRC 2011: XRCE, associated with Xerox Research Centre Europe and CIII, is reported at 25.8 and 56.5.The listed contributors are Florent Perronnin and Jorge Sanchez.

ILSVRC 2012

ILSVRC 2012 marked a major breakthrough in large-scale recognition when SuperVision won both image classification and single-object localization. Its extra-data submissions achieved 15.3% classification error and 33.5% localization error.

  • ILSVRC 2012: 15.3% classification error and 33.5% localization error were achieved by SuperVision entries trained with extra ImageNet Fall 2011 data.The table reports flat top-5 classification and single-object localization error, with lower values better.
  • ILSVRC 2012: 2012 marked the major breakthrough: SuperVision won both image classification and single-object localization.The win came as neural networks began learning directly from image data with efficient algorithms and GPU computing.
  • ILSVRC 2012: By 2014, all top contestants relied heavily on convolutional neural networks, following the 2012 SuperVision breakthrough.Large-scale training data, efficient implementations, and GPU computing enabled direct learning from image data instead of multi-stage hand-tuned pipelines.

6 Results and analysis · 6.1 Improvements over the years

Winning ILSVRC performance improved substantially from 2010 to 2014 across classification, localization, and detection. Classification and localization errors declined steadily, while detection mAP gains reflected both algorithmic innovation and expanded training data.

  • 6.1 Improvements over the years: Winning entries show substantial, steadily improving performance across ILSVRC2010–2014 in all three tasks.The paper quantifies and analyzes this five-year improvement.
  • 6.1.1 Image classification and single-object localization improvement over the years: 4.2x reduction in image classification error occurred from 28.2% to 6.7% since the challenge began.The comparison considers only teams using the provided training data; dataset scale remained comparable despite category changes.
  • 6.1.1 Image classification and single-object localization improvement over the years: 1.7x reduction in single-object localization error occurred from 42.5% to 25.3% since the challenge began.Over the past three years, single-object localization error also fell from 33.5% to 25.3%.
  • 6.1.2 Object detection improvement over the years: 1.9x improvement in object-detection mAP occurred from 22.6% mAP in ILSVRC2013 to 43.9% mAP in ILSVRC2014.The comparisons are not direct because detection training data increased and the 2014 result used classification and localization training data.
  • 6.1.2 Object detection improvement over the years: 3.7% absolute mAP increase resulted when the same UvA framework used ILSVRC2014 rather than ILSVRC2013 detection data.The framework achieved 22.6% with ILSVRC2013 data and 26.3% with ILSVRC2014 data without other modifications.
  • 6.1.2 Object detection improvement over the years: 3.1% absolute mAP increase resulted from expanding ILSVRC2013 detection data to ILSVRC2014.This isolates the effect of the larger detection dataset in the reported comparison.
  • 6.1.2 Object detection improvement over the years: Approximately 1 −4% in absolute mAP improvement was attributed to expanded detection data and added classification training data.The paper separately considers algorithmic innovation as another contributor to detection gains.
  • 6.1.2 Object detection improvement over the years: 21.3% absolute mAP increase separated the winning ILSVRC2013 and ILSVRC2014 entries, indicating algorithmic innovation was not the sole consequence of increased training data.Further enlarging the ILSVRC2014 detection training dataset was expected to improve current algorithms.

6.2 Statistical significance

The section uses bootstrap sampling to quantify statistical differences between ILSVRC submissions. Across ILSVRC2012–2014, winning methods differ significantly from other top entries even at the 99.9% confidence level.

  • 6.2 Statistical significance: Bootstrap sampling constructs confidence intervals for each method’s score by repeatedly sampling N test images with replacement.The procedure runs 20,000 rounds or more until convergence and discards the lower and upper α intervals.
  • 6.2 Statistical significance: Table 8 reports 99.9% confidence intervals for up to five submissions per ILSVRC task in 2012–2014.The winning method’s difference from the runner-up is significant at the 99.9% level, including comparisons among winners using provided data.
  • 6.2 Statistical significance: 20,000+ bootstrap rounds show that winning methods are statistically significantly different from other methods at the 99.9% level.The analysis covers the top entries for each ILSVRC task from 2012 through 2014.

6.3 Current state of categorical object recognition

The optimistic models achieve strong average performance but remain far from saturating ILSVRC, with substantial variation across object categories. Accuracy depends on object scale and intrinsic properties including real-world size, deformability, texture, and whether objects are natural or man-made.

  • Overall performance: 94.6% average image classification accuracy, 81.5% localization accuracy, and 44.7% detection AP coexist with large cross-category ranges, showing ILSVRC remains far from saturated.Accuracy ranges are 41.0% for classification, 77.0% for localization, and 84.7% for detection.
  • Object scale: Scale correlates weakly with classification accuracy (ρ = 0.14) but more strongly with localization (ρ = 0.40) and detection (ρ = 0.41), so scale alone cannot explain variation.The analysis relates performance to average object scale across object classes.
  • Intrinsic object properties: Deformable objects outperform rigid objects across tasks, but man-made classes remain harder than natural classes, with detection mAP of 38.7% versus 50.9%.Classification is 92.8% versus 97.0%, and localization is 75.5% versus 88.5%, for man-made versus natural classes respectively.
  • Real-world size: After scale normalization, classification accuracy is 93.6%−93.9% for XS−M objects versus 97.0% for L and 96.4% for XL objects, while detection favors XS objects at 44.5% mAP.Localization is easiest for L objects at 82.4% but hardest for XL objects at 73.4%.

6.4 Human accuracy on large-scale image classification

On a 1,500-image ILSVRC sample, a sufficiently trained human annotator outperformed GoogLeNet, while limited training produced substantially worse accuracy. Human and model errors differed: humans struggled mainly with fine-grained categories, whereas GoogLeNet struggled with small objects and image distortions.

  • Annotator performance: A1 achieved 5.1% human error versus 6.8% GoogLeNet error on 1,500 images, outperforming the model by approximately 1.7%.The reported difference was statistically significant (p = 0.022).
  • Annotator performance: A2’s 12.0% Top-5 error after training on only 100 images shows that significant training time is necessary for competitive human performance.48.8% of A2’s errors resulted from failing to consider the ground-truth label as an option.
  • Annotator comparison: On 204 jointly labeled images, A1 and A2 were both correct on 174 images (85%), while an optimistic combination yielded 2.4% error versus 4.9% for GoogLeNet.The annotators’ predictions showed limited overlap, with 19 images correct only for A1 and 6 correct only for A2.
  • Error analysis: Multiple-object images caused 24% of GoogLeNet errors and 16% of human errors, while small or thin objects caused 21% of GoogLeNet errors and none of the human errors.These findings indicate different weaknesses in model and human recognition under the single-label classification setting.
  • Error analysis: Fine-grained recognition caused 37% of human errors but only 7% of GoogLeNet errors, with more than 120 dog species contributing to the difficulty.The analysis also identifies class unawareness as a major source of human error.
  • Implications: Human superiority over state-of-the-art models requires significant effort, expertise, and time, while model errors may be reduced through greater robustness and multi-scale reasoning.The paper highlights filters, rotations, collages, and abstract object representations as distinct improvement areas.

7 Conclusions

The conclusions identify lessons from five years of ILSVRC, address criticisms of dataset difficulty, annotation errors, and competition rules, and outline future directions for large-scale image understanding and evaluation.

  • Lessons learned: ILSVRC showed that human intelligence tasks require exceptionally well-designed interfaces and procedures, both for dataset annotation and human-accuracy evaluation.The first labeling interface was generally unusable, requiring iterative improvement.
  • Lessons learned: Scaling datasets revealed unexpected challenges, requiring multi-step annotation strategies and modifications to evaluation procedures while enabling major recognition breakthroughs.The authors also analyzed the strengths and weaknesses of current algorithms in the large-scale setting.
  • Addressing criticisms: ILSVRC was comparable with, and often more challenging than, PASCAL VOC despite criticism that its objects were large and centered.The authors analyzed dataset statistics to address this criticism.
  • Addressing criticisms: Crowdsourced annotations underwent multiple rounds of in-house post-processing, but fine-grained object classes remained a major source of errors.The authors describe this as a budget-driven tradeoff required to annotate data at ILSVRC scale.
  • Future directions: Future large-scale benchmarks should pursue pixel-level object segmentation, rely more on weak supervision, and develop evaluation methods suited to increasingly incomplete manual annotation.The authors note that billions of images may make even one clean label per image impossible.

Appendix A ILSVRC2012-2014 image classification and single-object localization object categories

This appendix enumerates object categories used for ILSVRC2012–2014 image classification and single-object localization, spanning diverse animals, artifacts, vehicles, foods, and scenes.

  • The category list begins with entries such as abacus, abaya, academic gown, accordion, acorn, acoustic guitar, admiral, and Afghan hound.
  • It includes categories ranging from African elephant and aircraft carrier to ambulance, American black bear, lobster, and scoreboard.
  • Later entries cover machine, Shetland sheepdog, shopping cart, Siamese cat, snow leopard, soccer ball, space shuttle, and spaghetti squash.

Appendix B Additional single-object localization dataset statistics

The appendix compares ILSVRC single-object localization with PASCAL VOC using chance performance of localization and clutter. ILSVRC has higher average CPL overall, while some categories are exceptionally difficult to localize and clutter-based localization difficulty is substantial.

  • Chance performance of localization: CPL estimates localization accuracy by reusing a randomly sampled same-class instance’s bounding box as the proposed window on other images.Images are rescaled to the same size before applying the sampled bounding box.
  • Chance performance of localization: 0.2% CPL or less occurs for difficult categories including basketball, swimming trunks, ping pong ball, and rubber eraser.CPL correlates strongly with average object scale, with ρ = 0.9.
  • Chance performance of localization: 20.8% is the average CPL across 1000 ILSVRC categories, versus 8.7% across 20 PASCAL categories.The 20 PASCAL categories have the same CPL as the 562 most difficult ILSVRC categories.
  • Clutter: Clutter measures how many generic object-looking windows must be considered before a target instance is localized with intersection-over-union of at least 0.5.Images lacking localization within the first 1000 windows are assigned obj(m) = 1001.
  • Clutter: 1% of ILSVRC images per category and 5% of PASCAL images require more than 1000 windows, while more than 95% of objects are localized within those windows.Higher clutter indicates greater localization difficulty using generic cues.

Appendix C Manually curated queries for obtaining object detection scene images

The appendix lists 129 manually curated queries used to collect object-detection scene images, complementing single-category and paired-category queries. These manual queries targeted one or more object categories with insufficient data and included diverse scene descriptions.

  • Query types: 129 manually curated queries were provided for collecting object-detection images.The queries were introduced as a distinct collection category in addition to other query types.
  • Query types: The collection used three query types: single-category names or synonyms, pairs of category names, and manual queries.Manual queries typically targeted one or more categories with insufficient data.
  • Query examples: Examples of manual queries include “afternoon tea,” “ant bridge building,” “carpenter drilling,” and “dragonfly pond.”The list spans activities, locations, objects, meals, and category-specific scenes.

Appendix D Hierarchy of questions for full image annotation

Appendix D defines a manually constructed question hierarchy for crowdsourcing full-image annotation of 200 object-detection categories in ILSVRC2013 and ILSVRC2014. Binary parent questions enable descendant-label propagation, while leaf nodes specify the 200 categories and include detailed examples and disambiguation rules.

  • Hierarchy and annotation rules: The annotation hierarchy asks whether each image contains objects from 200 detection categories in ILSVRC2013 and ILSVRC2014.Questions are manually constructed and take the form “is there a ... in the image?”
  • Hierarchy and annotation rules: Questions marked with • are asked on every image, and a “no” answer propagates to all descendant questions.This hierarchy reduces annotation effort by assuming descendants are absent when their parent category is absent.
  • Category organization: The hierarchy organizes leaf categories under broad groups including medical items, musical instruments, food, electronics, appliances, furniture, clothing, organisms, and vehicles.Examples include stethoscopes, accordions, pretzels, laptops, chairs, cats, fish, and cars.
  • Category definitions: Category definitions include explicit visual distinctions to reduce ambiguity between related objects and concepts.Examples distinguish electric fans from hair dryers, printers from typewriters, bowls from cups, ties from bow ties, and snakes from lizards.
  • Category organization: The hierarchy also separates organisms by locomotion and habitat and vehicles by transportation function and subtype.It distinguishes flying from non-flying organisms, water-dwelling and land-sliding organisms, and wheeled vehicles such as cars, buses, trains, and bicycles.

Appendix E Modification to bounding box system for object detection

The object detection dataset required two manual post-processing steps beyond the shared bounding-box annotation system: resolving ambiguous object labels and removing duplicate boxes. These corrections addressed errors that were more consequential for detection because every object instance had to be localized without duplicate detections.

  • Additional post-processing: Object detection annotation used the same bounding-box system as single-object localization but required two additional manual post-processing steps for accuracy.The two steps addressed ambiguous object labels and duplicate bounding boxes.
  • Ambiguous objects: Ambiguous labels arose when workers confused visually or semantically similar classes, including seals with sea otters, backpacks with purses, and banjos with guitars.Other confusions included violins and cellos, brass instruments, flutes and oboes, and ladles and spatulas.
  • Ambiguous objects: About 3% of collected boxes significantly overlapped boxes from different classes, and about a quarter of these were incorrect objects that were removed after manual review.This review corrected class-confusion errors that could not be resolved by simply discarding ambiguous images in the detection setting.
  • Duplicate annotations: Duplicate boxes persisted despite annotation instructions and interface constraints because later labelers sometimes drew slightly improved alternatives around the same object.Such duplicates were more serious for detection than localization because detection evaluates every object instance and penalizes duplicate detections.
  • Duplicate annotations: Approximately 1% of boxes overlapped by more than 50% with another box of the same class; manual verification retained both when they represented distinct nearby instances and randomly removed one otherwise.In approximately 40% of these cases the boxes correctly represented different instances, while in the other 60% one box was removed.

Appendix F Competition protocol

The competition releases data and specifications, collects deadline-bound test predictions, and evaluates submissions centrally. Test annotations are generally hidden, while validation code supports local accuracy assessment.

  • Competition format: Each year, organizers release training, validation, and test images, training and validation annotations, and a competition specification before setting a submission deadline usually approximately 4 months later.Teams upload a text file of predicted test annotations to a provided server, after which organizers evaluate all submissions and release results.
  • Competition format: For every task, released code compares automatically generated annotations with ground truth and returns a quantitative accuracy measure for validation-data evaluation.
  • Competition format: The competition uses organizer-assessed test evaluation: ILSVRC2010 classification test annotations were released, but all other test annotations remained hidden to discourage test-data fine-tuning.This follows the PASCAL VOC choice among three possible test-performance measurement options.
  • Evaluation protocol after the challenge: After the challenge, an automatic evaluation server remains available throughout the year, with teams limited to 2 submissions per week to discourage test-data parameter tuning.The authors report no practical abuse of the system.
Loading 1409.0575v3…