Source-linked AI summary
Microsoft COCO: Common Objects in Context
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, Piotr Dollár
TL;DR
Existing object-recognition datasets provide limited support for recognizing non-iconic, cluttered objects and reasoning about their context and precise localization. MS COCO addresses this with a large, densely annotated dataset of everyday scenes, whose models perform better on everyday-scene evaluation than models trained on prior datasets.
Problem
Existing datasets provide limited evidence for recognizing non-iconic, occluded, cluttered objects, reasoning about object context, and precisely localizing every instance.
Method
MS COCO collects non-iconic everyday-scene images and densely annotates common object instances with per-instance segmentation masks for detection and localization.
Results
Models trained on MS COCO perform better on everyday scenes than models trained on prior datasets, while the dataset averages 3.5 categories and 7.7 instances per image.
Takeaways & Limitations
MS COCO provides a benchmark centered on contextual information, varied viewpoints, and precise instance-level localization for object detection and segmentation.
Takeaways & Limitations
The dataset labels only “thing” categories, leaving “stuff,” occlusion, keypoints, scene types, attributes, and sentence descriptions for potential future annotation.
Abstract
from arXiv · showhide
We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.
1 INTRODUCTION
The introduction frames scene understanding as extending beyond object recognition to localization, attributes, relationships, and semantic description. It presents MS COCO as a large-scale dataset targeting non-iconic views, contextual reasoning, precise localization, and instance-level segmentation in complex everyday scenes.
- Motivation: Scene understanding includes recognizing, localizing, and characterizing objects, their attributes, relationships, and the overall scene.Existing object classification and detection datasets primarily address initial scene-understanding challenges.
- Research goals: The dataset targets non-iconic object views, contextual reasoning between objects, and precise 2D object localization.Its images reflect objects that are partially occluded, cluttered, and embedded in everyday scenes.
- Annotation novelty: Instance-level segmentation masks label every object instance, enabling precise localization and distinguishing COCO from prior datasets.The dataset focuses on segmenting individual object instances rather than only classifying images, localizing boxes, or producing semantic segmentation.
- Data collection: The data-collection pipeline used extensive Amazon Mechanical Turk involvement and retrieved images through object-pair and scene-based queries.This approach was designed to gather contextual relationships and non-iconic object views at scale.
- Dataset scale: 91 common object categories and 2,500,000 labeled instances in 328,000 images comprise the MS COCO dataset.82 categories contain more than 5,000 labeled instances, supporting detailed object models for precise 2D localization.
2 RELATED WORK
Related object-recognition datasets fall into classification, detection, and semantic scene-labeling groups. The paper positions its dataset between detection and scene labeling by using natural-context images and fully segmented instances for more accurate evaluation.
- Dataset taxonomy: Object-recognition datasets are broadly divided into object classification, object detection, and semantic scene labeling.The paper addresses each group in turn.
- Image Classification: Classification datasets evolved from single-object images with blank backgrounds toward more realistic internet images and larger category counts.Caltech 101 and Caltech 256 marked this transition; ImageNet later proposed 22k categories with 500-1000 images each.
- Object detection: Object detection requires both recognizing an object class and localizing the object, typically with a bounding box.PASCAL VOC contained 20 categories across 11,000 images, with over 27,000 labeled instance bounding boxes and almost 7,000 detailed segmentations.
- Object detection: Because objects such as sunglasses, cellphones, and chairs depend on context, detection datasets should show objects in natural environments.The paper also argues that fully segmented instances enable more accurate detector evaluation than bounding boxes.
- Semantic scene labeling: Semantic scene labeling assigns every pixel to a category without requiring separate instance segmentation, accommodating categories such as grass, streets, and walls.SUN combines properties of object detection and semantic scene labeling through scene categories and segmented objects.
3 IMAGE COLLECTION
The dataset uses entry-level object categories selected for representativeness, practical relevance, frequency, and balanced coverage across super-categories. Candidate images are collected to favor non-iconic scenes, using Flickr and category-pair searches that expose multiple objects and contexts.
- Category selection: Object categories were chosen to be representative, practically useful, frequent enough for large-scale collection, and balanced across super-categories.The selection also considered “thing” versus “stuff,” fine-grained, and object-part categories.
- Category selection: The dataset uses entry-level labels such as dog, chair, and person to enable collecting many instances per category.Category specificity can range from broad classes such as “mammal” to fine-grained classes such as “German shepherd.”
- Category selection: 272 candidate categories were assembled from PASCAL VOC, frequent visually identifiable object words, and names supplied by children aged 4 to 8.Co-authors then rated each category from 1 to 5 based on occurrence, practical usefulness, and diversity relative to other categories.
- Candidate image collection: The image collection targeted a majority of non-iconic images because such datasets generalize better.Images were collected from Flickr and through pairwise object or scene/object category searches rather than single-category searches alone.
- Candidate image collection: Pairwise searches such as “dog + car” produced non-iconic images containing numerous categories beyond those specified.At most 5 photos were downloaded from a single photographer within a short time window.
4 IMAGE ANNOTATION
MS COCO uses a cost-efficient, multi-stage Mechanical Turk pipeline to annotate object categories, instances, and per-instance segmentations at scale. Hierarchical category screening, worker training, verification, and multiple annotators were used to improve efficiency and annotation quality, while five captions were added per image.
- Annotation pipeline: The annotation pipeline used Amazon Mechanical Turk to label over 2.5 million object instances while emphasizing cost efficiency and quality.The pipeline consisted of category labeling, instance spotting, and instance segmentation stages.
- Category labeling: Workers first screened images through 11 super-categories, reducing the need to answer 91 separate category-presence questions per image.Workers examined each super-category in turn and considered its constituent categories only when the super-category was present.
- Instance spotting: For each detected category, workers marked every object instance, with previously found locations shown to subsequent workers to boost recall.The instance-spotting stage addressed cases where multiple instances of the same category appeared in one image.
- Instance segmentation: Over 22 worker hours per 1,000 segmentations were required for 2,500,000 instances, so each instance initially received one segmentation supported by category-specific training.Training was introduced because workers often produced coarse outlines when first completing the segmentation task.
- Quality analysis: Over 99% of unambiguous object-presence cases were estimated to be detected in practice by combining annotations from eight workers.For cases with over 50% annotation probability, the probability that all 8 annotators missed the case was at most .58 ≈.004.
- Captions: Five written caption descriptions were added to each MS COCO image.Caption statistics and collection procedures were deferred to a separate publication.
5 DATASET STATISTICS
MS COCO is compared with ImageNet, PASCAL VOC 2012, and SUN across category coverage, contextual richness, and object size. It has more instances per category than ImageNet and SUN despite fewer categories, exceeds PASCAL VOC in both categories and instances, contains multiple objects per image, and has smaller objects than SUN.
- Comparison datasets: The comparison covers ImageNet, PASCAL VOC 2012, and SUN, which differ substantially in dataset size, labeled categories, and image types.ImageNet emphasizes many fine-grained object categories, SUN labels scene types and commonly occurring objects, and PASCAL VOC provides another comparison dataset.
- Category and instance statistics: MS COCO has fewer categories than ImageNet and SUN but more instances per category, while exceeding PASCAL VOC in both categories and instances.The paper hypothesizes that more instances per category will aid learning complex models for precise localization.
- Contextual information: 3.5 categories and 7.7 instances per image characterize MS COCO’s non-iconic images containing objects in their natural context.Average categories and instances per image are used to estimate the contextual information present in images.
- Object size: MS COCO and SUN have smaller average object sizes than the other datasets analyzed.The paper notes that smaller objects are generally harder to recognize and require more contextual reasoning.
6 DATASET SPLITS
MS COCO was divided into two roughly equal releases to enable faster publication, with the 2014 release providing train, validation, and test images and the cumulative 2015 release expanding all three splits. The split construction minimized near duplicates, while annotations were released for train and validation but withheld for test; the 2014 release covered 80 categories.
- Release sizes: 82,783 training, 40,504 validation, and 40,775 testing images comprised the 2014 release.These corresponded to approximately 1/2 train, 1/4 validation, and 1/4 test.
- Annotation scale: 886k segmented object instances, including nearly 270k segmented people, were present in the 2014 train+validation data.These counts apply to the 2014 train+val data alone.
- Split construction: Near-duplicate images were explicitly removed, and images were grouped by photographer and capture date to reduce duplication across splits.Near duplicates were detected using the cited method.
- Evaluation protocol: Train and validation annotations were to be released, whereas test annotations were withheld for automatic evaluation on the test set.The evaluation server was still being finalized.
- Category coverage: 80 categories were included in the 2014 release because segmentations were not collected for 11 categories with excessive instances, ambiguity, labeling confusion, or too few instances.Some omitted categories might receive segmentations in the cumulative 2015 release.
7 ALGORITHMIC ANALYSIS
The analysis finds MS COCO substantially harder than PASCAL VOC for bounding-box detection, while MS COCO-trained models improve cross-dataset generalization but not uniformly across categories. It also introduces segmentation-from-detection methods and evaluates segmentation quality separately for correct detections.
- Bounding-box detection: Average DPMv5-P performance on MS COCO drops by nearly a factor of 2 compared with PASCAL VOC.The drop is attributed to more difficult non-iconic images with partial occlusion and clutter.
- Bounding-box detection: DPMv5-C outperforms DPMv5-P in 6 out of 20 categories when testing on PASCAL VOC.MS COCO-trained models perform worse for some categories, including dog, cat, and people, but better for others, including bus, tv, and horse.
- Bounding-box detection: Including difficult non-iconic images during training may not always help when the model cannot capture their appearance variability.Such examples may act as noise and pollute the learned model if it is not sufficiently rich.
- Bounding-box detection: 12.7 AP is the DPMv5-P cross-dataset performance difference, versus 7.7 AP for DPMv5-C.Overall performance is much lower on MS COCO, supporting the hypotheses that MS COCO is significantly harder than PASCAL VOC and that models trained on MS COCO generalize better across datasets.
- Generating segmentations from detections: The segmentation method learns aspect-specific, mixture-specific pixel-level masks by averaging aligned training-instance masks and pastes thresholded masks onto candidate detections.Segmentation quality is evaluated only on correct detections, defined by predicted-ground-truth bounding-box intersection over union of at least 0.5, then compared using segmentation overlap.
8 DISCUSSION
MS COCO introduces a large, crowd-annotated dataset for detecting and segmenting everyday objects in natural, varied contexts. The discussion highlights future extensions beyond object “things,” including contextual, occlusion, keypoint, scene, and attribute annotations.
- Contributions: MS COCO used over 70,000 worker hours to gather, annotate, and organize object instances in non-iconic everyday images across natural environments and varied viewpoints.The dataset emphasizes rich contextual information and diverse object appearances.
- Future annotations: Future annotations could label “stuff” alongside currently labeled “things” to provide additional contextual information useful for detection.The paper identifies this as a promising direction for expanding the dataset’s contextual coverage.
- Future annotations: Additional annotations could record instance occlusion and object keypoint locations, supporting detection algorithms that benefit from such information.The discussion cites occlusion amount and keypoint location as examples of useful supplementary labels.
- Future annotations: MS COCO could also benchmark labels for scene types and object attributes, extending evaluation beyond object detection and segmentation.These label types are proposed as further dataset extensions.
APPENDIX OVERVIEW
The appendix details the AMT user interfaces, lists 272 candidate object categories from which the final 91 were selected, and defines 40 scene categories for scene-object queries.
- Appendix overview: The appendix describes the AMT user interfaces and lists 272 candidate object categories, from which the final 91 were selected.It also provides 40 scene categories used for scene-object queries.
APPENDIX I: USER INTERFACES
The appendix presents a sequence of crowd interfaces for collecting non-iconic images, labeling categories and instances, segmenting and verifying masks, and efficiently labeling crowded scenes.
- Workflow: The workflow covers non-iconic image collection, category labeling, instance spotting, instance segmentation, segmentation verification, and crowd labeling.These interfaces support successive stages from image acquisition through object annotation and quality control.
- Category Labeling: Workers annotate categories by dragging icons onto corresponding object instances, with only one instance per category required initially.Icons are grouped by super-categories to help workers skip unlikely categories quickly.
- Instance Spotting: Workers spot up to 10 instances of a category by clicking within each region, aided by a magnifying-glass feature for small objects.The interface begins from an instance identified during category labeling.
- Instance Segmentation: The segmentation interface assigns one previously labeled object instance per task and adds a category-icon visualization to guide workers.It adapts source code from the OpenSurfaces project for single-object instance segmentation.
- Segmentation Verification: Verification presents 64 segmentations, including four known bad examples; workers must identify 3 of 4 to complete the task.Each segmentation is initially shown to 3 annotators, with 2 additional workers if any flags it as bad.
- Crowd Labeling: For images with many instances, crowd labeling lets workers paint all object pixels instead of drawing exact polygonal masks around each instance.This method is used when a category has more than 10 instances in an image.
APPENDIX II: OBJECT & SCENE CATEGORIES · TABLE 3: Scene category list.
The appendix defines the dataset’s object and scene-category coverage. It reports 91 object categories, alongside 40 SUN-derived scene categories evenly divided between indoor and outdoor settings.
- APPENDIX II: OBJECT & SCENE CATEGORIES: 91 object categories comprise the dataset’s category inventory.The 2014 release provides segmentation masks for 80 of these categories.
- APPENDIX II: OBJECT & SCENE CATEGORIES: 80 object categories have segmentation masks in the 2014 release.
- APPENDIX II: OBJECT & SCENE CATEGORIES: The initial object-category list drew from WordNet, LabelMe, SUN, other sources, and free recall by young children.
- APPENDIX II: OBJECT & SCENE CATEGORIES: 272 candidate object categories were voted on to sample a diverse and computationally challenging set.
- TABLE 3: Scene category list.: 40 scene categories were selected from SUN because they frequently co-occurred with object categories of interest.
- TABLE 3: Scene category list.: The 40 selected scene categories are evenly split between indoor and outdoor scenes.