Source-linked AI summary
COCO-Stuff: Thing and Stuff Classes in Context
Holger Caesar, Jasper Uijlings, Vittorio Ferrari
TL;DR
Research has underrepresented stuff classes despite their role in describing scenes and contextualizing things. The paper introduces COCO-Stuff and an efficient superpixel-based annotation protocol, then uses the dataset to study annotation quality, dataset coverage, context, and segmentation. The authors conclude that stuff is important but not generally easier to segment than things.
Problem
Research has focused on thing classes, leaving limited attention to stuff classes despite their roles in scene description and contextual reasoning.
Method
The paper augments COCO with pixel-wise annotations for 91 stuff classes and develops a superpixel-based protocol leveraging existing thing annotations.
Results
COCO-Stuff shows that stuff covers most image surfaces, appears frequently in human descriptions, occurs in varied spatial contexts, and is not generally easier to segment than things.
Takeaways & Limitations
COCO-Stuff supports detailed study of stuff, things, and their contextual relations in complex images.
Takeaways & Limitations
The annotation analysis uses freedraw annotations as the pixel-level accuracy reference.
Abstract
from arXiv · showhide
Semantic classes can be either things (objects with a well-defined shape, e.g. car, person) or stuff (amorphous background regions, e.g. grass, sky). While lots of classification and detection works focus on thing classes, less attention has been given to stuff classes. Nonetheless, stuff classes are important as they allow to explain important aspects of an image, including (1) scene type; (2) which thing classes are likely to be present and their location (through contextual reasoning); (3) physical attributes, material types and geometric properties of the scene. To understand stuff and things in context we introduce COCO-Stuff, which augments all 164K images of the COCO 2017 dataset with pixel-wise annotations for 91 stuff classes. We introduce an efficient stuff annotation protocol based on superpixels, which leverages the original thing annotations. We quantify the speed versus quality trade-off of our protocol and explore the relation between annotation time and boundary complexity. Furthermore, we use COCO-Stuff to analyze: (a) the importance of stuff and thing classes in terms of their surface cover and how frequently they are mentioned in image captions; (b) the spatial relations between stuff and things, highlighting the rich contextual relations that make our dataset unique; (c) the performance of a modern semantic segmentation method on stuff and thing classes, and whether stuff is easier to segment than things.
1. Introduction
COCO-Stuff addresses the limited attention to amorphous stuff classes by enriching COCO with dense stuff annotations, enabling richer scene and contextual understanding.
- Motivation: Object-recognition research has focused on thing classes with characteristic shapes, sizes, and parts, while giving less attention to amorphous stuff classes.The paper asks whether this strong focus on things is justified.
- Motivation: Stuff helps describe scene type, constrain thing locations through 3D layout, determine relative positions, and provide context for recognizing uncommon things.Examples include beaches, contact points between regions, and metal objects interpreted differently in sky versus water.
- Dataset: COCO-Stuff augments COCO with pixel-wise annotations for 91 stuff classes alongside the original 80 thing classes.The annotations support studying stuff-thing interactions in complex COCO images.
- Dataset: COCO-Stuff provides more comprehensive scene annotations by adding surrounding stuff, interactions, and spatial arrangements absent from the original thing-only annotations.The train example includes track and relations among the train and surrounding regions, matching information expressed in human captions.
- Contributions: The paper introduces an efficient annotation protocol and analyzes stuff importance, contextual relations, and segmentation performance across stuff and thing classes.The dataset and trained segmentation models are released online.
2. Related Work
Prior work distinguishes stuff from things and includes several stuff-focused datasets, but COCO-Stuff targets richer, joint pixel-level annotations for complex images.
- Definitions: Things have characteristic shapes, sizes, and parts, whereas stuff is amorphous, variable in size, and generally lacks identifiable parts or countable instances.These distinctions motivate treating stuff and things as related but different semantic categories.
- Contextual recognition: Previous studies report that stuff and things require different detection techniques and that each can provide contextual cues for detecting the other.This related work frames stuff-thing interaction as relevant to recognition.
- Stuff-only datasets: Earlier stuff-only datasets often used simple textured patches, while later datasets addressed wild textures, materials, or limited stuff categories.The cited datasets generally do not provide the same joint coverage of stuff and thing classes in complex scenes.
- COCO-Stuff: COCO-Stuff is presented as the largest dataset with dense pixel-level annotations for both stuff and thing classes.Its dataset comparison includes image counts and class coverage across prior resources.
- COCO-Stuff: COCO-Stuff introduces a protocol for high-quality pixel-wise stuff annotations at low human cost by using superpixels and existing COCO thing annotations.The protocol is presented as a response to limitations in prior annotation resources.
3. The COCO-Stuff dataset
COCO-Stuff augments COCO with dense stuff annotations, creating a large, richly labeled dataset for studying stuff–thing context. Its superpixel-based protocol is efficient and accurate, while the dataset offers broad, balanced, and consistently defined classes compared with prior resources.
- Dataset: COCO-Stuff adds dense pixel-wise annotations for 91 stuff classes to COCO’s 80 thing classes across 164K complex images.The resulting dataset contains 172 classes, including one unlabeled category.
- Annotation protocol: The protocol uses boundary-respecting superpixels, existing COCO thing annotations, paintbrush labeling, and a label hierarchy to reduce annotation effort.Annotators label superpixels rather than manually tracing every boundary, while thing overlays clamp already annotated object pixels.
- Annotation analysis: Annotation time correlates with boundary complexity, while freedraw and polygon trendlines are 3.4x and 2.0x steeper than the superpixel trendline.This relationship explains why superpixels produce especially large average speedups on images with complex boundaries.
- Annotation analysis: Thing overlays provide 1.8x speedup for freedraw and 1.2x for superpixel annotation without changing superpixel agreement, which remains 96.1%.Thing-adjacent pixels account for 46.8% of boundary pixels, helping explain the efficiency benefit of overlays.
- Comparison to other datasets: COCO-Stuff provides more large-scale, usable stuff classes than prior datasets, with 84 classes occurring in at least 1,000 images and 92% meeting that threshold.Its labels are mutually exclusive and comparable in granularity, and each pixel receives exactly one label.
4. Analysis of stuff and things
COCO-Stuff shows that stuff occupies most annotated image content, participates in rich spatial context, and is not uniformly easier to segment than things. Its segmentation baseline also benefits from more training data.
- Relative importance: Stuff accounts for 69.1% of pixels, 69.4% of labeled regions, and 38.2% of caption nouns in COCO-Stuff.These measurements quantify stuff's prevalence in image content and human descriptions.
- Spatial context: Trains tend to occur above railroads, TVs in front of persons, tiled walls above tiled floors, and roads beside persons.The reported relations span thing-stuff, thing-thing, stuff-stuff, and stuff-thing interactions.
- Spatial context: Vertical context generally has higher confidence than horizontal context, suggesting support relations are especially informative.Nearby high-confidence regions can also indicate attachment, such as a person close to a backpack.
- Spatial context: COCO-Stuff has the highest mean contextual entropy among compared datasets, at 3.22 versus 2.42 for PASCAL Context, 2.18 for ADE20K, and 1.20 for SIFT Flow.Higher entropy reflects the richer and more diverse contextual distributions reported for COCO-Stuff.
- Semantic segmentation: DeepLab V2 with VGG-16 achieves 33.2% mIOU over all classes using the 118K training images.The evaluation uses pixel accuracy, class accuracy, mean IOU, and frequency-weighted IOU.
- Semantic segmentation: Performance significantly increases across all reported metrics as the training set grows from 1K to 118K images.On COCO-Stuff, DeepLab performs substantially better on thing classes than on stuff classes.
5. Conclusion
COCO-Stuff enriches COCO with dense stuff annotations and supports analysis of coverage, context, and segmentation. The paper concludes that stuff is important, contextually rich, not generally easier to segment, and useful with larger training sets.
- Conclusion: COCO-Stuff adds dense pixel-level annotations for diverse stuff classes to the existing COCO thing annotations.The dataset enables analysis of stuff-thing relations in complex images.
- Conclusion: Stuff covers most image pixels, appears in more than a third of caption nouns, and occurs in varied spatial contexts.The conclusion reports these properties as evidence of stuff's importance and contextual richness.
- Conclusion: Stuff is not generally easier to segment than things, while larger training sets improve semantic segmentation performance.These conclusions are stated as dataset-level findings from the COCO-Stuff analyses.