Source-linked AI summary
Semantic Understanding of Scenes through the ADE20K Dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, Antonio Torralba
TL;DR
The paper addresses the shortage of datasets with dense, detailed annotations spanning diverse scenes and object categories. It introduces ADE20K and benchmarks scene parsing and instance segmentation, finding that the dataset supports broad scene and object coverage while exposing challenges from small, diverse, and crowded content.
Problem
Few existing image datasets cover broad scenes and object categories with dense, detailed annotations needed to recognize and segment objects, stuff, and parts.
Method
The paper constructs ADE20K with exhaustive annotations of scenes, stuff, objects, parts, and attributes, then builds benchmarks and evaluates segmentation networks.
Results
ADE20K contains 20,210 training images, 2,000 validation images, 3,000 testing images, and 3,169 annotated class labels, including 476 object-part classes.
Takeaways & Limitations
The released dataset, benchmarks, data, and pretrained models provide a broad basis for evaluating scene parsing and instance segmentation across visual concepts.
Takeaways & Limitations
ADE20K remains challenging because its small objects, diverse scenes, and crowded layouts make recognition, classification, and instance segmentation difficult.
Abstract
from arXiv · showhide
Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision. Despite the community's efforts in data collection, there are still few image datasets covering a wide range of scenes and object categories with dense and detailed annotations for scene parsing. In this paper, we introduce and analyze the ADE20K dataset, spanning diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. A generic network design called Cascade Segmentation Module is then proposed to enable the segmentation networks to parse a scene into stuff, objects, and object parts in a cascade. We evaluate the proposed module integrated within two existing semantic segmentation networks, yielding significant improvements for scene parsing. We further show that the scene parsing networks trained on ADE20K can be applied to a wide variety of scenes and objects.
1 Introduction
Scene understanding requires dense pixel-level recognition and segmentation of objects, stuff, and object parts across diverse scenes. ADE20K addresses limitations in existing datasets by providing detailed annotations over a broad vocabulary and scene range.
- Motivation: Scene understanding must localize objects and their parts, not merely predict an image-level scene category.Examples include recognizing sofa, table, and TV along with parts such as a chair seat or cup handle.
- Motivation: Existing datasets often limit object categories or scene diversity, constraining pixel-level scene understanding.COCO and Pascal contain limited object sets, while Cityscapes covers a limited set of scenes.
- Dataset contribution: ADE20K was designed with every pixel semantically labeled across diverse scenes, objects, and object parts using an open vocabulary.The dataset uses detailed manual segmentation and covers stuff, objects, and parts of objects.
- Dataset contribution: A single expert annotator was used to provide exhaustive annotations and consistent object naming despite the difficulty of open-vocabulary labeling.The annotation process addresses both annotator reliability and naming inconsistency challenges.
- Benchmarks and models: The work builds scene-parsing and instance-segmentation benchmarks on ADE20K and evaluates baseline and state-of-the-art segmentation networks.The paper also examines batch normalization, joint object-part training, and challenge results.
- Relation to prior datasets: ADE20K extends prior datasets with high-resolution imagery, a wider selection of scenes, many object classes per image, and annotations for stuff, objects, and parts.This advances earlier efforts such as Pascal-Part and OpenSurfaces.
2 ADE20K: Fully Annotated Image Dataset
ADE20K is constructed as a densely annotated, open-vocabulary dataset covering diverse scenes, objects, stuff, and hierarchical object parts. Its annotation analysis shows broad coverage but also measurable inconsistency from segmentation quality, naming ambiguity, and missed objects.
- Dataset construction: ADE20K targets diverse scenes with dense pixel-level annotations for objects, stuff, object parts, and parts of parts.Images were selected across 900 SUN scene categories and annotated with object segments, parts, and attributes.
- Dataset construction: Objects are independently segmented and ordered by depth layers to convert overlapping polygons into segmentation masks.Background classes are assigned the farthest layers, while contained objects receive closer layers.
- Dataset construction: The open-ended labeling protocol grows a visual-concept dictionary because new object categories frequently appear during exhaustive annotation.This supports consistent naming without fixing the complete class list in advance, although rare classes and naming consistency remain difficult.
- Annotation consistency: 82.4% of pixels received the same label when 61 validation images were re-annotated by the same expert six months apart.The remaining errors included 5.7% segmentation-quality errors, 6.0% object-naming errors, and 5.9% missing-label errors; external annotators showed higher inconsistency.
3 Pixel-wise Scene Understanding Benchmarks
ADE20K supports two pixel-wise benchmarks: scene parsing and instance segmentation. SceneParse150 covers 150 semantic categories, while baseline studies compare segmentation architectures, context modules, cascade designs, and batch-normalization settings.
- ADE20K defines scene parsing as dense semantic labeling and instance segmentation as detecting object instances with precise masks.
- Scene parsing benchmark: SceneParse150 contains 35 stuff and 115 discrete object classes, covering 92.75% of annotated pixels.Stuff accounts for 60.92% and discrete objects for 31.83%.
- Scene parsing benchmark: The benchmark links its 150 object categories through WordNet hypernym relations into semantic clusters such as furniture, conveyance, and living things.This structured annotation connects image labels to a wider knowledge base.
- Scene parsing benchmark: Dilated-convolution networks generally outperform FCN and SegNet, while cascade integration further improves scene-parsing performance.In mean IoU, Cascade-SegNet and Cascade-DilatedVGG exceed SegNet and DilatedVGG by 6% and 2.5%, respectively.
- Scene parsing benchmark: Small, infrequent objects and fine details account for much of the performance difference among baseline networks.Common, large objects and stuff classes are correctly predicted more consistently.
- Scene parsing benchmark: PPM and UPerNet improve mean IoU by 3-7% and pixel accuracy by 1-2% over plain DilatedResNet, underscoring the role of contextual information.The comparison is reported on the SceneParse150 validation set.
- Effect of batch normalization: A BN size of 4 is the observed inflection point: synchronized BN remains stable from sizes 4 to 16, whereas size 2 significantly degrades performance.With batch and BN size 16, the model exceeds frozen BN by 1.41% in Pixel Acc. and 3.17% in Mean IoU.
- Effect of batch normalization: For semantic segmentation, BN size matters more than total batch size, as changing batch size without changing BN size yields nearly identical results.
4 Places Challenges
The Places Challenges benchmarked scene parsing and instance segmentation on ADE20K, revealing strong performance alongside persistent difficulties with unusual, small, occluded, and crowded objects.
- The 2016 and 2017 Places Challenges included scene parsing and instance segmentation tracks built on ADE20K benchmarks.
- Scene Parsing Track: Scene parsing submissions were ranked by the average of mean IoU and pixel-wise accuracy, with PSPNet winning in 2016 and retaining the highest score.
- Scene Parsing Track: 20.30% pixel accuracy was achieved by using the image mode as the testing prediction, providing a possible lower bound for scene-parsing models.
- Scene Parsing Track: Scene-parsing models commonly failed on occluded concepts, non-canonical object views, and objects requiring high-level context reasoning, such as a camouflaged muddy car.
- Instance Segmentation Track: The 2017 instance-segmentation track received 12 submissions from 5 teams, and two teams surpassed the Mask R-CNN baseline by a substantial margin.
- Instance Segmentation Track: Megvii’s small-object mAP improvement over G-RMI was 41%, compared with 19% for medium objects and 6% for large objects.
- Take-aways from the Challenge: ADE20K challenges instance segmentation through numerous small objects, diverse scenes, and crowded layouts with inter-class and intra-class occlusion.
5 Object-Part Joint Segmentation
ADE20K’s part annotations support joint object-part segmentation, implemented as a cascade that first predicts objects and then selects corresponding part classes.
- ADE20K contains parts for 59 of its 150 objects, with 153 part classes included for joint object-part segmentation.
- UPerNet jointly trains object and part segmentation using a non-part class and computes part losses within the ground-truth object’s corresponding part set.
- The cascade predicts object classes first, then obtains part predictions from the corresponding object-specific part set.
- UPerNet produces qualitative joint object-part predictions and quantitative part-segmentation results for selected objects.
6 Applications
The scene-parsing networks support hierarchical semantic segmentation, automatic object removal, and scene synthesis from semantic masks.
- Hierarchical semantic segmentation: ADE20K’s WordNet object hierarchy enables progressive merging of the 150 categories into increasingly general semantic concepts.
- Hierarchical semantic segmentation: The hierarchy can detect general concepts such as furniture by merging hyponyms including chairs, tables, benches, and bookcases.
- Automatic image content removal: Predicted object probability maps automatically identify target regions for removal, after which image completion or inpainting fills the resulting holes.
- Scene synthesis: Scene parsing networks can predict semantic masks that condition an image generator to synthesize scenes with similar spatial configurations of visual concepts.
7 Conclusion
The paper introduces ADE20K as a densely annotated, diverse dataset covering stuff, objects, and parts, and establishes scene-parsing and instance-segmentation benchmarks with public resources.
- ADE20K densely annotates stuff, objects, and parts across diverse visual concepts in scenes.
- A single annotator provided precise object boundaries and consistent object naming across images.
- The paper constructs scene-parsing and instance-segmentation benchmarks, organizes challenges, and evaluates state-of-the-art models.
- The dataset and pretrained models are released publicly.