PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Amodal Segmentation

Yan Zhu, Yuandong Tian, Dimitris Mexatas, Piotr Dollár

arXiv:1509.01329v2cs.CV

TL;DR

As conventional visual recognition advances, the paper studies recognition beyond visible pixels through semantic amodal segmentation. It introduces amodal annotations and datasets, finding strong annotator agreement while establishing metrics and baselines for amodal segmentation and depth ordering.

  • Problem

    Recent progress in conventional visual recognition motivates studying scene information beyond visible pixels, including occluded regions and object relationships.

  • Method

    The paper annotates regions amodally with semantic labels and partial depth order, creating BSDS and COCO datasets with evaluation metrics and algorithmic baselines.

  • Results

    The BSDS annotations show strong agreement between annotators, while COCO experiments provide strong baselines for amodal segmentation and depth ordering.

  • Takeaways & Limitations

    Semantic amodal segmentation is a well-posed annotation task, and the datasets, metrics, and baselines define concrete challenges for future research.

Abstract

from arXiv · show

Common visual recognition tasks such as classification, object detection, and semantic segmentation are rapidly reaching maturity, and given the recent rate of progress, it is not unreasonable to conjecture that techniques for many of these problems will approach human levels of performance in the next few years. In this paper we look to the future: what is the next frontier in visual recognition? We offer one possible answer to this question. We propose a detailed image annotation that captures information beyond the visible pixels and requires complex reasoning about full scene structure. Specifically, we create an amodal segmentation of each image: the full extent of each region is marked, not just the visible pixels. Annotators outline and name all salient regions in the image and specify a partial depth order. The result is a rich scene structure, including visible and occluded portions of each region, figure-ground edge information, semantic labels, and object overlap. We create two datasets for semantic amodal segmentation. First, we label 500 images in the BSDS dataset with multiple annotators per image, allowing us to study the statistics of human annotations. We show that the proposed full scene annotation is surprisingly consistent between annotators, including for regions and edges. Second, we annotate 5000 images from COCO. This larger dataset allows us to explore a number of algorithmic ideas for amodal segmentation and depth ordering. We introduce novel metrics for these tasks, and along with our strong baselines, define concrete new challenges for the community.

1. Introduction

The paper proposes semantic amodal segmentation as a next frontier for visual recognition, capturing full regions, occlusion, depth order, and semantic structure beyond visible pixels. It introduces BSDS and COCO datasets, reports strong annotation agreement, and establishes metrics and baselines for new algorithmic challenges.

  • Proposed task: Semantic amodal segmentation marks each region’s full extent, including visible and occluded portions, rather than only visible pixels.Annotators also name salient regions and specify a partial depth order, yielding semantic labels, figure-ground edges, and object overlap.
  • BSDS dataset: 500 BSDS images were labeled by multiple annotators, producing surprisingly strong agreement for regions, edges, and amodal completions.The study compares this consistency with the original BSDS labels and also evaluates state-of-the-art edge detectors.
  • COCO dataset: 5000 COCO images were annotated by one expert each with quality control and split into 2500/1250/1250 train/validation/test images.The dataset includes novel metrics for amodal segment quality and pairwise depth ordering; semantic labels were not used for evaluation because they come from an open vocabulary.
  • Algorithmic baselines: Both amodal segmentation baselines achieve large gains over modal counterparts, especially under heavy occlusion, while depth-ordering models exceed 80% accuracy.The baselines directly predict amodal masks or expand modal masks into amodal ones.
  • Related work: Unlike prior modal or small-scale datasets, the proposed annotations combine dense segmentation with occlusion, figure-ground ordering, and semantic structure in natural images.The paper notes that no large-scale dataset of amodally segmented natural images was previously available and contrasts its scope with related resources.
  • Research scope: The annotation scheme subsumes modal segmentation, edge detection, and figure-ground edge labeling, while COCO is an order of magnitude larger than BSDS.The authors position the datasets, metrics, and baselines as concrete challenges intended to stimulate research.

2. Dataset Annotation

The annotation pipeline extends an existing polygon-labeling tool to collect semantically meaningful, dense, amodal regions together with depth order and shared boundaries. These guidelines make occlusion and object relationships explicit while supporting consistent scene annotations.

  • Tool: The authors extend the Open Surfaces tool with region ordering, naming, improved editing, and features supporting semantic amodal segmentation.The updated tool is intended to facilitate annotation and will be open-sourced.
  • Annotation guidelines: Four guidelines require semantically meaningful regions, dense coverage, complete depth ordering, and explicit marking of shared boundaries.Together, these requirements encourage annotators to reason about object interactions and scene geometry.
  • Semantic annotation: Regions must be namable, so material boundaries and non-namable object parts are excluded from the annotation.This constrains annotation granularity and is intended to improve consistency between annotators.
  • Dense annotation: All foreground objects larger than 600 pixels are labeled, including occluders of annotated regions.With dense labeling and depth order, visible and occluded portions and visible and hidden edges can be determined.
  • Depth ordering: For overlapping regions, annotators place the occluder before the occludee; non-overlapping regions may use any depth order.In ambiguous cases, ordering is chosen so edges render correctly, such as eyes in front of a face.
  • Edge sharing: Shared boundaries between adjacent regions are marked explicitly because they lack a figure-ground side and should not be duplicated.This differs from an occlusion edge, which belongs to the foreground region.

3. Dataset Statistics

The BSDS and COCO annotations provide detailed statistics on semantic amodal regions, including coverage, occlusion, labels, shape complexity, edges, and scene depth structure.

  • 500 BSDS images were annotated by 5–7 annotators, averaging 7.3 labeled regions and 64 polygon points per image.About 84% of pixels are covered by at least one region polygon.
  • Occlusion: 62% of regions are partially occluded, with average occlusion of 21%.Most regions are slightly occluded, while a small portion are heavily occluded.
  • Semantic labels: The dataset contains both ‘things’ and ‘stuff’ labels, with stuff comprising about a quarter of all regions.Most labels correspond to basic-level categories and entire objects rather than object parts.
  • Shape complexity: Amodal regions are simpler in shape than modal regions, and their simplicity can make them more efficient to label.The analysis uses convexity and simplicity statistics, each maximized at 1.0 for a circle.
  • Edge density: The annotation has fewer visible edges than original BSDS because material boundaries and interior object-part edges are excluded.Despite lower edge density, the resulting edge maps can train effective edge detectors.
  • Scene complexity: Depth-order graphs summarize scene complexity through connected components and depth layers, whose required depth layers appear to grow logarithmically with component size.Most components require only a few depth layers, although some are substantially more complex.

4. Dataset Consistency

The proposed amodal annotations are more consistent than original modal BSDS annotations for both regions and edges, while remaining challenging for current machines.

  • Region consistency: Amodal region consistency has median F score 0.723 versus 0.425 for original modal BSDS regions.This higher agreement holds despite amodal regions including visible and occluded portions.
  • Edge consistency: Amodal edge consistency has median score 0.795 versus 0.728 for original BSDS edges.The amodal edges are more consistent even though they are less dense.
  • Human performance: Human performance reaches precision/recall/F-Score of .98/.83/.90 on the amodal test set, compared with .92/.73/.81 on original BSDS.Human performance is evaluated using one annotation against the union of the others as ground truth.
  • Human–machine gap: HED achieves ODS .69 on the amodal annotations versus human F score .90, leaving substantial room for improvement.On original BSDS, HED achieves ODS .79 versus human F score .81, a gap of just .02.

5. Metrics and Baselines

The paper evaluates amodal segmentation and pairwise depth ordering with dedicated metrics, COCO baselines, and qualitative analyses. Amodal methods improve under occlusion, while learned ordering outperforms simple heuristics.

  • Datasets and metrics: Amodal segment quality is measured by average recall across IoU thresholds 0.5-0.95, using 1000 segments per image and separate occlusion levels.The evaluation reports AR for things and stuff as well as none, partial, and heavier occlusion conditions.
  • Amodal segmentation baselines: ExpandMask expands a SharpMask modal mask, whereas AmodalMask directly predicts amodal masks from image patches.Both models share SharpMask’s architecture, with ExpandMask adding a mask input channel and larger input size.
  • Amodal segmentation results: With more occlusion, amodal baselines outperform SharpMask, but they are worse on unoccluded objects; synthetic data improves occluded-region AR but trails real-data training.Human accuracy remains substantially higher on this task.
  • Depth ordering: Pairwise depth ordering evaluates which of two overlapping masks is in front, using matched predicted masks with IoU of at least 0.5 to decouple ordering from segmentation quality.Ordering models are tested across multiple segmentation sources.

6. Discussion

The paper presents semantic amodal segmentation as a dataset-based framework for studying perceptual grouping, occlusion reasoning, and scene structure. It concludes that the task is well posed and supplies metrics and baselines for future research.

  • Discussion: The dataset marks both visible and occluded portions of regions to support amodal perception and reasoning about object interactions and scene structure.It is designed to study perceptual grouping tasks.
  • Discussion: Extensive analysis shows that semantic amodal segmentation is a well-posed annotation task.The paper also provides evaluation metrics and strong baselines for the proposed tasks.
  • Discussion: The authors hope the dataset will stimulate new research directions for the community.

A.1. Annotation Tool

The annotation tool extends polygon-based region labeling with controls for amodal segmentation, depth ordering, semantic naming, shared edges, and polygon editing. The modified tool is intended to make the annotation process intuitive and reproducible.

  • Annotation Tool: The tool is extended from Open Surfaces to support semantic amodal segmentation and facilitate annotation.The original interface labels multiple regions using closed polygons.
  • Annotation Tool: Annotators specify depth order with a draggable ordered list and receive visual feedback from region fills overlaid on the image.
  • Annotation Tool: The same ordered list lets annotators enter free-form names for every segment, and annotation is incomplete until all segments are named.
  • Annotation Tool: Polygon snapping lets new vertices attach to existing edges or vertices, enabling shared-edge annotation.
  • Annotation Tool: The interface adds controls for inserting and removing vertices while editing existing polygons.
  • Annotation Tool: The authors will release the modified annotation-tool code.

A.2. Corner Cases

The annotation protocol defines special handling for holes, blurry background objects, intertwined depth, groups, and truncation. These rules limit or simplify annotations where full amodal interpretation is difficult.

  • Corner Cases: Regions with holes are represented by one segment using only exterior boundaries; holes are ignored.
  • Corner Cases: For blurry background objects, annotators label only the most salient objects individually rather than every detail.
  • Corner Cases: When two regions lack a valid depth ordering, annotators choose the least-wrong order and may label object parts to preserve visibility and occlusion information.
  • Corner Cases: Groups of similar objects are annotated as one enclosing region because they are often perceived as a single visual entity.
  • Corner Cases: Regions extending beyond image boundaries are not annotated amodally because the unseen occluder is unavailable.

A.3. Annotators

The annotation process uses expert workers, quality monitoring, and manual correction of explicit instruction violations while preserving subjective judgments. Training edge detectors with the resulting annotations produces sparser, more semantically meaningful edges than original BSDS annotations.

  • A.3. Annotators: Expert workers complete annotations, with iterative quality checks and manual review used to reach sufficient quality.The process supports complex instructions and allows obvious errors to be returned for correction.
  • A.3. Annotators: Obvious errors are corrected, whereas subjective judgments remain at annotators’ discretion.Examples of obvious errors include incorrect depth ordering and modal regions; subjective judgments include hidden-edge location and semantic labels.
  • A.3. Annotators: Original BSDS annotations produce dense HED edge maps with interior edges, while the paper’s annotations yield sparser, more semantically meaningful edges.This qualitative pattern holds when training with annotations from either one or five annotators per image.
  • A.3. Annotators: The Table 4 setup varies training rows and testing columns by annotator count, showing that annotator number affects testing more than training.Switching from BSDS to COCO has only a minor effect on SE but a larger effect on HED.

B. Appendix: Edge Detection on COCO

The appendix evaluates SE and HED on COCO using COCO train images and validation images, while examining how annotation count affects edge-detection measurement. It finds that test accuracy is highly sensitive to annotator count, whereas training is comparatively stable, and COCO presents a substantial challenge.

  • B. Appendix: Edge Detection on COCO: SE and HED are trained on 2500 COCO train images and evaluated on the 1250-image COCO validation set.The appendix reports these state-of-the-art edge detectors on COCO, complementing results previously presented for BSDS.
  • B. Appendix: Edge Detection on COCO: Edge-detection metrics use the union of human annotations as ground truth, so adding annotators produces denser evaluation edges.This makes the number of annotators part of the testing setup rather than merely a data-collection detail.
  • B. Appendix: Edge Detection on COCO: Reducing annotators during testing lowers ODS substantially, even with identical models, while reducing annotations during training leaves results largely unchanged.The comparison isolates annotation-count effects across test and train conditions.
  • B. Appendix: Edge Detection on COCO: COCO and BSDS results are quite similar after accounting for annotator count, despite COCO appearing worse under unadjusted accuracy measurement.The paper attributes the apparent gap to the evaluation artifact and notes that COCO has 10× more images than BSDS.
  • B. Appendix: Edge Detection on COCO: The COCO annotations provide a substantial challenge for current state-of-the-art edge detectors.The appendix reports complete SE and HED results on the COCO validation set and qualitative HED comparisons across training-data choices.
Loading 1509.01329v2…