PaperlayerView exact revision on arXiv

Source-linked AI summary

Amodal Instance Segmentation

Ke Li, Jitendra Malik

arXiv:1604.08202v2cs.CV

TL;DR

Amodal instance segmentation must infer both visible and occluded object regions despite ambiguity and a lack of public amodal annotations. The paper trains the first reported method using modal annotations with synthetic occlusions and infers amodal boxes iteratively. The method is reported to predict plausible amodal masks on real occlusions and to be effective qualitatively and quantitatively.

  • Problem

    Amodal segmentation is difficult because hidden object regions are ambiguous, while publicly available supervised amodal segmentation data is unavailable.

  • Method

    The method overlays objects to synthesize occlusions, trains on existing modal annotations, and uses Iterative Bounding Box Expansion to infer amodal boxes.

  • Results

    The resulting model is reported to be effective at predicting plausible amodal masks for images containing real occlusions, despite synthetic training occlusions.

  • Takeaways & Limitations

    The paper demonstrates that amodal instance segmentation can be trained from modal annotations and evaluated qualitatively and quantitatively.

  • Takeaways & Limitations

    The evaluation assumes the modal bounding box and object category are given, and errors can arise from unusual poses, varied hidden configurations, similar adjacent objects, or incorrect modal predictions.

Abstract

from arXiv · show

We consider the problem of amodal instance segmentation, the objective of which is to predict the region encompassing both visible and occluded parts of each object. Thus far, the lack of publicly available amodal segmentation annotations has stymied the development of amodal segmentation methods. In this paper, we sidestep this issue by relying solely on standard modal instance segmentation annotations to train our model. The result is a new method for amodal instance segmentation, which represents the first such method to the best of our knowledge. We demonstrate the proposed method's effectiveness both qualitatively and quantitatively.

1 Introduction

Amodal instance segmentation predicts both visible and occluded object regions, but the task is ambiguous and lacks publicly available supervised annotations. The paper addresses this gap by training a new method solely from modal annotations and synthetic occlusions.

  • Amodal segmentation marks both visible and occluded regions, unlike modal segmentation, which marks only visible object pixels.
  • The task is inherently ambiguous because multiple hidden configurations can explain the same visible object, especially under heavy occlusion.Humans nevertheless tend to complete occluded regions consistently.
  • Amodal masks support occlusion reasoning, including estimating occlusion extent, depth ordering, and real-world object dimensions.
  • The task requires detecting weak visible evidence while avoiding misleading signals from occluding objects and hypothesizing unseen object shape.
  • Publicly available amodal training data is unavailable, motivating a method that reuses existing modal annotations.
  • The proposed method adds synthetic occlusions to modal objects, trains a network to recover the original mask, and infers amodal boxes through Iterative Bounding Box Expansion.

2 Related Work

Prior work addressed figure-ground and semantic segmentation, while amodal completion received limited attention. Before this paper, general-purpose amodal segmentation lacked algorithmic methods and publicly available annotations.

  • Figure-ground segmentation identifies foreground pixels, while semantic segmentation assigns pixels to object categories.
  • Earlier amodal-completion work predicted amodal bounding boxes or completed planar surfaces using depth information.
  • The authors report no prior algorithmic work on general-purpose amodal segmentation and note that collected annotations were not publicly available.

3 Generating Training Data

The method creates amodal training examples from standard modal annotations by compositing randomly positioned occluders over a main object. Original masks supervise recovery of the hidden object extent, with uncertain occluded pixels labeled unknown.

  • Training data consists of image patches, modal bounding boxes, and target segmentation masks generated solely from standard modal annotations.
  • Synthetic occlusion is produced by overlaying randomly cropped object instances onto a main-object patch using modal masks as alpha mattes.
  • The composite procedure makes the original main-object mask the supervisory target while other objects occlude parts of the image.
  • The visible main-object box is computed and randomly jittered to simulate noisy modal localization at test time.
  • Target pixels are labeled positive for the object, negative for background, and unknown for pixels belonging to other objects.
  • Training examples are generated on the fly by sampling objects, overlapping boxes, and overlay configurations.

4 Predicting Amodal Mask and Bounding Box

At test time, the model starts from a modal box and iteratively expands an amodal box while predicting amodal heatmaps. A convolutional network uses the image patch, modal heatmap, and category as inputs.

  • Testing assumes the modal bounding box and object category are given, with the modal heatmap computed using IIS.
  • Iterative Bounding Box Expansion initializes the amodal box as the modal box and repeatedly predicts heatmaps inside the current box.
  • The amodal box expands until average heat intensities outside it fall below the stopping threshold.
  • The network takes an image patch, modal segmentation heatmap, and category, then outputs an amodal segmentation heatmap.Its hypercolumn architecture combines fine-scale and coarse-scale features.
  • Training inputs are prepared by cropping and resizing the modal-box region, aligning and upsampling the modal heatmap, and supplying the target amodal mask.
  • The model is trained end-to-end with momentum-based stochastic gradient descent on mini-batches of 32 patches.Training starts from IIS weights and runs for 50,000 iterations.

5 Experiments

Experiments evaluate the proposed amodal segmentation method qualitatively on occluded and unoccluded objects and quantitatively through indirect and direct measurements. The method produces plausible hidden-region completions, avoids hallucinating on unoccluded objects, improves segmentation over IIS, and supports occlusion prediction from area ratios.

  • Evaluation Setup: Direct evaluation uses 100 randomly chosen occluded PASCAL VOC 2012 validation objects with newly annotated amodal masks, alongside qualitative and indirect evaluations.The indirect evaluation uses coarse occlusion annotations on the full validation set.
  • Qualitative Results: The method produces plausible amodal masks for interior and exterior occlusions, including cases where modal predictions are poor and hidden shapes are challenging.The authors attribute this qualitative capability partly to learning general object shape.
  • Qualitative Results: Incorrect predictions may result from rare poses, diverse plausible hidden configurations, similar adjacent-object appearance, or erroneous modal predictions.
  • Qualitative Results: For unoccluded objects, amodal predictions are similar to or more accurate than modal predictions, indicating that the method generally avoids hallucinating occluded regions.Because these objects are unoccluded, their amodal and modal masks should coincide.
  • Indirect Evaluation: 77.17% average precision is obtained when area ratio thresholds are used to predict the absence of occlusion.Unoccluded objects concentrate at high area ratios, while occluded objects peak around 0.75.
  • Segmentation Performance: 73% of objects receive better masks than with IIS, with overlap improvements of 20–50% in many cases, while most regressions are under 5%.
  • Segmentation Performance: The proposed method outperforms IIS at every IoU cutoff and on all reported accuracy metrics, while the combined pipeline improves by 11.1 points at 50% overlap and 8.6 points at 70% overlap.The evaluation uses modified mAPr matching based on bounding-box overlap while correctness still depends on region overlap.

6 Conclusion

The paper presents a new amodal instance segmentation method and reports qualitative and quantitative effectiveness. It uses synthetic amodal data generated from modal annotations to address the lack of public amodal segmentation data.

  • The paper presents a new method for amodal instance segmentation, described as the first such method to the authors’ knowledge.
  • Synthetic amodal instance segmentation data are generated from modal instance segmentation annotations to train the model without publicly available amodal data.
  • The method also iteratively predicts the amodal bounding box from amodal segmentations.
  • The proposed method is evaluated qualitatively and quantitatively for predicting amodal segmentation masks.

Supplementary Material

The supplementary material identifies the authors and their affiliation with the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley.

  • The paper is authored by Ke Li and Jitendra Malik.
  • The authors are affiliated with the Department of Electrical Engineering and Computer Sciences.
  • The affiliation is the University of California, Berkeley.

1 Ablation Analysis

The ablation analysis compares the original pipeline with variants removing modal segmentation prediction as input or dynamic occlusion generation. The original model largely performs better, indicating both components are important ingredients for good performance.

  • The ablation study compares the original model with variants that remove modal segmentation prediction input or dynamic sample generation.
  • The original model largely performs better than both ablation variants on the collected dataset.
  • Feeding modal segmentation prediction as input is identified as an important ingredient for achieving good performance.
  • Generating diverse occlusion configurations dynamically is identified as another important ingredient for achieving good performance.

2 Results on PASCAL 3D+

The paper evaluates the combined detection and segmentation pipeline on rigid objects in PASCAL 3D+. CAD-model projections provide approximate amodal masks because CAD shapes may mismatch the objects depicted in images.

  • Performance is evaluated on PASCAL 3D+ using CAD models aligned to rigid objects in PASCAL images.
  • Projections of CAD models onto the image plane are treated as amodal segmentation masks.
  • The CAD-derived masks are only approximately correct because CAD-model shapes may differ from the objects depicted in images.
  • Region average precision is reported for rigid objects in the PASCAL VOC 2012 validation set with available CAD models.

3 Additional Visualizations

This section provides additional visual examples of modal and amodal heatmap and segmentation-mask predictions from the proposed method and IIS.

  • Additional examples compare modal and amodal heatmap predictions produced by the proposed method and IIS.
  • Additional examples also compare modal and amodal segmentation-mask predictions from the proposed method and IIS.
Loading 1604.08202v2…