PaperlayerView exact revision on arXiv

Source-linked AI summary

Amodal Instance Segmentation

Ke Li, Jitendra Malik

arXiv:1604.08202v2cs.CV

TL;DR

Amodal segmentation must infer hidden object regions, but publicly available supervised annotations are lacking. This paper trains a model from modal annotations using synthetic occlusions, and reports effective amodal mask prediction on real occlusions despite synthetic training data.

  • Problem

    Amodal segmentation must hypothesize hidden object regions with multiple plausible configurations, while publicly available supervised training annotations are lacking.

    Source: §1 · pp. 1–2

  • Method

    The method creates synthetic occlusions from modal instance annotations and trains a model to predict amodal masks.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Despite training on synthetic occlusions, the method is quite effective at predicting plausible amodal masks for real occlusions.

    Source: §1 · p. 3, §5.1 · p. 8

  • Takeaways & Limitations

    The work demonstrates effective amodal mask prediction without publicly available amodal training data.

    Source: §6 · p. 14

    At test time, the method takes the visible-part bounding box and object category as given.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

We consider the problem of amodal instance segmentation, the objective of which is to predict the region encompassing both visible and occluded parts of each object. Thus far, the lack of publicly available amodal segmentation annotations has stymied the development of amodal segmentation methods. In this paper, we sidestep this issue by relying solely on standard modal instance segmentation annotations to train our model. The result is a new method for amodal instance segmentation, which represents the first such method to the best of our knowledge. We demonstrate the proposed method's effectiveness both qualitatively and quantitatively.

1 Introduction

Amodal instance segmentation predicts both visible and occluded object regions, but publicly available amodal training annotations were lacking. The paper introduces a method trained only on modal annotations by adding synthetic occlusion and recovering original masks, with amodal bounding boxes inferred through Iterative Bounding Box Expansion.

1 Introduction

  • Modal segmentation marks visible regions, whereas amodal segmentation includes visible and occluded regions; hidden configurations can be ambiguous, although humans predict them consistently [40].The introduction illustrates ambiguity with a partly occluded person whose hidden legs could plausibly be sitting or standing.Source: §1 · p. 1
  • Amodal masks can support occlusion reasoning about presence, extent, boundaries, regions, and relative depth, as well as real-world object-dimension estimates from amodal boxes.Source: §1 · pp. 1–2
  • Amodal segmentation must infer whether and where objects are occluded and hypothesize hidden shapes while detecting weak visible signals without being misled by occluders.Source: §1 · p. 2
  • No amodal segmentation data was publicly available at the time of writing, motivating training from existing modal segmentation data.The paper notes efforts to collect amodal annotations were underway.Source: §1 · p. 2
  • The first proposed amodal instance segmentation method trains a CNN on modal annotations by adding synthetic occlusion and recovering the original mask, then infers boxes with Iterative Bounding Box Expansion.The method retains the original mask as the composite image’s amodal target and does not assume an amodal bounding box at test time.Source: §1 · pp. 2–3
  • Despite synthetic-data training, the resulting model is described as effective at predicting amodal masks on images with real occlusions.Source: §1 · p. 3

2 Related Work

Prior work progresses from figure-ground and semantic segmentation to instance-level masks, while general-purpose amodal segmentation remains largely unexplored. The paper positions amodal instance segmentation as the next step and its method as the first for this task.

2 Related Work

  • Segmentation research progressed from figure-ground foreground identification through semantic segmentation to instance masks, making amodal instance segmentation a natural next step.Instance methods typically detect each object's bounding box and predict a figure-ground mask within it; newer approaches use convolutional neural nets.Source: §2 · p. 3
  • Prior amodal-completion work addressed amodal bounding boxes and depth-guided completion of occluded planar surfaces, but not general-purpose amodal segmentation.Zhu et al. [40] collected amodal segmentation annotations on BSDS images, but had not made them publicly available.Source: §2 · p. 3
  • The authors characterize their method as the first for amodal segmentation, to their knowledge.They note that no algorithmic work on general-purpose amodal segmentation had been reported, despite annotation collection efforts.Source: §2 · p. 3

3 Generating Training Data

The method creates amodal training examples from standard modal annotations by simulating occlusion, then uses visible-object labels and unknown occluder pixels to supervise mask prediction. Training examples are generated on the fly with randomized crops, overlays, and jittered modal boxes.

3.1 Overview

  • Training data comes solely from SBD annotations on PASCAL VOC 2012 train, yielding image patches and modal boxes as inputs and target masks as supervision.Source: §3.1 · p. 4
  • Occlusion is simulated by overlaying randomly selected objects onto a crop containing a main object, producing composite patches where that object is partially occluded.Overlay objects are positioned and scaled to create moderate overlap with the main object.Source: §3.1 · pp. 4–5
  • For each composite patch, the smallest box enclosing the still-visible main object is jittered to simulate noisy modal localization at test time.Source: §3.1 · p. 5
  • Target masks mark main-object pixels positive, background negative, and other-object pixels unknown, preserving originally visible but now occluded object regions as amodal targets.Unknown labels reflect that the main object may be occluded by other objects, while background cannot occlude it.Source: §3.1 · p. 5

3.2 Implementation Details

  • On-the-fly sampling uses 0–2 overlays, requires at least 70% box overlap per dimension, and rejects operations that leave under 30% of the main object visible.The sampled crop dimensions range from 70% to 200% of the object box; the final jittered box overlaps the visible-object box by at least 75% per dimension and differs in size by at most 10%.Source: §3.2 · p. 5

4 Predicting Amodal Mask and Bounding Box

The method predicts amodal masks and bounding boxes by iteratively expanding the modal box, while training a heatmap-prediction network using modal annotations and target amodal masks.

4.1 Testing

  • Starting from the modal box and category from a detector,, or, iterative expansion uses outside-box heat intensity to grow the amodal box until all directions fall below 0.1.The network predicts a mask heatmap within an expanded box at each iteration; pixels above 0.7 form the final amodal mask, while the modal mask uses a threshold of 0.8.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 Training

  • The network takes an image patch, modal heatmap, and category to predict an amodal heatmap, using an IIS variant of the VGG-based hypercolumn O-Net architecture, [33].Its category-dependent input channel accepts an initial heatmap hypothesis, enabling iterative refinement by feeding the preceding heatmap prediction back into the model.Source: §4.2 · p. 7
  • Training examples pair an image patch, modal box, and target amodal mask; inputs are resized to 224 × 224, and patches with under 10% visible-object pixels are rejected.The modal heatmap is produced by one IIS iteration and aligned to the patch; the model sees a patch enlarged by removing 10% from each side.Source: §4.2 · pp. 7–8
  • The model is trained end-to-end from IIS weights with SGD on 32-patch batches for 50,000 iterations, using pixelwise negative log likelihood, learning rate 10−5, weight decay 10−3, and momentum 0.9.Instance-specific loss weights are inversely proportional to each patch’s upsampling factor.Source: §4.2 · p. 8

5 Experiments

Because no public amodal segmentation annotations exist, the experiments combine qualitative results, indirect evaluation on the full PASCAL VOC 2012 validation set, and direct evaluation on a newly annotated 100-object subset.

5.1 Qualitative Results

  • Predictions plausibly complete interior and exterior occlusions despite unrealistic synthetic occlusions, and unoccluded predictions are similar to or more accurate than modal predictions.Exterior occlusions are ambiguous and require category-shape knowledge to choose mask extensions; errors may stem from rare poses, varied hidden configurations, similar adjacent objects, or erroneous modal predictions.Source: §5.1 · pp. 8–9

5.2 Indirect Evaluation

  • 77.17% average precision is achieved by classifying objects as unoccluded when their area ratio exceeds a threshold; occluded objects peak around 0.75 while unoccluded objects skew toward higher ratios.The area ratio measures the proportion of pixels in an amodal mask that also belong to the modal mask; its flatter distribution for occluded objects reflects variation in occlusion amount.Source: §5.2 · pp. 11–12, Figure 8 · p. 12

5.3 Direct Evaluation

  • 100 randomly selected occluded VOC objects, five per category, were annotated with amodal masks for direct IoU evaluation against IIS modal-mask predictions.IIS is a fairly strong baseline when occlusion is not severe, since omitting hidden portions may not substantially reduce IoU.Source: §5.3 · pp. 12–13, Segmentation Performance · p. 13

Segmentation Performance

  • 73% of objects have more accurate proposed-method masks than IIS, and the proposed method performs better on all reported accuracy metrics and at every IoU cutoff.Overlap improves by 20–50% in many cases; for most of the remaining objects, the overlap drop is less than 5%.Source: Table 1 · p. 12, Figure 9 · p. 13, Segmentation Performance · p. 13

Combined Detection and Segmentation Performance

  • 11.1 points at 50% overlap and 8.6 points at 70% overlap: the Faster R-CNN pipeline with the proposed segmenter outperforms Faster R-CNN with IIS in mAPr.Because not all instances have amodal masks, mAPr uses bounding-box overlap to assign predictions to ground-truth instances but region overlap to determine correctness.Source: Table 2 · p. 14, Combined Detection and Segmentation… · p. 14

6 Conclusion

The paper presents what it describes as the first method for amodal instance segmentation, trained using synthetic data generated from modal annotations despite the lack of public amodal annotations. It also introduces iterative amodal bounding-box prediction and reports qualitative and quantitative effectiveness.

6 Conclusion

  • The proposed method predicts amodal segmentation masks and is demonstrated effective both qualitatively and quantitatively.The authors describe it as the first amodal instance segmentation method to their knowledge.Source: §6 · p. 14
  • A novel strategy generates synthetic amodal segmentation data from modal annotations, enabling model training without publicly available amodal data.Source: §6 · p. 14
  • The paper also introduces an approach that iteratively predicts amodal bounding boxes from amodal segmentations.Source: §6 · p. 14

1 Ablation Analysis

The ablation compares the original model with variants that remove modal segmentation prediction or dynamic occlusion-sample generation. The original performs better than these variants, indicating that both components contribute to good performance.

Ablation Analysis

  • The original model largely outperforms variants without modal segmentation prediction or dynamic sample generation on the collected dataset.The comparison uses faster R-CNN for detection and different proposed-method variants for segmentation.Source: §1 · p. 18, Table 1 · p. 18
  • Replacing the modal heatmap input with an amodal heatmap produced by a separately trained model defines the variant without modal segmentation prediction.That separate model uses the same architecture as IIS but predicts amodal masks directly.Source: §1 · p. 18
  • Using one fixed occlusion configuration per object instead of varied configurations defines the variant without dynamic sample generation.The varied configurations change the identities, positions, and scales of occluding objects.Source: §1 · p. 18

2 Results on PASCAL 3D+

The section evaluates amodal segmentation on rigid objects in PASCAL 3D+ using CAD-model projections as approximate masks. It reports region average precision on the PASCAL VOC 2012 validation subset with available CAD models.

2 Results on PASCAL 3D+

  • PASCAL 3D+ provides CAD models aligned to rigid objects in PASCAL images, whose image-plane projections serve as approximate amodal masks.The masks are only approximate because CAD-model shapes can mismatch the objects depicted in images.Source: §2 · pp. 18–19
  • The evaluation reports region average precision on rigid objects in the PASCAL VOC 2012 validation subset for which CAD models are available.Table 2 compares the combined detection and segmentation pipeline on these objects and annotations.Source: §2 · p. 19, Table 2 · p. 19

3 Additional Visualizations

This section presents additional modal and amodal prediction examples from the proposed method and IIS, covering heatmaps and segmentation masks.

3 Additional Visualizations

Related papers

Loading 1604.08202v2…