PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentation

Patrick Follmann, Rebecca König, Philipp Härtinger, Michael Klostermann

arXiv:1804.08864v1cs.CV

TL;DR

Semantic amodal segmentation must infer invisible portions of object instances, a capability needed to reason about occlusion. The paper presents an end-to-end model that predicts amodal, visible, and invisible masks together, outperforming the COCO amodal baseline and establishing strong results on new datasets. It also shows that augmentation can support amodal segmentation without amodal training annotations.

  • Problem

    Semantic amodal segmentation requires predicting invisible regions in addition to instance masks, but extending visible masks through occlusions is difficult because hidden parts may lack visual cues.

  • Method

    The paper introduces an all-in-one end-to-end model that predicts amodal, visible, and invisible masks for each instance in a single forward pass, supported by new D2S amodal and COCOA cls datasets.

  • Results

    The model outperforms previous work on COCOA and establishes strong benchmarks on COCOA cls and D2S amodal, while predicting invisible masks even without visual cues.

  • Takeaways & Limitations

    Amodal instance segmentation can be learned for occluded objects, including on D2S amodal without amodal training data when suitable augmentation is used.

  • Takeaways & Limitations

    Invisible-mask prediction remains difficult and requires further research to reduce false-positive occlusion predictions.

Abstract

from arXiv · show

Semantic amodal segmentation is a recently proposed extension to instance-aware segmentation that includes the prediction of the invisible region of each object instance. We present the first all-in-one end-to-end trainable model for semantic amodal segmentation that predicts the amodal instance masks as well as their visible and invisible part in a single forward pass. In a detailed analysis, we provide experiments to show which architecture choices are beneficial for an all-in-one amodal segmentation model. On the COCO amodal dataset, our model outperforms the current baseline for amodal segmentation by a large margin. To further evaluate our model, we provide two new datasets with ground truth for semantic amodal segmentation, D2S amodal and COCOA cls. For both datasets, our model provides a strong baseline performance. Using special data augmentation techniques, we show that amodal segmentation on D2S amodal is possible with reasonable performance, even without providing amodal training data.

1 Introduction

Semantic amodal segmentation extends instance segmentation by predicting each object's invisible occluded regions alongside its visible and amodal masks. The paper introduces an end-to-end model and datasets for jointly learning these predictions, including settings without amodal training annotations.

  • Motivation: Semantic amodal segmentation predicts each instance's amodal mask, visible mask, occluded regions, and corresponding occluder.The task combines instance segmentation with reasoning about invisible object parts.
  • Definitions: The amodal mask is the union of an object's visible mask and invisible occlusion mask.
  • Motivation: Predicting visible, invisible, and amodal masks simultaneously enables occlusion-region estimation for object-access decisions.The predicted regions can indicate which objects must be removed or how to move to access a target.
  • Challenges: Invisible-mask prediction is difficult because occlusion may remove contour cues or leave ambiguous same-category features around the hidden region.
  • Contributions: The paper proposes an all-in-one end-to-end trainable model that predicts amodal, visible, and occlusion masks for every instance in one forward pass.
  • Contributions: The authors introduce D2S amodal with complete high-quality amodal, visible, and invisible annotations, depth ordering, and 60 object categories.
  • Results: The model outperforms the current COCO amodal baseline, establishes a strong D2S amodal baseline, and can train without amodally annotated data using augmentation.

2 Related Work

Prior work addressed amodal completion for semantic regions, 3D surfaces, bounding boxes, and visible instance masks, but generally did not predict occluded instance regions. This work instead targets class-specific 2D object instances with a lightweight end-to-end model predicting visible, invisible, and amodal masks together.

  • Semantic Segmentation and 3D Scene Reconstruction: Earlier semantic-segmentation methods completed occluded background regions, while 3D-reconstruction methods completed occluded semantic surfaces.
  • Scope: This paper differs by focusing on object instances in 2D images rather than background regions or 3D object surfaces.
  • Object Detection: Amodal bounding-box methods estimated object extent and depth-related properties without predicting object masks or occluded parts.
  • Instance Segmentation: Visible instance-segmentation methods predicted categories and visible masks, while related approaches modeled depth ordering or improved visible masks without producing occlusion or amodal masks.
  • Amodal Instance Segmentation: Early amodal instance-segmentation work used iterative box enlargement or modal data augmentation because amodal ground truth was unavailable.
  • Amodal Instance Segmentation: Unlike prior methods, the proposed model is class-specific, end-to-end trainable, lightweight, and predicts amodal, visible, and invisible masks in one forward pass.

3 End-to-End Architecture for Prediction of Amodal, Visible, and Invisible Masks

ORCNN extends Mask R-CNN to predict amodal, visible, and occlusion masks for each instance in one forward pass. Its evaluation combines amodal and visible-mask criteria while treating invisible-mask quality separately because empty or small invisible regions complicate AP.

  • Architecture: ORCNN extends Mask R-CNN with amodal and occlusion mask heads alongside the visible mask head.
  • Architecture: The amodal and visible heads share RoI features and box proposals, ensuring both masks correspond to the same object prediction.The proposals use amodal-instance bounding boxes, so the visible head predicts visible masks from amodal boxes.
  • Architecture: The occlusion head derives occlusion logits by subtracting ReLU-processed visible logits from amodal logits.The ReLU prevents occlusion predictions where neither amodal nor visible masks are predicted.
  • Training: The total training loss sums classification, box, amodal-mask, visible-mask, and invisible-mask losses.Although the invisible-mask loss is theoretically redundant, it aligns the scales of amodal and visible logits.
  • Evaluation: The extended AP evaluates amodal and visible masks independently and jointly, while invisible-mask quality is evaluated separately on occluded objects.The combined APAV criterion requires correct class, amodal IoU, and visible IoU; invisible masks are excluded indirectly because empty or small regions make all-mask AP problematic.
  • Evaluation: Models without visible or invisible predictions reuse amodal masks as visible-mask predictions when computing APAV and APV.

4 Experiments

The experiments evaluate amodal and multi-task mask prediction on COCOA, COCOA cls, and D2S amodal, including architectural comparisons, data augmentation, and annotation limitations. ORCNN predicts amodal, visible, and invisible masks together, providing strong results but with trade-offs in amodal accuracy and difficult invisible-mask prediction.

  • COCOA: AmodalMRCNN outperforms the AmodalMask baseline by a large margin in average precision on COCOA while predicting only 30 results per image on average.AmodalMRCNN also achieves higher recall than AmodalMask, which predicts 1000 regions per image.
  • COCOA: COCOA data augmentation with artificially overlaid objects does not improve performance, consistent with prior augmented-training results.The models overfit after one epoch, but synthetically enlarging the training set does not improve results.
  • COCOA: ORCNN improves visible-mask quality while predicting occlusion masks, but its amodal-mask results are slightly worse than the single-task AmodalMRCNN.The comparison evaluates ORCNN against models whose amodal masks are also used as visible-mask predictions.
  • COCOA cls: COCOA cls models perform much better than on COCOA, while AmodalMRCNN outperforms ORCNN and class-specific mask proposals help on most measures.For occluded objects, class-agnostic ORCNN achieves the best average precisions for visible and invisible masks; however, incomplete annotations make AP interpretation difficult.
  • D2S amodal: D2S amodal failure cases include false-positive occlusions from reflections or lighting, extension into touching neighboring instances, and missed occlusions.Invisible-mask prediction is difficult for both small and large ground-truth invisible regions because small IoU differences or incorrect shapes reduce overlap.
  • D2S amodal: On D2S amodal, ORCNN is the best compromise for jointly predicting visible and invisible masks, with highest APAV and APV but slightly lower APA than an amodal-only model.ORCNN trained only on modal augmented data performs slightly worse than ORCNN trained on D2S amodal data, unlike the COCOA augmentation result.

5 Conclusion

The paper presents ORCNN as an end-to-end trainable model for semantic amodal segmentation and reports strong benchmark performance. Its results also show that invisible-mask prediction remains difficult and needs further research to reduce false-positive occlusions.

  • ORCNN is an end-to-end trainable, instance-aware model for semantic amodal segmentation.
  • ORCNN predicts amodal, visible, and invisible masks together with each instance’s category in one forward pass.
  • ORCNN outperforms previous work on COCOA and establishes strong benchmarks on COCOA cls and D2S amodal.
  • Invisible-mask prediction remains difficult because the results contain false-positive occlusion predictions that require further research.

Appendix

The supplementary material provides training details and qualitative results for the evaluated amodal segmentation datasets.

  • The supplementary material includes a description of the training configuration.
  • It also includes qualitative results on COCOA and D2S amodal.

A Training Setup

The training setup uses Detectron-based initialization, dataset-specific output-layer handling, fixed batch and proposal settings, and a scheduled learning rate.

  • AmodalMRCNN and ORCNN are trained using the Detectron framework built on caffe2 and Caffe.
  • Training uses two NVIDIA GTX 1080 Ti GPUs, with one image per GPU on D2S amodal and two images per GPU on COCOA and COCOA cls.
  • The region-of-interest batch size is 256, and the region proposal network permits up to 1000 RoIs per image per feature-pyramid level before suppression.
  • Models are initialized from COCO-pretrained weights, while dataset-specific class-specific output layers are randomly initialized for COCOA and D2S amodal.
  • Training runs for 10000 iterations with learning rate 0.0025, weight decay 0.0001, and learning-rate drops at 6000 and 8000 iterations.

B Qualitative Results

The qualitative evaluation compares ORCNN’s simultaneous amodal, visible, and invisible mask predictions with AmodalMRCNN’s amodal-only predictions across three datasets.

  • Qualitative results are shown for COCOA no stuff, COCOA cls, and D2S amodal.
  • ORCNN is compared with AmodalMRCNN, which predicts only amodal masks.

B.1 COCOA

On COCOA, qualitative evaluation is complicated by incorrect or incomplete ground-truth annotations, while AmodalMRCNN generally produces slightly better amodal regions than ORCNN. Occlusion predictions appear promising despite relatively low APIV values.

  • Ground-truth annotations in COCOA are sometimes wrong or incomplete, complicating qualitative interpretation.This issue is attributed to large-scale Mechanical Turk annotation and, for COCOA cls, the merging strategy.
  • AmodalMRCNN generally predicts slightly better amodal regions than ORCNN.The paper states that quantitative evaluation confirms this qualitative observation.
  • Occlusion predictions are often promising despite relatively low APIV values.

B.2 D2S

On D2S amodal, AmodalMRCNN and ORCNN produce qualitatively similar amodal masks, while ORCNN often localizes invisible regions correctly but achieves low IoU against their ground truth. The reported qualitative examples use class-agnostic masks, with COCOA figures shown for comparison.

  • D2S amodal: D2S amodal qualitative examples use class-agnostic mask versions of AmodalMRCNN and ORCNN.
  • D2S amodal: The amodal masks of AmodalMRCNN and ORCNN are difficult to distinguish qualitatively on D2S amodal.
  • D2S amodal: ORCNN often places invisible masks correctly, but their IoU with ground-truth invisible masks is often rather low.The paper links this mismatch to low APIV values.
  • COCOA qualitative comparisons: COCOA no stuff and COCOA cls figures compare input images, ground truth, AmodalMRCNN results, and ORCNN results.The COCOA no stuff overlays use a minimum score of 0.8, while COCOA cls uses 0.5.
  • D2S amodal: D2S amodal results similarly compare input images, ground truth annotations, and AmodalMRCNN and ORCNN outputs.The displayed D2S results use a minimum score of 0.8.
Loading 1804.08864v1…