PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Scene De-occlusion

Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, Chen Change Loy

arXiv:2004.02788v1cs.CV

TL;DR

Scene de-occlusion seeks to recover occlusion ordering and invisible object parts, but existing supervision is costly, subjective, or mismatched to real scenes. The paper introduces self-supervised PCNet-M and PCNet-C with progressive ordering, amodal, and content completion. On real-world datasets, the framework achieves comparable performance to fully supervised counterparts and supports amodal annotation conversion and occlusion-aware image manipulation.

  • Problem

    Scene de-occlusion must recover hidden object structure, but existing approaches rely on costly or subjective amodal and ordering annotations, synthetic data, or limited assumptions.

  • Method

    The framework uses self-supervised PCNet-M and PCNet-C with partial completion to progressively recover ordering, amodal masks, and occluded content.

  • Results

    Comparable performance to fully supervised counterparts is achieved on real-world datasets, with equivalent efficacy to manual annotations for converting modal annotations into amodal ones.

  • Takeaways & Limitations

    The framework supports converting modal datasets into amodal annotations and enables high-quality occlusion-aware scene manipulation and recomposition.

Abstract

from arXiv · show

Natural scene understanding is a challenging task, particularly when encountering images of multiple objects that are partially occluded. This obstacle is given rise by varying object ordering and positioning. Existing scene understanding paradigms are able to parse only the visible parts, resulting in incomplete and unstructured scene interpretation. In this paper, we investigate the problem of scene de-occlusion, which aims to recover the underlying occlusion ordering and complete the invisible parts of occluded objects. We make the first attempt to address the problem through a novel and unified framework that recovers hidden scene structures without ordering and amodal annotations as supervisions. This is achieved via Partial Completion Network (PCNet)-mask (M) and -content (C), that learn to recover fractions of object masks and contents, respectively, in a self-supervised manner. Based on PCNet-M and PCNet-C, we devise a novel inference scheme to accomplish scene de-occlusion, via progressive ordering recovery, amodal completion and content completion. Extensive experiments on real-world scenes demonstrate the superior performance of our approach to other alternatives. Remarkably, our approach that is trained in a self-supervised manner achieves comparable results to fully-supervised methods. The proposed scene de-occlusion framework benefits many applications, including high-quality and controllable image manipulation and scene recomposition (see Fig. 1), as well as the conversion of existing modal mask annotations to amodal mask annotations.

1. Introduction

Scene de-occlusion addresses the difficulty of recovering hidden object structure when scenes contain complex occlusion relationships and existing systems mainly parse visible regions. The proposed self-supervised framework progressively recovers ordering, amodal masks, and occluded content without ordering or amodal annotations.

  • Motivation: Scene understanding requires amodal perception because real scenes contain multiple objects with varying ordering and positioning, including occluded objects.Amodal perception concerns intact entity structures, including invisible parts.
  • Motivation: Scene de-occlusion recovers occlusion ordering and completes invisible parts of occluded objects, despite complex interactions among multiple occluders and occludees.These interactions can form complex occlusion graphs.
  • Motivation: Synthetic supervision creates a domain gap, whereas manual amodal annotations are subjective, noisy, laborious, and costly.These limitations motivate learning de-occlusion directly from real-world data.
  • Approach: Partial completion enables self-supervision by trimming an occludee and training recovery of its previous untrimmed mask, supporting progressive completion and ordering reasoning.Multiple-occluder completion is decomposed into sequential partial completions involving one occluder at a time.
  • Approach: PCNet-M recovers partially invisible masks, while PCNet-C fills recovered regions with RGB content; together they support scene de-occlusion without ordering or amodal annotations.The networks are trained as the framework’s two core components.
  • Approach: The inference scheme progressively recovers a directed occlusion graph, completes amodal masks using that graph, and completes invisible content under amodal-mask guidance.It takes a real-world scene and corresponding modal object masks as input.

2. Related Work

Prior work addresses ordering recovery and amodal segmentation through synthetic occluders, class-level priors, depth cues, or other auxiliary strategies. These approaches remain limited by toy-data demonstrations, unrealistic scene assumptions, unreliable depth, or absent explicit ordering.

  • Ordering Recovery: Unsupervised ordering recovery has been demonstrated on toy data, while class-level occlusion priors ignore realistic scene complexity.Other approaches use quadratic programming or additional depth cues.
  • Amodal Segmentation: Modal segmentation labels visible pixels but cannot recover de-occluded objects or their invisible regions.Amodal instance segmentation instead seeks object detection together with integrated masks.
  • Amodal Segmentation: Existing amodal approaches may paste artificial occluders for dummy supervision, while lacking explicit ordering makes complicated occlusion harder to resolve.The cited related work contrasts modal and amodal segmentation paradigms.

3. Our Scene De-occlusion Approach

The framework learns partial mask and content completion from modal masks, then progressively recovers occlusion ordering, amodal masks, and invisible content without manual ordering or amodal annotations.

  • Partial Completion Networks: Self-supervised partial completion trains PCNet-M to recover masks and PCNet-C to fill RGB content, enabling full completion through successive partial completions.The framework uses randomly trimmed masks as targets for partial completion, which approximates supervised full completion at test time.
  • Partial Completion Networks: PCNet-M switches between recovering an erased region and retaining an intact mask, discouraging over-completion when no occlusion exists.The two cases use an invading eraser for partial completion and a non-invading eraser as regularization.
  • Partial Completion Networks: PCNet-C predicts erased RGB content from the remaining object mask and image patch, rather than applying object-agnostic image inpainting.The remaining mask identifies which object should be reconstructed, while the loss combines l1, perceptual, and adversarial terms.
  • Dual-Completion for Ordering Recovery: Dual-completion recovers pairwise ordering by switching target and eraser roles and identifying the instance with the larger incremental completion as the occludee.Applying this comparison to neighboring pairs produces a directed graph whose edges encode occlusion directions; the graph need not be acyclic.
  • Amodal and Content Completion: For each target, the method finds all graph ancestors, predicts its amodal mask with PCNet-M, and uses ancestor-mask intersections to constrain PCNet-C content completion.Higher-order ancestors are included because they can indirectly occlude the target; the predicted invisible region is filled only where the amodal mask intersects the ancestors.

4. Experiments

Experiments evaluate scene de-occlusion across ordering recovery, amodal completion, amodal instance segmentation, and scene manipulation on real-world datasets. The method is competitive with supervised alternatives, supports pseudo-amodal annotation generation, and enables controllable scene synthesis.

  • Experimental setup: Experiments use KINS and COCOA, training PCNets with modal annotations and testing on held-out splits.KINS contains 7,474 training images and 95,311 instances; COCOA contains 2,500 training images and 22,163 instances.
  • Ordering recovery: Ordering recovery achieves much higher accuracy than heuristic baselines and results comparable to supervised counterparts on both benchmarks.The method also recovers cyclic directed graphs in circularly overlapped cases through pair-wise ordering.
  • Amodal completion: The ordering-grounded amodal completion method surpasses baseline approaches and is comparable to the supervised counterpart.Comparisons between ordering-grounded and non-ordering-grounded completion show the importance of using occlusion ordering; some outputs are potentially more natural than manual annotations.
  • Amodal completion: With predicted modal masks at 52.7% mAP, the method is also evaluated for amodal completion on KINS.The supplied passage reports the predicted-mask setup but not its completion score.
  • Amodal instance segmentation: Using inferred amodal annotations yields mAP 29.3%, matching manual-amodal training, while inferred masks reach mIoU 95.22% against manual annotations.This converts modal datasets into pseudo-amodal datasets for training amodal instance segmentation without manual amodal annotations.
  • Scene manipulation: Scene decomposition into isolated completed objects, background, and an occlusion graph enables high-quality manipulation by controlling object order and position.Figure 10 demonstrates scene synthesis by changing the ordering graph, including uncommon circular orderings.

5. Conclusion

The framework performs scene de-occlusion progressively without ordering or amodal annotations, achieving comparable performance to fully supervised methods on real-world datasets. It also supports occlusion-aware image manipulation and conversion from modal to amodal annotations.

  • The unified framework recovers occlusion orderings, then performs amodal and content completion progressively.
  • It achieves comparable performance to fully supervised counterparts on real-world datasets without ordering or amodal annotations.
  • The approach enables rich, high-quality manipulations including deleting, swapping, shifting, and repositioning instances.
  • The framework converts existing modal annotations to amodal annotations with equivalent efficacy to manual annotations.

A. Implementation Details

The experiments use UNet-based backbones for PCNet-M and PCNet-C, with different input and optimization configurations for mask and content completion.

  • PCNet-M uses a UNet backbone with widening factor 2, while PCNet-C uses a UNet with partial convolution layers.
  • PCNet-M training uses SGD with dataset-specific iteration schedules, batch size 256 across eight GPUs, and γ set to 0.8.
  • PCNet-C takes concatenated image and modal-mask inputs, adds adversarial loss, and is fine-tuned from a pretrained inpainting network.

B.1. Analysis on varying occlusion ratio.

Amodal completion performance decreases as the occluded area grows, but the full method surpasses baselines by a large margin under high occlusion ratios.

  • Larger occlusion ratios naturally result in lower amodal completion performance.
  • Under high occlusion ratios, Ours (OG) surpasses the baseline methods by a large margin.

B.2. Does it support mutual occlusion?

The approach does not support mutual occlusion between two objects because its object-level formulation cannot define an ordering graph for such cases.

  • The approach does not support cases where two objects are mutually occluded.
  • Mutual occlusion requires fine-grained boundary-level de-occlusion because the ordering graph cannot be defined.
  • The approach can nevertheless handle scenes with more than two objects that are cyclically occluded.

B.3. Will case 2 mislead PCNet-M?

The paper examines whether the not-to-complete training strategy can mislead PCNet-M when objects have mutual occlusions. The authors argue that boundary and junction cues let the network infer the correct ordering and completion behavior.

  • B.3. Will case 2 mislead PCNet-M?: PCNet-M learns whether to complete a target object conditioned on a surrogate occluder.The strategy uses examples where the target is alternately completed or not completed under different occlusion configurations.
  • B.3. Will case 2 mislead PCNet-M?: The evaluation includes growing occlusion ratios on the KINS testing set and mutual-occlusion cases with opposite occlusion directions.Figure 12 evaluates performance as occlusion increases, while Figure 13 marks which object occludes the other using green and red boundaries.
  • B.3. Will case 2 mislead PCNet-M?: Among the illustrated testing cases, only object A in case (b) is completed.The figure distinguishes two training cases from possible testing cases and identifies the completion decision for case (b).
  • B.3. Will case 2 mislead PCNet-M?: Shape, shared-boundary, and junction cues help PCNet-M distinguish whether the target lies above or below the surrogate occluder.These cues support different completion decisions in the two training cases and during testing.
  • B.3. Will case 2 mislead PCNet-M?: In the testing case where C is the real occluder, PCNet-M can infer that C is above A and complete A.The authors state that the network therefore favors the training configuration in which A is completed conditioned on C.

C. Visualization

The visualization shows that the de-occlusion framework supports controllable scene recomposition by enabling objects’ spatial configurations to be adjusted. The authors note that image inpainting advances could further improve visual quality.

  • C. Visualization: The framework enables free adjustment of scene spatial configurations to recompose new scenes.The visualization presents scene manipulation results, with inconspicuous changes marked by red arrows.
  • C. Visualization: Image inpainting advances could further improve the quality of the recomposed scenes.The authors connect this possibility to PCNet-C’s similar network architecture and training strategy.
Loading 2004.02788v1…