Source-linked AI summary
SeGAN: Segmenting and Generating the Invisible
Kiana Ehsani, Roozbeh Mottaghi, Ali Farhadi
TL;DR
The paper addresses how to infer the appearance of object regions hidden by occlusion, including which pixels belong to the object and how they should look. SeGAN jointly segments and paints invisible regions using a GAN and photo-realistic synthetic training data. It generates occluded appearance, outperforms segmentation and painting baselines, generalizes to natural images, and infers depth ordering.
Problem
Inferring occluded object appearance requires segmenting invisible regions and generating their appearance, while accurate large-scale training data is difficult to obtain.
Method
SeGAN jointly optimizes invisible-region segmentation and appearance generation using visible-region masks, with photo-realistic synthetic data supplying accurate annotations.
Results
SeGAN generates invisible object appearance, outperforms segmentation and GAN-based painting baselines, reliably segments natural images, and infers occluder-occludee depth ordering.
Takeaways & Limitations
The approach supports category-agnostic reconstruction of occluded regions and depth-layering inference without semantic category information.
Abstract
from arXiv · showhide
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation. In this paper, we study the challenging problem of completing the appearance of occluded objects. Doing so requires knowing which pixels to paint (segmenting the invisible parts of objects) and what color to paint them (generating the invisible parts). Our proposed novel solution, SeGAN, jointly optimizes for both segmentation and generation of the invisible parts of objects. Our experimental results show that: (a) SeGAN can learn to generate the appearance of the occluded parts of objects; (b) SeGAN outperforms state-of-the-art segmentation baselines for the invisible parts of objects; (c) trained on synthetic photo realistic images, SeGAN can reliably segment natural images; (d) by reasoning about occluder occludee relations, our method can infer depth layering.
1. Introduction
SeGAN addresses the challenge of inferring occluded object appearance by jointly segmenting invisible regions and generating their appearance. It uses photo-realistic synthetic data to support training and reports segmentation, generation, natural-image generalization, and depth-layering results.
- Occluded-object appearance inference requires identifying invisible pixels, extending visible object boundaries, and reasoning about viewpoint, occlusion, and object shape.
- The model receives an image and a visible-region mask, then predicts how occluded object regions appear after occluders are removed.
- SeGAN jointly learns invisible-region segmentation and appearance generation from an image and visible-region mask.The segmentation mask acts as an intermediate step before the generator paints selected pixels.
- Photo-realistic synthetic scenes provide large-scale, accurate training data for invisible-region boundaries and appearance generation.
- SeGAN segments and generates invisible regions, outperforms segmentation baselines, generalizes to natural images, and infers occluder-occludee depth ordering.The method is category-agnostic and does not require semantic category information.
2. Related Works
Prior work addresses visible-object recognition, occlusion reasoning, amodal completion, and image generation, but SeGAN focuses on reconstructing occluded regions through simultaneous segmentation and painting.
- Standard detection and segmentation methods target visible object regions and do not capture occlusions or provide object depth ordering.
- Earlier occlusion-reasoning methods infer partially occluded masks, occlusion edges, scene labels, layered ordering, or occluder-occludee patterns.
- Amodal segmentation methods complete occluded object masks, but prior annotations can rely on subjective human judgments; SeGAN obtains accurate ground truth by removing occluders.
- Unlike general cGAN applications and image inpainting, SeGAN combines segmentation and painting to reconstruct occluded object regions.
- Unlike semantic-layout image synthesis, SeGAN reconstructs occluded regions while performing both segmentation and painting without category information.
3. Model
SeGAN combines segmentation and painting to reconstruct occluded object regions. Its segmentation output guides a conditional generator, while joint losses train both tasks together.
- Architecture: SeGAN predicts occluded-region masks and paints their appearance jointly, with segmentation feeding the painting stage.The model has segmentation and painting parts trained jointly; the segmentation mask identifies which pixels should be reconstructed.
- Segmentation: The segmentation CNN receives an RGB image and visible-region mask, then produces a full-object mask.The network uses a four-channel input and upsamples its output to a 256 × 256 mask.
- Generator: Painting uses a conditional GAN with a Unet generator and a convolutional discriminator.The generator is an encoder-decoder with skip connections, while the discriminator has four convolutional layers and a sigmoid layer.
- Generator: The generator input encodes visible pixels, predicted invisible pixels, and background in separate channels before reconstruction.The invisible region is represented in red and the area outside the object mask in blue.
- Training objective: The training objective combines segmentation and painting losses, distinguishing errors on visible regions, invisible regions, and background.The segmentation objective uses customized binary cross entropy, while the adversarial painting objective includes L1 and L2 distance losses.
4. Dataset
DYCE is a photo-realistic synthetic dataset for occluded objects, providing visible and invisible-region annotations from controlled 3D scenes. Its rendering process supports varied viewpoints and accurate masks for training and evaluation.
- Dataset: DYCE contains photo-realistic indoor images from synthetic 3D scenes with visible- and invisible-region annotations.The dataset is designed around natural object configurations in scenes.
- Data generation: Synthetic 3D scenes provide scalable data generation and allow camera movement to create varied occlusion patterns.The authors use scene snapshots and vary camera location and viewpoint.
- Annotations: The dataset’s visible-mask annotations encode each object instance with a distinct color, while invisible regions identify the occluded object behind them.In the example, grey regions behind cushions indicate pixels belonging to the occluded grey sofa.
- Annotations: Invisible masks are obtained by comparing a scene with each object isolated against the original scene, then subtracting visible pixels from the full-object mask.This procedure separates visible and occluded portions for each object.
- Dataset split: The dataset uses 11 scenes, with 7 for training and validation and 4 for testing, without shared object instances across splits.The scenes comprise five living rooms and six kitchens.
5. Experiments
SeGAN is evaluated for joint segmentation and painting on synthetic data, using predicted or groundtruth visible-region masks. Experiments also test human preferences, natural-image generalization, and depth layering.
- Experimental setup: The evaluation uses 41,924 training objects and 27,617 test objects from 3,500 and 2,000 images, respectively.The segmentation network uses ImageNet-pretrained ResNet-18, while the painting network is randomly initialized.
- Experimental setup: SeGAN is tested with either Multipath-predicted visible-region masks or groundtruth masks to separate mask-quality effects from the joint model.Training selects the Multipath mask with the largest intersection over union with the groundtruth visible region; evaluation considers all generated masks.
- Metrics and baselines: Segmentation is evaluated on invisible, visible, and full-object masks using intersection over union, while painting uses L1 and L2 image distances.The full-object mask is defined as SF=SV ∪SI, and visible-region evaluation checks whether the input mask is distorted.
- Segmentation results: SeGAN significantly outperforms Multipath on full-object and invisible-region segmentation and also improves visible-region segmentation.Using groundtruth visible-region masks produces further gains; comparisons include Multipath, IBBE, and Pix2Pix on synthetic test data.
- Generalization and depth layering: Trained only with synthetic occlusion information, SeGAN accurately segments natural images, outperforms Multipath on invisible regions, and supports depth-layer inference.Combining natural and synthetic data worsens performance, likely because it makes training more difficult; predicted masks retain only slight visible-region degradation.
6. Conclusion
SeGAN addresses invisible-object-region completion by jointly segmenting and generating appearance, outperforming relevant baselines while generalizing from synthetic training data to natural images and depth layering.
- SeGAN jointly segments invisible regions and generates their appearance, rather than producing only binary masks.
- The method uses photo-realistic synthetic data to obtain exact invisible-region boundaries for training.
- SeGAN outperforms segmentation baselines for invisible-region segmentation and GAN-based baselines for appearance generation and painting.
- When trained on synthetic scenes, SeGAN generalizes to natural images and improves depth layering over a single-image depth-estimation baseline.