PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Scene Completion from a Single Depth Image

Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, Thomas Funkhouser

arXiv:1611.08974v1cs.CV

TL;DR

The paper addresses semantic scene completion: jointly recovering volumetric occupancy and semantic labels from a single depth image, where prior work handled these components separately. It introduces SSCNet, an end-to-end 3D ConvNet with dilated 3D context learning and trains it using the SUNCG synthetic scene dataset. The joint model outperforms isolated component methods and alternative semantic-scene-completion approaches, while context learning and synthetic data improve performance.

  • Problem

    The central problem is predicting complete volumetric occupancy and object semantics from a single depth image, beyond the visible surface.

  • Method

    SSCNet is an end-to-end 3D convolutional network that jointly predicts occupancy and semantic labels for all voxels in the camera view frustum.

  • Results

    The joint model outperforms methods addressing either component in isolation and alternative approaches on semantic scene completion.

  • Takeaways & Limitations

    Joint occupancy-semantic modeling, 3D contextual information, and synthetic volumetric training data are supported as effective components for semantic scene completion.

  • Takeaways & Limitations

    The method does not use color, limiting performance for objects missing depth and for geometrically similar objects, while reduced output resolution loses detail and small objects.

Abstract

from arXiv · show

This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG - a manually created large-scale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task.

1. Introduction

Semantic scene completion jointly infers volumetric occupancy and object categories from a single depth image, extending predictions beyond visible surfaces. SSCNet exploits the coupling between geometry and semantics with 3D context learning and dense synthetic supervision.

  • Task: Semantic scene completion predicts volumetric occupancy and object categories for every voxel in a scene view frustum from one depth image.Predictions extend beyond the projected depth surface into occluded regions.
  • Motivation: Prior approaches separately handled visible-surface labeling, geometric completion, or isolated object completion rather than the full joint scene task.
  • Motivation: Occupancy patterns and semantic labels are tightly coupled: object identity informs unobserved shape, while occupancy supports semantic recognition.The paper gives a chair behind a table as an example of semantic context aiding completion.
  • Method: SSCNet jointly trains a deep network with supervision for occupancy and semantic prediction, producing one of N+1 labels per voxel.Labels represent N object categories plus free space.
  • Method: A dilation-based 3D context module expands the receptive field efficiently, while SUNCG supplies dense volumetric annotations from more than 45,622 synthetic indoor environments.The dataset is manually created and contains labeled 3D object meshes.
  • Results: Joint occupancy-semantic prediction outperforms methods addressing either component in isolation, with 3D context learning and synthetic data improving performance significantly.

2. Related work

Prior work addressed visible-surface labeling, shape completion, model fitting, or voxel reasoning in separate or constrained settings. SSCNet instead learns features and long-range 3D context jointly from large-scale synthetic scenes.

  • RGB-D segmentation: RGB-D segmentation labels observed pixels but does not recover full object shape or labels beyond visible surfaces.
  • Shape completion: Shape-completion methods typically target single objects or fill limited holes using geometric regularity, requiring additional masks for scene application.Large missing regions can cause plane-fitting and symmetry-based methods to fail.
  • 3D model fitting: 3D model-fitting approaches retrieve meshes to complete and label scenes, but predictions depend on the quality and variety of available model libraries.Objects not represented by the library tend to be missed.
  • Voxel space reasoning: Voxel-space reasoning methods complete and label scenes with predefined features or separated feature-learning and context-modeling stages.
  • SSCNet: SSCNet jointly learns low-level features and high-level contextual information end-to-end from large-scale 3D scene data.Its large receptive field directly models long-range contextual cues.
  • Synthetic data: The synthetic dataset contains 45,622 houses and 775,574 rooms, substantially exceeding the scale of existing 3D scene datasets described here.

3. Semantic scene completion network

SSCNet encodes a single depth observation as a 3D volume, learns local geometry and multi-scale context with 3D convolutions, and predicts voxel occupancy and categories throughout the view frustum. Its design uses flipped TSDF input, dilated context, balanced sampling, and synthetic volumetric supervision.

  • Pipeline: The pipeline converts a single depth map into a 3D volume and predicts a probability distribution over occupancy and object categories for every view-frustum voxel.
  • Volumetric data encoding: TSDF encoding places the observation in the same 3D physical space as the output, with sign indicating free versus occluded regions.
  • Volumetric data encoding: The method removes projective view dependence by measuring distance to the closest point anywhere on the observed surface.
  • Volumetric data encoding: Flipped TSDF encoding uses d_flipped = sign(d)(dmax − d), concentrating the strongest gradient near surfaces for geometric feature learning.
  • Network architecture: A dilation-based 3D context module expands receptive fields without reducing resolution or increasing the parameter count, enabling long-range contextual modeling.
  • Network architecture: The network aggregates responses across scales because object categories vary substantially in physical size, from small TVs to larger beds.
  • Training: Training balances the roughly 9:1 empty-to-occupied voxel ratio by sampling 2N empty occluded voxels for each volume with N occupied voxels.
  • Training: SUNCG training examples pair rendered depth images with voxelized volumetric ground truth, while the network uses voxel-wise softmax supervision over object and empty-space classes.

4. Synthesizing training data

The paper addresses the lack of dense voxel-level semantic annotations by constructing SUNCG and generating synthetic depth observations with fully labeled volumetric ground truth.

  • SUNCG dataset: The scenes provide dense object labels through voxelization of individually labeled 3D object meshes.
  • SUNCG dataset: SUNCG contains 45,622 manually created synthetic 3D scenes with realistic room and furniture layouts.
  • Synthetic depth maps: Synthetic depth maps are rendered from camera viewpoints selected using floor-grid locations and NYU-Depth v2 camera-pose distributions.
  • Volumetric ground truth generation: Object-library instances are voxelized once and transformed according to each scene configuration and viewpoint to accelerate ground-truth generation.
  • Volumetric ground truth generation: The camera-view ground truth uses a 240×144×240 grid at 2 cm resolution, labeling objects, structural planes, and all remaining voxels as empty space.

5. Evaluation

Experiments on real and synthetic data show that jointly modeling occupancy and semantics improves scene completion and object recognition, while context modules and synthetic data further strengthen performance. The method also outperforms alternative semantic scene completion approaches, though depth-only input and reduced output resolution limit detail and some object distinctions.

  • Experimental setup: The evaluation uses NYU real depth maps and a SUNCG synthetic test set with voxel-level IoU measurements.NYU contains 1,449 Kinect depth maps; SUNCG contributes 500 rendered depth images from 184 held-out scenes.
  • Alternative approaches: 30.5% vs. 19.6% voxel-level IoU, while SSCNet runs in 7s versus 127s per image for Geiger and Wang.The comparison uses semantic scene completion methods with different inputs and model-fitting procedures.
  • Joint learning: Joint semantic supervision improves scene completion, with SSCNet better completing object geometry than occupancy-only approaches.The reported qualitative comparison specifically highlights improved chair completion and attributes the result to semantic 3D context.
  • Joint learning: 54.2% vs. 51.2% visible-surface IoU shows that completion supervision improves object recognition even when evaluation uses only visible surfaces.The comparison is between joint labeling-and-completion training and surface-labeling-only training.
  • Training data: 10.3% IoU improvement comes from pre-training on SUNCG and fine-tuning on NYU, especially for semantic scene completion.The comparison is against training only on NYU.
  • Ablations: Increasing the receptive field from 1.16 m to 2.26 m raises performance from 38.0% to 44.3%, while aggregation adds 3.1% and 2.1% IoU for the two tasks.The larger receptive field supports richer contextual information, and the aggregation comparison covers scene completion and semantic scene completion respectively.
  • Ablations: Flipped TSDF improves IoU by 10.1%, accurate TSDF by 2.4%, and removing empty-voxel balancing drops IoU from 46.4% to 42.7%.The encoding changes affect gradient placement and view dependency; balancing addresses the empty-versus-occupied voxel imbalance.
  • Limitations: Depth-only input makes missing-depth objects and geometrically similar categories difficult, while lower output resolution loses detail and small objects.The paper gives windows, desk-versus-furniture confusion, and small objects on a desk as examples.

6. Conclusion

The paper concludes that SSCNet jointly predicts volumetric occupancy and semantic labels for complete 3D scenes using a 3D ConvNet trained on a large synthetic dataset. Experiments report advantages over isolated task solutions and alternative semantic scene completion methods through 3D context and synthetic training data.

  • SSCNet is a 3D ConvNet for jointly predicting volumetric occupancy and semantic labels throughout full 3D scenes.
  • The network is trained on a new large-scale synthetic 3D scene dataset.
  • The joint model outperforms methods addressing either component task in isolation and alternative semantic scene completion approaches.The conclusion attributes the stronger performance to leveraging 3D contextual information and synthetic training data.

A. SUNCG Dataset Statistics

SUNCG contains diverse manually created indoor scenes spanning room layouts, physical scales, room types, object categories, and object-room relationships. Its statistics characterize both scene structure and the contextual arrangements of objects within rooms.

  • Scene structure: SUNCG ranges from single-room studios to multi-floor houses, averaging 8.9 rooms and 1.3 floors per house.The medians are 7 rooms and 1 floor per house.
  • Physical size: The dataset records real-world physical dimensions for rooms, floors, and houses.Physical-size statistics are reported at all three hierarchical levels.
  • Room types: SUNCG includes 24 user-labeled room types with example rooms illustrating the distribution.Types include living rooms, kitchens, bedrooms, offices, storage spaces, balconies, elevators, and garages.
  • Object categories: Its object library contains diverse common furniture and objects, while users can reshape, resize, and retexture them.
  • Object-room relationships: Rooms contain more than 14 objects on average, and object occurrence and arrangement vary by room type.The dataset provides complete object and room-type annotations for studying these relationships.
Loading 1611.08974v1…