Source-linked AI summary
3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior
Xiaokang Chen, Kwan-Yee Lin, Chen Qian, Gang Zeng, Hongsheng Li
TL;DR
Semantic scene completion must infer voxel occupancy and semantic labels from partial views despite the computational and detail limits of low-resolution representations. The paper introduces explicit 3D sketch-aware embedding with CVAE-guided sketch hallucination, achieving state-of-the-art performance on three benchmarks using 60 × 36 × 60 volumes.
Problem
Low-resolution voxel representations limit SSC detail prediction, while existing methods struggle with fine object details and large regions of missing geometry.
Method
The method explicitly embeds depth geometry as a 3D sketch and uses a CVAE-based 3D Sketch Hallucination Module to infer full sketches that guide SSC.
Results
The model consistently outperforms state-of-the-art methods on three public benchmarks using 60 × 36 × 60 input and output volumes.
Takeaways & Limitations
Explicit structure-preserving geometric embedding supports efficient SSC with low-resolution volumes while retaining performance across benchmarks.
Takeaways & Limitations
RGB-assisted approaches remain vulnerable to complex scenes because they rely heavily on cross-modality feature embedding design.
Abstract
from arXiv · showhide
The goal of the Semantic Scene Completion (SSC) task is to simultaneously predict a completed 3D voxel representation of volumetric occupancy and semantic labels of objects in the scene from a single-view observation. Since the computational cost generally increases explosively along with the growth of voxel resolution, most current state-of-the-arts have to tailor their framework into a low-resolution representation with the sacrifice of detail prediction. Thus, voxel resolution becomes one of the crucial difficulties that lead to the performance bottleneck. In this paper, we propose to devise a new geometry-based strategy to embed depth information with low-resolution voxel representation, which could still be able to encode sufficient geometric information, e.g., room layout, object's sizes and shapes, to infer the invisible areas of the scene with well structure-preserving details. To this end, we first propose a novel 3D sketch-aware feature embedding to explicitly encode geometric information effectively and efficiently. With the 3D sketch in hand, we further devise a simple yet effective semantic scene completion framework that incorporates a light-weight 3D Sketch Hallucination module to guide the inference of occupancy and the semantic labels via a semi-supervised structure prior learning strategy. We demonstrate that our proposed geometric embedding works better than the depth feature learning from habitual SSC frameworks. Our final model surpasses state-of-the-arts consistently on three public benchmarks, which only requires 3D volumes of 60 x 36 x 60 resolution for both input and output. The code and the supplementary material will be available at https://charlesCXK.github.io.
1. Introduction
Semantic scene completion is limited by low-resolution voxel representations, which hinder fine-detail classification and completion when geometry is missing. The paper addresses this with explicit 3D sketch-aware geometric embedding and CVAE-guided hallucination of full 3D sketches.
- Low-resolution voxel representations make existing SSC methods error-prone on object details and difficult to apply when large regions of geometry are missing.
- Explicit geometric information is hypothesized to help networks learn object structure from low-resolution partial observations, provided the geometry is resolution-insensitive.
- 3D sketch-aware feature embedding explicitly and compactly encodes geometric information from depth for SSC.
- The framework first infers a full 3D sketch from a partial observation, then uses its embedded features to guide scene reconstruction and semantic recognition.
- CVAE-based structure-prior learning samples diverse reasonable 3D sketches and supports accurate, realistic inference when substantial geometry is missing.
- 60 × 36 × 60 3D volumes suffice for both input and output while the model consistently outperforms state-of-the-art methods on three public benchmarks.
2. Related Work
Related work spans object-shape completion, semantic scene completion, 2D boundary detection, and structure representation learning. The paper distinguishes its explicit geometric embedding from prior methods that encode depth implicitly.
- Object Shape Completion: Knowledge-based shape completion uses geometric reasoning or CAD retrieval, but retrieval is expensive and generalization is poor for unseen shapes.
- Object Shape Completion: Learning-based shape completion improves flexibility and robustness, yet single-object methods are difficult to extend to multi-object semantic scenes.
- Semantic Scene Completion: SSC methods jointly produce complete voxel occupancy and semantic labels, using architectures such as sparse, projected, generative, or cascaded 3D networks.
- Semantic Scene Completion: RGB-based SSC methods complement depth with semantic information through two-stream, sequential, or dimensional-decomposition architectures.
- Semantic Scene Completion: Prior SSC approaches encode depth implicitly in high-dimensional features, whereas this paper predicts a 3D sketch first and uses it to guide reconstruction and recognition.
- 2D Boundary Detection: Boundary detection methods use handcrafted gradients or learned multilevel features, and boundary cues have been integrated into segmentation and landmark tasks.
- Structure Representation Learning: Deep generative models provide precedents for structured-output prediction, landmark discovery, and shape-guided image synthesis.
3. Methodology
The method uses a two-stage pipeline: it extracts a 3D sketch as structure prior information, refines it with a CVAE, and then guides semantic scene completion from RGB and sketch features. The architecture predicts dense voxel labels while using sketch hallucination to compensate for incomplete single-view geometry.
- Overall architecture: The network stacks sketch extraction and semantic scene completion stages, producing a semantic label for every voxel in the view frustum.Empty voxels are represented by C0.
- Sketch extraction: A 3D Sobel operator binarizes semantic-label gradients to extract boundary-like geometric structure while removing class-identity gaps.The resulting representation is called a 3D Sketch rather than a conventional 2D edge or boundary.
- Sketch prediction stage: The sketch prediction stage encodes TSDF volumes with convolutions and DDR blocks, downsamples for efficiency, then upsamples to produce the predicted sketch.Skip connections are added for gradient propagation, while dilated DDR blocks maintain large receptive fields.
- Sketch hallucination: The Sketch Hallucination Module uses a CVAE to sample diverse plausible full 3D sketches conditioned on the partial predicted sketch instead of directly regressing one ground-truth sketch.During training, the encoder uses the estimated and ground-truth sketches; the decoder reconstructs the ground-truth sketch from a sampled latent and the estimate.
- Sketch hallucination: At inference, multiple sampled sketches are decoded and averaged to obtain a refined sketch for downstream completion.The refined sketch is used with the original sketch as prior guidance for the RGB-based semantic stage.
- Semantic scene completion: The semantic stage extracts RGB features, projects them into 3D using depth and camera parameters, and adds mapped original and refined sketch features before 3D CNN processing.Training uses RGB, TSDF, ground-truth sketch, and ground-truth semantic labels within the complete architecture.
4. Experiments
Experiments evaluate the method on three datasets using SC and SSC metrics, with ablations examining modules, structure-prior representations, embeddings, modalities, and resolutions. The method consistently outperforms prior approaches while operating at 60 × 36 × 60 input and output resolution.
- Experimental Setup: The method is evaluated on NYU, NYUCAD, and SUNCG using precision, recall, and voxel-level IoU for SC and SSC.SSC evaluates object-class IoU on observed and occluded voxels, whereas SC treats all voxels as occupied or unoccupied.
- Ablation Study: The full sketch prior reaches 94.2% SC IoU in oracle ablation, and performance remains above the proposed method when 80% of the sketch is dropped.These results support the validity of accurate structure priors for inferring invisible areas.
- Comparisons with State-of-the-art Methods: 7.8% SC IoU and 2.6% SSC mIoU gains over CCPNet are reported on NYU.The authors attribute the improvement to the two-stage architecture using structure priors to infer invisible areas with structure-preserving details.
- Comparisons with State-of-the-art Methods: 1.8% SC IoU and 2.0% SSC mIoU gains over CCPNet are reported on NYUCAD with 60 × 36 × 60 input resolution.The method retains its advantage despite some competing methods using larger input resolutions.
- Ablation Study: 84.2% SC IoU and 55.2% SSC mIoU are achieved on NYUCAD after adding the 3D Sketch Hallucination Module.The ablation attributes further improvement to the hallucination module after introducing structure priors.
- Ablation Study: Sketch is the strongest structure-prior representation, while explicit sketch-supervised embedding improves over unsupervised implicit embedding by 3.1% SC IoU and 4.6% SSC mIoU.The experiments also report that TSDF produces better structure priors than RGB, yielding a 3.3% SC IoU gain, while shared TSDF structure priors yield a 3.9% SSC mIoU gain.
- Qualitative Results of 3D Sketch: The hallucinated sketch is more complete and precise, producing sharper semantic boundaries and fewer mislabeled regions in qualitative examples.Without CVAE, missing sketch boundaries correspond to mislabeled bookcase, wall-object, confusing semantic, and photo-frame regions.
5. Conclusion
The paper proposes a 3D sketch-aware feature embedding and a semantic scene completion framework using a 3D Sketch Hallucination Module. Experiments report effectiveness, efficiency, and state-of-the-art performance on three public benchmarks.
- The proposed 3D sketch-aware feature embedding explicitly embeds geometric information while preserving structural details.
- The semantic scene completion framework uses a 3D Sketch Hallucination Module to guide full 3D sketch inference from partial observations through a structure prior.
- The method achieves reported effectiveness, efficiency, and state-of-the-art performance on three public benchmarks.