Source-linked AI summary
NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, Hongsheng Li
TL;DR
Monocular SSC seeks to reconstruct volumetric scene geometry and semantics from a single RGB image, but existing methods face feature, pose, and computation ambiguities. NDC-Scene progressively restores depth in NDC space, shifts most 3D computation there, and adds a depth-adaptive dual decoder; it outperforms state-of-the-art methods on outdoor and indoor datasets.
Problem
Monocular SSC avoids costly additional 3D inputs, but existing methods suffer from feature ambiguity, pose ambiguity, and imbalanced 3D computation across depth.
Method
NDC-Scene progressively extends 2D features into NDC space with deconvolution, performs most 3D computation there, and jointly upsamples and fuses 2D and 3D features with a depth-adaptive dual decoder.
Results
NDC-Scene outperforms state-of-the-art monocular semantic scene completion methods on both outdoor and indoor datasets.
Takeaways & Limitations
Transferring most 3D computation to NDC space and using depth-adaptive dual decoding improves monocular SSC performance.
Takeaways & Limitations
The depth-adaptive decoder assumes that sufficiently large 2D receptive fields can infer both visible surfaces and scenes behind them.
Abstract
from arXiv · showhide
Monocular 3D Semantic Scene Completion (SSC) has garnered significant attention in recent years due to its potential to predict complex semantics and geometry shapes from a single image, requiring no 3D inputs. In this paper, we identify several critical issues in current state-of-the-art methods, including the Feature Ambiguity of projected 2D features in the ray to the 3D space, the Pose Ambiguity of the 3D convolution, and the Computation Imbalance in the 3D convolution across different depth levels. To address these problems, we devise a novel Normalized Device Coordinates scene completion network (NDC-Scene) that directly extends the 2D feature map to a Normalized Device Coordinates (NDC) space, rather than to the world space directly, through progressive restoration of the dimension of depth with deconvolution operations. Experiment results demonstrate that transferring the majority of computation from the target 3D space to the proposed normalized device coordinates space benefits monocular SSC tasks. Additionally, we design a Depth-Adaptive Dual Decoder to simultaneously upsample and fuse the 2D and 3D feature maps, further improving overall performance. Our extensive experiments confirm that the proposed method consistently outperforms state-of-the-art methods on both outdoor SemanticKITTI and indoor NYUv2 datasets. Our code are available at https://github.com/Jiawei-Yao0812/NDCScene.
1. Introduction
Monocular SSC reconstructs volumetric occupancy and semantics from one RGB image, avoiding costly 3D inputs, but prior feature projection and 3D convolution designs introduce feature, pose, and computation ambiguities. NDC-Scene addresses these issues by performing most 3D computation in NDC space and adding a depth-adaptive dual decoder.
- Motivation: Monocular SSC reconstructs volumetric 3D scenes from a single RGB image, eliminating the need for additional 3D inputs such as depth images or TSDFs.Prior SSC methods commonly rely on RGB images plus 3D data from specialized depth sensors.
- Challenges: Perspective projection creates feature-size and feature-depth ambiguity because 2D features spread across larger spaces and are shared among possible depths.Inconsistent feature density makes it difficult for convolution kernels to identify effective patterns and for 3D networks to infer reasonable voxel depths.
- Challenges: Pose ambiguity arises because prior 3D convolutions ignore camera extrinsic parameters, making projected convolution relationships vary across camera poses.The convolution should instead be conditioned on extrinsic parameters to maintain consistent projected neighborhoods.
- Challenges: Computation is imbalanced across depth: projected 3D convolution positions are sparse for close scenes and dense for far scenes, weakening representation of nearby structural details.This sparse allocation particularly misses rich structure and texture in close-scene regions such as the red shelf.
- Proposed approach: NDC-Scene directly recovers 3D features in NDC space, shifts most computation there, and uses a depth-adaptive dual decoder to jointly upsample and fuse 2D and 3D features.The method is designed to implicitly restore occupancy and semantics while using a more resilient fused representation.
- Results: The proposed method significantly outperforms state-of-the-art monocular SSC methods on both outdoor and indoor datasets.The reported evaluation covers outdoor and indoor settings without specifying dataset names in this passage.
2. Related Works
Prior work spans single-object and scene-level 3D reconstruction, indoor and outdoor SSC, and monocular semantic reconstruction, but dense reconstruction across varied scenes remains difficult. NDC-Scene places most 3D computation in NDC space and uses depth-adaptive decoding to improve feature restoration across depths.
- Single-View 3D Reconstruction: Single-view 3D reconstruction methods infer object- or scene-level geometry from one RGB image using explicit or implicit 3D representations.Related extensions include multi-object reconstruction, holistic scene layout estimation, and dense indoor semantic reconstruction.
- Monocular SSC: Monocular SSC removes the need for additional geometric inputs but prior dense reconstruction methods still suffer from pose ambiguity and computation imbalance.NDC-Scene transfers most 3D computation units to a different feature space to address these issues.
- Semantic Scene Completion: Semantic scene completion jointly infers scene geometry and semantics, with prior work addressing indoor small-scale and outdoor large-scale datasets.Outdoor-focused methods do not perform well in indoor scenes, while indoor methods have primarily targeted smaller environments.
- NDC-Scene: The depth-adaptive dual decoder restores voxel features at different depths and jointly processes 2D and 3D features for stronger occupancy and semantic representations.Its depth-adaptive design is intended to improve efficiency and feature restoration across depth levels.
3. Methodology
NDC-Scene addresses feature, pose, and computation ambiguities in monocular SSC by performing most 3D processing in NDC space and using a depth-adaptive dual decoder to fuse 2D and 3D features.
- Problem: Monocular SSC predicts voxel-wise occupancy and semantics from a single RGB image, but projecting 2D features into target 3D space creates feature-size and feature-depth ambiguities.Projected features become inconsistently distributed across depth, making effective patterns difficult for 3D convolution to identify.
- Normalized Device Coordinates Space: NDC-Scene directly extends the 2D feature map into NDC space by progressively reconstructing depth, enabling features to learn object density and depth while avoiding pose ambiguity and uneven computation allocation.NDC space adds a camera-depth dimension and provides consistent convolution scope across affine transformations.
- Depth Adaptive Dual Decoder: The depth-adaptive dual decoder upsamples 2D and 3D feature maps in separate branches and fuses them at each decoder layer through depth-adaptive attention.The decoder upsamples both feature types by a factor of 2 before fusing the 2D features into the 3D features.
- Problem: Camera-pose-dependent affine and perspective transformations cause 3D convolutions in target space to lose consistent correspondence with the original 2D feature arrangement.The convolution becomes unaware of which 2D positions its kernel is effectively operating on.
- Depth Adaptive Dual Decoder: Depth-adaptive attention computes similarities between 3D query and 2D key features, omitting value projection to reduce computation.The fused representation is subsequently processed by 3D convolution operations.
- Problem: Perspective projection also causes imbalanced computation allocation, with projected convolution positions sparse for close scenes and dense for far scenes.Sparse allocation can miss structural and textural details present in close-scene pixels.
4. Experiment
Experiments evaluate NDC-Scene on NYUv2 and SemanticKITTI against SSC baselines and ablations targeting feature ambiguity, computation imbalance, and pose ambiguity. The method outperforms competing approaches and remains more robust under the tested perturbations.
- Experimental setup: NDC-Scene is evaluated on indoor NYUv2 and outdoor SemanticKITTI using scene-completion and semantic-scene-completion metrics.NYUv2 contains indoor voxel grids, while SemanticKITTI provides large-scale outdoor voxel grids; IoU measures occupancy and mIoU measures semantic completion.
- Performance: NDC-Scene outperforms all compared methods on NYUv2 by a considerable margin, including MonoScene on both IoU and mIoU.The improvement also holds across all reported semantic categories.
- Performance: NDC-Scene beats the baselines significantly on SemanticKITTI across both small and large semantic categories.The reported comparison indicates adaptation across different outdoor scenarios.
- Qualitative performance: Qualitative results show more precise indoor layouts and instance-level information, plus more accurate outdoor depth ranges for terrains, trunks, and cars.The authors present these visualizations as evidence that feature ambiguity is substantially relieved.
- Ablation study: Introducing feature ambiguity reduces geometry by [-1.21, -0.09] IoU and semantics by [-1.34, -0.67] mIoU in the NDC-FA ablation.NDC-FA replaces the proposed dual decoder with a heavy 3D UNet whose inputs are lifted through FLoSP.
- Ablation study: The camera-space NDC-CI variant loses [-0.93, -0.44] mIoU and [-0.45, -0.04] IoU, while NDC-Scene degrades more slowly than MonoScene as pose perturbations increase.The pose study tests rotations with θ values of 5°, 10°, and 15°.
5. Conclusion
The paper addresses feature, pose, and computation ambiguities in monocular semantic scene completion with an NDC-space predictor and Depth-Adaptive Dual Decoder. Experiments support transferring most 3D computation to NDC space while progressively restoring depth and fusing 2D and 3D features.
- Conclusion: NDC-Scene progressively restores depth in Normalized Device Coordinates space with deconvolution operations instead of directly extending features to the target 3D space.The design transfers most computation from the target 3D space to NDC space.
- Conclusion: The Depth-Adaptive Dual Decoder simultaneously upsamples and fuses 2D and 3D feature maps to improve overall monocular SSC performance.The decoder is presented as an additional component of the proposed approach.