Source-linked AI summary
MonoScene: Monocular 3D Semantic Scene Completion
Anh-Quan Cao, Raoul de Charette
TL;DR
MonoScene addresses dense 3D semantic scene completion from a single monocular RGB image, avoiding reliance on 2.5D or 3D input. It combines 2D–3D feature projection, contextual modeling, and scene-level losses, outperforming comparable baselines on indoor and outdoor datasets.
Problem
Dense 3D geometry and semantics are difficult to infer jointly from one RGB image, while depth sensors are more expensive, less compact, and more intrusive than cameras.
Method
MonoScene uses successive 2D and 3D UNets bridged by Features Line of Sight Projection, with 3D Context Relation Prior and complementary global and local SSC losses.
Results
MonoScene outperformed all comparable baselines across indoor and outdoor evaluations, including some methods using 3D input.
Takeaways & Limitations
A single RGB image can support voxel-wise semantic scene completion across indoor and outdoor scenes within the proposed framework.
Takeaways & Limitations
The framework struggles with fine-grained geometry, semantically similar classes, small objects, and occlusion artefacts from the single viewpoint.
Abstract
from arXiv · showhide
MonoScene proposes a 3D Semantic Scene Completion (SSC) framework, where the dense geometry and semantics of a scene are inferred from a single monocular RGB image. Different from the SSC literature, relying on 2.5 or 3D input, we solve the complex problem of 2D to 3D scene reconstruction while jointly inferring its semantics. Our framework relies on successive 2D and 3D UNets bridged by a novel 2D-3D features projection inspiring from optics and introduces a 3D context relation prior to enforce spatio-semantic consistency. Along with architectural contributions, we introduce novel global scene and local frustums losses. Experiments show we outperform the literature on all metrics and datasets while hallucinating plausible scenery even beyond the camera field of view. Our code and trained models are available at https://github.com/cv-rits/MonoScene.
1. Introduction
MonoScene addresses dense 3D semantic scene completion from a single RGB image, including indoor and outdoor settings. It bridges 2D and 3D processing with feature projection, context modeling, and specialized losses.
- Existing SSC methods generally require depth-derived or other geometrical inputs and are tailored to either indoor or outdoor scenes.
- MonoScene infers dense voxelized semantic scenes from a single RGB image across both indoor and outdoor environments.
- FLoSP projects 2D features along their line of sight to bridge 2D and 3D networks while allowing the 3D network to select relevant features.
- A 3D context layer supplies global receptive-field information and semantic relations among voxels.
- Scene-class affinity and local frustum-proportion losses optimize semantic distributions beyond independent voxelwise cross-entropy.
- MonoScene outperformed comparable baselines on indoor and outdoor evaluations and produced inferred scenery beyond the camera field of view.
2. Related works
Prior work spans single-object reconstruction, holistic scene understanding, and SSC using geometric inputs. MonoScene instead densely estimates geometry and semantics for indoor and outdoor scenes from monocular RGB features.
- Deep-learning work initially emphasized explicit or implicit reconstruction of individual 3D objects.
- MonoScene uses 2D and 3D UNets bridged by FLoSP, with scene-class affinity and frustum-proportion losses supervising semantic distributions.
- Holistic 3D understanding predicts scene and object layouts, typically with sparse scene representations.
- Existing SSC methods commonly use depth, occupancy grids, point clouds, or TSDFs as geometric inputs.
- Contextual-awareness methods use multiscale skip connections, dilated convolutions, self-attention, global pooling, or explicit contextual learning.
3. Method
MonoScene performs voxel-wise semantic scene completion from a single RGB image by lifting multiscale 2D features into 3D, modeling global relations, and supervising global and local semantic distributions.
- 3. Method: MonoScene infers 3D geometry and semantics from a single RGB image rather than relying on depth, point clouds, or other 2.5D/3D inputs.The pipeline uses consecutive 2D and 3D UNets connected by FLoSP.
- 3.1. Features Line of Sight Projection (FLoSP): FLoSP backprojects multiscale 2D decoder features along optical rays into one 3D feature map used as 3D UNet input.Features are sampled at the 2D projections of 3D voxel centroids across scales S={1, 2, 4, 8}; out-of-image voxels receive zero vectors.
- 3.2. 3D Context Relation Prior (3D CRP): The 3D Context Relation Prior learns scene-wide voxel relations and uses their relation matrices to gather global context at the 3D UNet bottleneck.Relations may be supervised or self-discovered, and the component provides a global receptive field for spatio-semantic awareness.
- 3.2. 3D Context Relation Prior (3D CRP): 3D CRP represents four non-overlapping relations based on occupancy status and semantic similarity or difference.The relations are free-similar, free-different, occupied-similar, and occupied-different; supervoxel↔voxel relations reduce the quadratic voxel-relation cost.
- 3.3. Losses: Scene-class affinity loss directly optimizes class-wise precision, recall, and specificity, while Frustum Proportion Loss matches predicted and ground-truth class distributions locally.Local frustums are formed from equal-sized image patches to provide cues about visible and occluded structure.
4. Experiments
MonoScene is evaluated on indoor NYUv2 and outdoor SemanticKITTI under all-voxel metrics, against RGB-inferred and 2.5/3D-input baselines. It achieves stronger semantic completion while maintaining or improving scene geometry, with qualitative gains in thin, occluded, and out-of-view structures.
- Datasets: NYUv2 contains 1,449 indoor scenes with 13 labeled voxel classes, while SemanticKITTI provides outdoor 256×256×32 grids with 21 classes.NYUv2 uses 795/654 train/test splits; SemanticKITTI uses official 3,834/815 train/validation splits and hidden-test evaluation.
- Evaluation protocol: The evaluation reports occupied-voxel IoU for scene completion and semantic-class mIoU for semantic scene completion, using harder all-voxel metrics for both settings.The authors retrained all baselines under this common evaluation practice.
- Comparison with RGB-inferred baselines: MonoScene outperforms all methods by +4.03 mIoU on NYUv2 and +2.11 mIoU on SemanticKITTI.IoU also improves or remains comparable, by +3.87 and +0.16 respectively, indicating the semantic gains are not obtained by simply sacrificing geometry.
- Qualitative results: Qualitatively, MonoScene recovers thin indoor elements, better captures outdoor layouts and occluded car geometry, and hallucinates scenery beyond the camera field of view.Examples include table legs, paintings, TVs, cross-roads, and cars.
- Comparison with 2.5/3D-input baselines: Against 2.5/3D-input baselines, MonoScene still exceeds some indoor baselines in mIoU, while SemanticKITTI comparisons favor wider-FOV lidar-based inputs.The authors relate the outdoor gap to lidar’s 180° versus camera’s 82° horizontal FOV and greater scene complexity.
- Ablation studies: Ablations show that the architecture, 3D CRP, scene-class affinity losses, frustums proportion loss, and multiscale FLoSP projections each contribute to performance.Using projection scales (1,2,4,8) consistently improves IoU and mIoU, while FLoSP exceeds CoReNet’s ray-traced skip connections by +10.2 IoU and +8.56 mIoU on NYUv2.
5. Discussion
MonoScene extends monocular SSC across indoor and outdoor scenes but remains limited in fine-grained reconstruction, small-object recognition, viewpoint robustness, and safety-critical reliability.
- Limitations: MonoScene struggles with fine-grained geometry, separating semantically similar classes, and recognizing small objects.Small objects comprise less than 0.3% of SemanticKITTI.
- Limitations: Single-view reconstruction produces outdoor occlusion distortions along the line of sight and increasingly greater distortion when camera settings depart from training settings.The paper reports consistent results across camera setups, but with increasing distortion under FOV changes.
- Broader impact and ethics: The authors identify mixed reality, photo editing, and mobile robotics as potential applications, while warning that scene-understanding errors could be fatal in autonomous driving.They recommend that such algorithms be seconded by other means.
A. Architectures details
The RGB-inferred baseline comparisons reconstruct each baseline’s required 3D input from monocular RGB using pretrained depth and semantic models, then preserve the baselines’ original input formats.
- RGB-inferred baselines: AICNetrgb receives a depth map inferred by pretrained AdaBins from the RGB image.The implementation uses AdaBins on both NYUv2 and SemanticKITTI.
- RGB-inferred baselines: 3DSketchrgb converts AdaBins depth into a TSDF using the 3DMatch Toolbox.The TSDF serves as 3DSketch’s RGB-inferred input.
- RGB-inferred baselines: JS3CNetrgb unprojects AdaBins depth into a point cloud and augments it with pretrained 2D semantic labels for training.The semantic point cloud is required by JS3C-Net’s training procedure.
- RGB-inferred baselines: LMSCNetrgb discretizes the unprojected AdaBins point cloud into an occupancy grid.This occupancy grid supplies LMSCNet’s RGB-inferred input.
A.2. MonoScene
MonoScene’s 3D network uses a compact UNet with DDR-based encoding, deconvolutional decoding, and an ASPP completion head, with optional upsampling for SemanticKITTI.
- 3D UNet: The 3D encoder has two downscaling layers, each containing four DDR blocks, and the decoder has two deconvolution layers that double spatial scale.DDR blocks provide a large receptive field with low memory cost.
- Completion head: The completion head uses ASPP with dilation rates (1, 2, 3) to gather multiscale features.An optional deconvolution layer reaches the output size and is used for SemanticKITTI.
- Training cost: Training took 7 hours on NYUv2 with two V100 32G GPUs and 28 hours on SemanticKITTI with four V100 32G GPUs.The reported configurations used two items per GPU for NYUv2 and one item per GPU for SemanticKITTI.
B.1. SemanticKITTI
On SemanticKITTI, MonoScene generalizes better than RGB-inferred baselines, captures landscapes and objects more effectively, and performs better on visible than unseen voxels.
- MonoScene’s validation-to-test gap is −0.42, smaller than JS3C-Netrgb’s −1.34 and AICNetrgb’s −1.22.
- Using only RGB, MonoScene outperforms some SSC baselines that use 3D input on the complete hidden-test benchmark.
- MonoScene captures better landscapes and objects than all baselines, including cars, pedestrians, and traffic signs.
- Thin small objects, consecutive distant cars, and highly cluttered scenes remain difficult for MonoScene.
- In-FOV performance exceeds Whole Scene performance, while out-FOV performance is significantly lower because unseen voxels are not observed in the image.
B.2. NYUv2
On NYUv2, MonoScene produces better scene layouts and object geometry overall, but struggles with complex or rare objects and semantically similar classes.
- MonoScene predicts better scene layouts and object geometry in several NYUv2 qualitative examples.The cited examples include rows 1–4, 6, 9, and 10.
- Complex objects such as bookshelves and rare objects such as running machines are frequently mispredicted.
- MonoScene sometimes confuses semantically similar classes, including windows with objects, beds with objects, and furniture with tables.
B.3. Generalization
Generalization degrades as camera setups depart from training conditions, while domain differences further affect predictions across cities and datasets.
- Predictions show increasing distortion as camera setups increasingly differ from the training configuration.
- Domain gap is smaller for KITTI-360 because it shares similar Karlsruhe residential scenes and narrow roads with SemanticKITTI.
- nuScenes and Cityscapes are affected by both camera changes and wider metropolitan scenes than the Karlsruhe training environment.
- SemanticKITTI qualitative results visualize scenery outside the viewing frustum using darker voxels.
- The domain-gap analysis compares Cityscapes, nuScenes, SemanticKITTI, and KITTI-360 under differing camera setups.