Source-linked AI summary
VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion
Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M. Alvarez, Sanja Fidler, Chen Feng, Anima Anandkumar
TL;DR
Camera-based semantic scene completion must infer complete 3D geometry and semantics from images despite occlusions and ambiguous 2D-to-3D correspondences. VoxFormer addresses this with sparse depth-based voxel queries followed by MAE-like sparse-to-dense completion, outperforming prior camera-based methods and matching LiDAR-based methods at close range. Its long-range performance remains limited by unreliable depth.
Problem
Camera-based SSC must recover complete 3D scene geometry and semantics from incomplete visual observations, while dense projection can assign visible features to empty or occluded voxels.
Method
VoxFormer uses depth-based class-agnostic sparse voxel proposals followed by class-specific semantic segmentation with a sparse-to-dense MAE-like Transformer.
Results
VoxFormer outperforms the state-of-the-art camera-based method and performs on par with LiDAR-based methods at close range.
Takeaways & Limitations
The framework provides complete 3D voxelized semantic scenes from camera images and motivates further research in camera-based SSC for autonomous-vehicle perception.
Takeaways & Limitations
Long-range performance needs improvement because depth is very unreliable at corresponding locations; decoupling long- and short-range SSC is proposed as future work.
Abstract
from arXiv · showhide
Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures rather than the occluded or empty spaces. Therefore, starting with the featurization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training to less than 16GB. Our code is available on https://github.com/NVlabs/VoxFormer.
1. Introduction
VoxFormer addresses camera-based semantic scene completion by first reconstructing visible occupied structures and then completing occluded or empty regions. Its two-stage sparse-to-dense design achieves state-of-the-art performance on SemanticKITTI, especially at safety-critical short range.
- Holistic 3D scene understanding supports autonomous-vehicle planning and map construction but is limited by sensing resolution, field of view, and occlusions.
- SSC jointly infers complete scene geometry and semantics from limited observations, combining visible-region reconstruction with occluded-region hallucination.
- Camera-based SSC is motivated by cameras’ lower cost and richer visual cues, but dense 2D-to-3D projection can assign visible-region features to empty or occluded voxels.
- VoxFormer sets a new state-of-the-art for camera-based SSC on SemanticKITTI and achieves significant improvements in safety-critical short-range areas.
- VoxFormer uses a two-stage framework: depth-based class-agnostic query proposals produce sparse occupied voxels, followed by class-specific semantic completion with an MAE-like Transformer.
2. Related Works
Related work spans 3D reconstruction and completion, semantic segmentation, and semantic scene completion for indoor and outdoor environments. Camera-based SSC reduces sensing cost but faces false 3D features and efficiency challenges from dense projection and heavy 3D convolutions.
- 3D reconstruction and completion: 3D reconstruction infers scene or object geometry from single or multiple 2D images, using explicit or implicit representations.
- 3D reconstruction and completion: 3D completion extends reconstruction by hallucinating unseen structure from partial inputs, including point, voxel, and distance-field representations.
- Semantic segmentation: Semantic segmentation methods analyze images or point clouds, while 3D segmentation supports interaction with physical environments that are inherently three-dimensional.
- 3D semantic scene completion: Semantic scene completion jointly infers geometry and semantics, producing dense voxelized representations useful for autonomous-vehicle mapping and dynamic-object perception.
- 3D semantic scene completion: MonoScene performs camera-only SSC through 2D-to-3D feature projection and successive 2D and 3D UNets, but projection introduces false features and 3D convolution reduces efficiency.
- Camera-based 3D perception: Camera-based 3D perception is attractive because cameras are low-cost, easy to deploy, widely available, and rich in visual attributes.
3. Methodology
VoxFormer predicts dense semantic voxel scenes from RGB images through sparse depth-based query proposals followed by Transformer-based voxel completion and segmentation.
- Problem setup: VoxFormer predicts a dense semantic voxel grid from current and previous RGB images in the vehicle’s coordinate system.The output assigns each voxel either an empty label or one of M semantic classes.
- Stage-2: Class-Specific Segmentation: Stage 2 combines updated proposals with learnable mask tokens, applies voxel self-attention, then upsamples and projects refined features into the final semantic map.The output contains M semantic classes plus one empty class.
- Overall architecture: The framework uses class-agnostic stage-1 proposals followed by class-specific stage-2 segmentation.Stage 1 proposes sparse occupied voxels, while stage 2 completes the scene representation from those proposals.
- Stage-1: Class-Agnostic Query Proposal: Stage 1 uses estimated depth and occupancy prediction to select sparse voxel queries while removing many empty-space queries.This reduces computation and memory and eases attention learning by reducing erroneous 2D-to-3D correspondence ambiguities.
- Stage-2: Class-Specific Segmentation: The proposed queries attend to projected image features through deformable cross-attention that samples local regions around reference points.For each proposed voxel, its 3D location is projected into hit images, where nearby features are sampled and weighted.
4. Experiments
VoxFormer is evaluated on SemanticKITTI for camera-based semantic scene completion across geometry, semantics, ranges, object sizes, efficiency, and ablations. It achieves strong close-range performance and remains limited at long range because depth estimates are unreliable there.
- Experimental setup: SemanticKITTI evaluates geometry with IoU and semantic segmentation with mIoU across 12.8m, 25.6m, and 51.2m volumes.The benchmark covers 20 voxel labels, including 19 semantic classes and free space, at 0.2m voxel resolution.
- Camera-based comparison: mIoU improves by 8.10%, 11.77%, and 22.03% with temporal information in the 51.2m, 25.6m, and 12.8m volumes, respectively.Category IoU also improves for building, parking, and terrain in the full volume.
- Range and modality comparison: 21.55 and 18.42 mIoU within 12.8m and 25.6m outperform MonoScene by 75.92% and 50.74%, respectively, in safety-critical short-range areas.At 12.8m, VoxFormer also exceeds LiDAR-based SSCNet in mIoU by 7.63% and JS3CNet in IoU by 3.00%.
- Object-level comparison: 5.22 versus 0.07 for bicycle, 2.98 versus 0.05 for motorcycle, and 21.39 versus 2.02 for trunk show large small-object gains over MonoScene.Other reported gains include bicyclist, pole, traffic sign, and person categories.
- Efficiency and ablations: VoxFormer uses approximately 60M parameters and less than 16GB training GPU memory, while dense and random queries are less effective or less stable than occupancy-based queries.Stereo depth performs best; temporal input improves mIoU but increases memory, and long-range performance remains constrained by unreliable depth.
5. Conclusion
VoxFormer is a camera-based 3D semantic scene completion framework that combines depth-based sparse query proposal with sparse-to-dense semantic segmentation. It outperforms the state-of-the-art camera-based method and performs on par with LiDAR-based methods at close range.
- VoxFormer combines class-agnostic query proposal based on depth estimation with class-specific segmentation using a sparse-to-dense MAE-like design.
- The framework lifts images into complete 3D voxelized semantic scenes.
- VoxFormer outperforms the state-of-the-art camera-based method and performs on par with LiDAR-based methods at close range.
Appendix
The appendix reports quantitative and qualitative comparisons on SemanticKITTI’s hidden test set, where ground-truth labels are unavailable. Consequently, results are reported only for the full-range volume.
- The appendix compares VoxFormer with MonoScene on SemanticKITTI’s hidden test set.
- Because ground truth is unavailable for the test set, performance is reported only within the full-range 51.2×51.2×6.4m3 volume.
A. Quantitative Comparison
On SemanticKITTI, VoxFormer improves geometric completion and semantic segmentation over MonoScene, with stronger performance also reported in safety-critical short-range areas. Test-set short-range evaluations are unavailable, so those results are inferred from validation-set trends.
- 25.73% relative IoU gain over MonoScene is achieved by VoxFormer-S without historical observations.
- VoxFormer performs much better than MonoScene in safety-critical short-range areas for both geometric completion and semantic segmentation.
- 21.03% and 10.11% relative mIoU improvements over MonoScene are achieved by VoxFormer-T and VoxFormer-S, respectively.
- 12.35 validation mIoU versus 12.20 test mIoU is reported for VoxFormer-S over the full-range volume.
B. Qualitative Comparison
Qualitative comparisons show that VoxFormer performs better than MonoScene at short range, while its long-range completion remains imperfect.
- VoxFormer performs much better than MonoScene in short-range areas, where MonoScene misses objects at close range.
- Long-range completion remains incomplete for some objects, including trunks in the visualized scenes.
Method
On the hidden SemanticKITTI test set, VoxFormer is compared quantitatively with MonoScene and qualitatively with other methods. The qualitative results emphasize scene-layout completion and performance on small objects.
- Table I compares VoxFormer quantitatively with the state-of-the-art MonoScene on SemanticKITTI’s hidden test set.
- Figure I shows that VoxFormer better captures large-scale scene layout in self-driving scenarios than other methods.
- Figure I reports satisfactory completion of small objects, including trunks and poles.