Source-linked AI summary
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving
Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu
TL;DR
Existing vision-based perception methods struggle to represent arbitrary-shaped objects and recover occluded scene regions comprehensively. SurroundOcc predicts dense 3D occupancy from multi-camera images using 2D-3D volume attention, progressive 3D upsampling, and LiDAR-based dense-label generation. Experiments on nuScenes and SemanticKITTI demonstrate its effectiveness, including state-of-the-art monocular semantic scene completion performance.
Problem
Object detection has difficulty representing arbitrary shapes and unlimited real-world classes, while depth maps cannot recover occluded scene parts.
Method
SurroundOcc lifts multi-scale multi-camera image features into 3D volumes with 2D-3D spatial attention, progressively upsamples them, and generates dense labels from fused LiDAR points using Poisson Reconstruction and voxelization.
Results
SurroundOcc demonstrates effectiveness on nuScenes and achieves state-of-the-art performance on the monocular 3D semantic scene completion benchmark.
Takeaways & Limitations
3D occupancy provides a scene representation that maintains multi-camera geometric consistency, recovers occluded parts, and can extend to downstream 3D tasks.
Takeaways & Limitations
The paper explores only single-frame occupancy prediction and identifies occupancy flow as important for downstream motion prediction and planning.
Abstract
from arXiv · showhide
3D scene understanding plays a vital role in vision-based autonomous driving. While most existing methods focus on 3D object detection, they have difficulty describing real-world objects of arbitrary shapes and infinite classes. Towards a more comprehensive perception of a 3D scene, in this paper, we propose a SurroundOcc method to predict the 3D occupancy with multi-camera images. We first extract multi-scale features for each image and adopt spatial 2D-3D attention to lift them to the 3D volume space. Then we apply 3D convolutions to progressively upsample the volume features and impose supervision on multiple levels. To obtain dense occupancy prediction, we design a pipeline to generate dense occupancy ground truth without expansive occupancy annotations. Specifically, we fuse multi-frame LiDAR scans of dynamic objects and static scenes separately. Then we adopt Poisson Reconstruction to fill the holes and voxelize the mesh to get dense occupancy labels. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the superiority of our method. Code and dataset are available at https://github.com/weiyithu/SurroundOcc
1. Introduction
SurroundOcc addresses limitations of object detection and depth-only reconstruction by predicting dense 3D occupancy from multi-camera images. It combines 2D-3D volume modeling with dense-label generation from sparse LiDAR and reports strong results on nuScenes and SemanticKITTI.
- Motivation: 3D occupancy represents each voxel’s occupied probability, preserves multi-camera geometric consistency, recovers occluded parts, and supports downstream 3D tasks.This representation is proposed as a complement to object detection and depth estimation.
- Method: SurroundOcc lifts multi-scale image features into 3D volume features using 2D-3D spatial attention rather than BEV features.The method then progressively upsamples and fuses volume features with 3D convolutions while applying supervision at each level.
- Dense supervision: Dense occupancy labels are generated from sparse LiDAR using separate multi-frame fusion for dynamic objects and static scenes, followed by Poisson Reconstruction and voxelization.The pipeline uses existing 3D detection and semantic segmentation labels instead of expensive occupancy annotations.
- Results: Experiments on nuScenes and SemanticKITTI demonstrate the effectiveness of SurroundOcc, including state-of-the-art monocular semantic scene completion performance.The method was not designed for the monocular setting, but achieves that result on the monocular benchmark.
2. Related Work
Prior work spans voxel-based scene representations, depth estimation, implicit and explicit 3D reconstruction, and vision-based surround perception. SurroundOcc distinguishes itself by constructing dense occupancy supervision and preserving 3D structure through volume-based attention.
- Voxel-based Scene Representation: Voxel representations discretize 3D space into voxels described by vector features and support lidar segmentation and 3D scene completion.The paper advocates voxel representation for occupancy because it models the scene’s occupancy field directly.
- Voxel-based Scene Representation: MonoScene reconstructs outdoor scenes from RGB inputs, while TPVFormer extends semantic occupancy prediction to multiple cameras but lacks dense supervision.SurroundOcc addresses this gap by generating dense occupancy ground truth and denser predictions.
- 3D Scene Reconstruction: Explicit 3D reconstruction methods directly recover scene geometry, including multi-view occupancy prediction and online fusion of learned image features.The related methods include SurfaceNet, Atlas, NeuralRecon, and TransformerFusion.
- Vision-based 3D Perception: Depth-based methods predict image depth before projecting semantic features into 3D, whereas other approaches learn 3D features without explicit depth maps.These are two major paradigms in vision-based 3D perception.
3. Approach
SurroundOcc predicts 3D occupancy by lifting multi-camera features into 3D volume space, then progressively refining multiscale volume features. Its training uses dense occupancy labels generated from fused LiDAR observations and mesh voxelization.
- Problem formulation: 3D occupancy assigns an occupied probability to each voxel, supports multi-camera consistency, recovers occluded areas, and extends to downstream 3D tasks.Semantic occupancy additionally assigns class labels to voxels, with class 0 denoting non-occupied grids.
- 2D-3D spatial attention: The method extracts multiscale image features and uses 2D-3D spatial attention to lift them into 3D volume features rather than BEV features.Cross-view attention projects 3D reference points into 2D views and aggregates sampled features using deformable attention.
- 2D-3D spatial attention: Cross-view attention avoids equally averaging views by weighting information from projected camera features, which helps address occluded or blurred views.The design uses 3D volume queries to preserve 3D spatial information and 3D convolutions to interact neighboring voxels instead of expensive 3D self-attention.
- Dense occupancy supervision: Dense occupancy labels stitch multiframe LiDAR points for dynamic objects and static scenes, merge them, reconstruct a mesh, and voxelize it.The pipeline uses Poisson Reconstruction to densify points and Nearest Neighbor to assign semantic labels; its dense voxels are presented as more realistic than sparse alternatives.
- Multi-scale occupancy prediction: A 2D-3D U-Net upsamples lower-resolution volume features with 3D deconvolution and fuses them with multiscale features for fine-grained representations.Each level produces an occupancy prediction, and multiscale supervision uses cross-entropy and scene-class affinity losses with decayed weights.
4. Dense Occupancy Ground Truth
SurroundOcc generates dense occupancy ground truth by separately stitching static scenes and movable objects across frames, then filling point-cloud holes and voxelizing the reconstructed mesh. Semantic labels are transferred to dense voxels by nearest-neighbor search, though this propagation remains sensitive to annotation noise.
- Motivation: The pipeline uses existing 3D detection and semantic segmentation labels to generate dense occupancy ground truth without extra human annotations.Sparse LiDAR supervision cannot produce dense enough occupancy, while dense voxel annotation is costly for scenes containing millions of voxels.
- Multi-frame Point Cloud Stitching: Static scene points and movable-object points are extracted separately, accumulated across frames, transformed into world coordinates, and merged for each current frame.Object identity across frames is tracked using bounding-box indices, while calibrated matrices and ego-poses align the segments.
- Densifying with Poisson Reconstruction: Poisson Surface Reconstruction converts the stitched point cloud with estimated normals into a triangular mesh that fills holes before dense voxelization.The mesh provides more evenly distributed vertices for conversion into dense occupancy voxels.
- Semantic Labeling with NN Algorithm: Nearest-neighbor search transfers semantic labels from sparse semantic voxels to each occupied dense voxel.The dense occupancy voxels are generated geometrically first, then semantic labels are assigned from the nearest voxelized semantic point.
- Qualitative Results: Dense voxels provide more realistic occupancy labels with clear semantic boundaries than sparse occupancy labels.The comparison is presented alongside single-frame LiDAR points and sparse occupancy labels.
- Limitations: Semantic-label propagation is ill-posed, and nearest-neighbor labeling is sensitive to noise in the original LiDAR semantic annotations.The authors identify reducing this sensitivity as future work.
5. Experiments
Experiments evaluate SurroundOcc on nuScenes and SemanticKITTI for occupancy prediction and reconstruction, including challenging conditions, ablations, and efficiency. The method achieves state-of-the-art results while benefiting from dense supervision, multi-scale design, and spatial attention.
- Experimental Setup: nuScenes experiments use 200x200x16 occupancy outputs covering [−50m, 50m] in X and Y and [−5m, 3m] in Z, with 0.5m voxels.Training uses the nuScenes training set and evaluation uses its validation set.
- Evaluation Metrics: Scene completion is evaluated with occupied-voxel IoU, while semantic scene completion uses mIoU across semantic classes.For reconstruction, predictions are converted to point clouds and evaluated mainly with Chamfer distance and F-score against dense occupancy ground truth.
- 3D Semantic Occupancy Prediction: SurroundOcc achieves state-of-the-art performance for multi-camera 3D semantic occupancy prediction on nuScenes.Qualitative results include rainy-day and nighttime scenes.
- 3D Semantic Occupancy Prediction: SurroundOcc also achieves state-of-the-art monocular 3D semantic scene completion performance on SemanticKITTI despite not being designed for monocular perception.The benchmark uses the SemanticKITTI test set and compares against RGB-inferred versions of prior methods.
- 3D Scene Reconstruction: SurroundOcc achieves state-of-the-art performance on most 3D reconstruction metrics and outperforms other methods by a large margin.The comparison includes multi-camera and monocular depth estimation methods and 3D reconstruction methods on the nuScenes validation set.
- Ablation Study: Dense occupancy supervision improves performance over sparse LiDAR supervision, while Poisson reconstruction and nearest-neighbor processing further densify the labels.The ablation also reports gains from multi-scale structure, multi-scale supervision, and 3D-based spatial attention.
- Efficiency: SurroundOcc achieves high performance and efficiency, with only slightly higher inference time and memory than BEVFormer.Efficiency experiments use six 1600x900 multi-camera images on one RTX 3090.
6. Limitations and Future Work
The paper is limited to single-frame occupancy prediction and identifies occupancy flow from multi-frame inputs as future work. It also highlights self-supervised occupancy prediction from RGB-only data as a challenging direction because LiDAR is not always available.
- Limitations and Future Work: The current method explores only single-frame occupancy prediction, while downstream motion prediction and planning require occupancy flow.Future work proposes multi-frame surrounding images and an occupancy flow dataset.
- Limitations and Future Work: LiDAR data is not always available, motivating self-supervised occupancy prediction using only RGB data as a valuable but challenging direction.
7. Conclusion
SurroundOcc combines multiscale 2D–3D feature integration with 3D deconvolution and introduces a pipeline for dense occupancy ground truth. Comparisons on nuScenes and SemanticKITTI demonstrate the method’s superiority.
- SurroundOcc integrates 2D features into 3D volumes using multiscale 2D–3D spatial attention, then upsamples and fuses them with 3D deconvolution.
- The ground-truth pipeline separately stitches multi-frame LiDAR points for dynamic objects and static scenes before filling holes with Poisson Reconstruction.
- The resulting occupancy predictions are reported as much denser than those of TPVFormer, whose sparse predictions are attributed to lacking dense supervision.
- Comparisons on nuScenes and SemanticKITTI demonstrate the superiority of SurroundOcc.
A. Baseline Method Details
The nuScenes baselines are organized into depth-estimation, occupancy-prediction, and related method categories. Depth-estimation baselines include SurroundDepth, AdaBins, and NeWCRFs, with additional supervision used for SurroundDepth.
- The nuScenes baselines are roughly classified into four categories.
- Depth estimation: The depth-estimation category includes SurroundDepth, AdaBins, and NeWCRFs.
- Depth estimation: SurroundDepth is evaluated as a multi-camera self-supervised method using depth ground truth together with self-supervised photometric loss.
- Depth estimation: AdaBins and NeWCRFs are described as state-of-the-art depth-estimation methods in outdoor and indoor scenes.
B. More Visualizations
Figure 8 presents qualitative comparisons with other methods, while accompanying demonstrations cover validation predictions, generated ground truth, comparisons, and a wild Beijing-street setting. Table 9 defines metrics for 3D scene reconstruction.
- Figure 8 provides qualitative comparisons between SurroundOcc and other methods.
- The reported visualizations show SurroundOcc predictions as more accurate and denser than those of other methods.
- The demonstrations include nuScenes validation results, generated ground truth, method comparisons, and Beijing-street predictions from a nuScenes-trained model.
- Table 9 lists evaluation metrics for 3D scene reconstruction and identifies p and p∗ as predicted and ground-truth point clouds.