Source-linked AI summary
OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception
Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, Xingang Wang
TL;DR
Existing semantic occupancy benchmarks are limited in urban-scene diversity and often evaluate only front-view predictions. OpenOccupancy extends nuScenes with dense surrounding annotations, establishes multiple baselines, and introduces CONet, which relatively improves baseline performance by ∼30%.
Problem
Existing occupancy benchmarks lack urban-scene diversity, and SemanticKITTI evaluates only front-view predictions.
Method
OpenOccupancy extends nuScenes with dense semantic occupancy annotations using AAP, establishes camera-, LiDAR-, and multi-modal baselines, and adds CONet's coarse-to-fine pipeline.
Results
∼30% relative performance improvement is reported for CONet over the baseline, while multi-modal fusion relatively improves camera- and LiDAR-based baselines by 47% and 29%.
Takeaways & Limitations
OpenOccupancy provides a surrounding semantic occupancy benchmark with dense nuScenes annotations and camera-, LiDAR-, and multi-modal evaluation baselines.
Abstract
from arXiv · showhide
Semantic occupancy perception is essential for autonomous driving, as automated vehicles require a fine-grained perception of the 3D urban structures. However, existing relevant benchmarks lack diversity in urban scenes, and they only evaluate front-view predictions. Towards a comprehensive benchmarking of surrounding perception algorithms, we propose OpenOccupancy, which is the first surrounding semantic occupancy perception benchmark. In the OpenOccupancy benchmark, we extend the large-scale nuScenes dataset with dense semantic occupancy annotations. Previous annotations rely on LiDAR points superimposition, where some occupancy labels are missed due to sparse LiDAR channels. To mitigate the problem, we introduce the Augmenting And Purifying (AAP) pipeline to ~2x densify the annotations, where ~4000 human hours are involved in the labeling process. Besides, camera-based, LiDAR-based and multi-modal baselines are established for the OpenOccupancy benchmark. Furthermore, considering the complexity of surrounding occupancy perception lies in the computational burden of high-resolution 3D predictions, we propose the Cascade Occupancy Network (CONet) to refine the coarse prediction, which relatively enhances the performance by ~30% than the baseline. We hope the OpenOccupancy benchmark will boost the development of surrounding occupancy perception algorithms.
1. Introduction
OpenOccupancy addresses limited diversity and front-view-only evaluation in existing occupancy benchmarks by introducing a surrounding benchmark built on nuScenes. It also provides dense annotations, modality-specific baselines, and CONet for more efficient high-resolution prediction.
- Semantic occupancy perception assigns semantic labels to every spatially occupied region within the perceptive range, supporting fine-grained 3D urban-scene understanding for safe driving.
- Existing benchmarks are mainly indoor, while SemanticKITTI is relatively small, lacks driving-scene diversity, and evaluates only front-view occupancy.
- The AAP pipeline efficiently annotates and densifies nuScenes occupancy labels using multi-frame LiDAR initialization, pseudo occupancy labels, and subsequent purification.
- Camera-, LiDAR-, and multi-modal baselines are established, with multi-modal fusion relatively improving camera- and LiDAR-based performance by 47% and 29%.
- OpenOccupancy extends nuScenes with dense semantic occupancy annotations and is presented as the first benchmark for surrounding occupancy perception in driving scenarios.
- ∼30% relative performance improvement is obtained by CONet, which adds a coarse-to-fine pipeline to address the computational burden of high-resolution occupancy prediction.
2. Related Work
Semantic occupancy benchmarks span indoor and driving scenarios, but most prior methods focus on front-view settings. Surrounding occupancy methods have begun using multi-view representations, though TPVFormer produces relatively sparse outputs.
- SUNCG, NYUv2, NYUCAD, SUN3D, SceneNN, and ScanNet provide benchmarks focused on indoor stationary scenarios.
- MonoScene is described as the first camera-based occupancy perception method that infers occupancy semantics from a single image.
- Most occupancy perception approaches focus on front-view indoor scenarios despite continued development of the field.
- TPVFormer generates surrounding occupancy predictions with a tri-perspective view representation, but its output is relatively sparse because it is designed for LiDAR segmentation.
3. The OpenOccupancy Benchmark
OpenOccupancy defines a surrounding semantic occupancy benchmark using dense nuScenes annotations and evaluates camera, LiDAR, and multi-modal approaches. It also introduces a coarse-to-fine CONet framework to construct high-resolution occupancy efficiently.
- 3.1. Surrounding Semantic Occupancy Perception: Surrounding semantic occupancy predicts complete volumetric occupancy and semantic labels from 360-degree inputs across the entire scene.The surround-view input covers approximately 5× the perceptive range of front-view sensors, making efficient high-resolution occupancy construction a core challenge.
- 3.2. nuScenes-Occupancy: OpenOccupancy extends nuScenes with dense semantic occupancy annotations using the AAP pipeline, which initializes labels through LiDAR point superimposition and purifies augmented labels with human annotation.The final annotation contains approximately 400K occupied voxels per frame, about 2× the density of the initial annotation, across 28130 training and 6019 validation frames with 17 semantic labels.
- 3.3. Evaluation Protocol: The benchmark evaluates occupancy over a [−51.2m, 51.2m] × [−51.2m, 51.2m] × [−3m, 5m] range at 0.2m voxel resolution using IoU and semantic mIoU.IoU treats all occupied voxels as one category, while semantic mIoU averages class-wise intersection-over-union scores and ignores the noise class.
- 3.4. OpenOccupancy Baselines: OpenOccupancy establishes camera-based, LiDAR-based, and multi-modal baselines that coherently learn surrounding occupancy from 360-degree inputs.The camera branch transforms multi-view 2D features into height-preserving 3D voxel features, while the LiDAR branch encodes voxelized points with sparse 3D convolutions.
- 3.4. OpenOccupancy Baselines: CONet refines coarse occupancy through a coarse-to-fine pipeline that splits occupied voxels into high-resolution queries and predicts fine-grained occupancy from sampled image and voxel features.The framework is designed to alleviate the computational burden of high-resolution occupancy prediction while building on the multi-modal baseline.
- 3.4. OpenOccupancy Baselines: The multi-modal baseline adaptively fuses LiDAR and camera voxel features before 3D decoding and occupancy prediction.Its fused features are produced through 3D convolution, channel concatenation, sigmoid weighting, and element-wise combination, and training uses cross-entropy, Lovasz-softmax, affinity, and camera-aware depth losses where applicable.
4. Cascade Occupancy Network
CONet addresses the computational cost of high-resolution surrounding occupancy prediction with a coarse-to-fine refinement pipeline. It generates high-resolution queries from coarse occupied voxels, samples semantic and geometric features, and produces fine-grained occupancy outputs.
- Motivation: The surrounding task covers approximately 5× the perceptive range of front-view occupancy, making high-resolution 3D prediction computationally burdensome.The baselines therefore use stride S=4 and output volumetric size (10 × 128 × 128).
- Coarse-to-fine pipeline: CONet first generates coarse occupancy with a baseline, then splits occupied voxels into high-resolution occupancy queries for refinement.The split ratio η is typically set to 4.
- Feature refinement: The refinement samples semantic features from multi-view images and geometric features from voxel features before fusion and regularization.The sampled features are used to produce fine-grained occupancy predictions that are reshaped into a volumetric representation.
- Output construction: CONet maps refined predictions to high-resolution query locations while assigning empty labels outside the queried occupied-voxel region.The pipeline can be generalized to camera-based and LiDAR-based baselines, with LiDAR-only CONet sampling geometric features.
5. OpenOccupancy Experiment
The experiments evaluate surrounding occupancy methods, baseline configurations, and CONet efficiency on nuScenes-Occupancy. Results show complementary camera and LiDAR strengths, adaptive multi-modal fusion benefits, and substantial CONet efficiency gains.
- Experiment setup: The benchmark evaluates camera-based, LiDAR-based, multi-modal, CONet, and modern occupancy approaches on nuScenes-Occupancy.Single-view methods process each camera view separately and concatenate their predictions for surrounding outputs.
- Surrounding occupancy assessment: The camera baseline improves TPVFormer by 26% on IoU and 32% on mIoU, while the multi-modal baseline improves several comparison methods by 41% and 42%.The reported multi-modal improvements are relative enhancements for 3DSketch, AICNet, LMSCNet, and JS3C-Net.
- Surrounding occupancy assessment: Camera-based methods perform better on small objects, whereas LiDAR-based methods perform better on large structured regions.Examples include bicycles, pedestrians, and motorcycles for small objects, and drivable surfaces and sidewalks for large regions.
- Surrounding occupancy assessment: 47% and 29% relative mIoU improvements result from the multi-modal baseline over the LiDAR-based and camera-based baselines, respectively.The multi-modal baseline adaptively fuses intermediate features from both modalities.
- Efficiency and effectiveness of CONet: CONet improves mIoU by 24%, 35%, and 33% on camera-based, LiDAR-based, and multi-modal baselines, respectively, with marginal latency overhead.Compared with high-resolution baselines, CONet reduces approximately 15 GB of training GPU memory and approximately 70% of GFLOPs.
- Baselines under different settings: 16% and 20% relative gains in IoU and mIoU come from increasing camera input size, while ResNet101 adds 11% mIoU over ResNet50.Using 10 LiDAR sweeps improves the single-sweep baseline by 43% on IoU and 5% on mIoU.
- Efficiency and effectiveness of CONet: Combining semantic and geometric feature sampling improves the baseline by 33% on mIoU, compared with 28% from geometric features alone.The ablation indicates that 2D semantic features alone are insufficient for high-resolution 3D predictions.
6. Conclusion
OpenOccupancy introduces a surrounding semantic occupancy benchmark and dense nuScenes-Occupancy annotations, alongside modality-specific baselines and CONet. Experiments report complementary camera and LiDAR performance, multi-modal gains, and approximately 30% relative improvement from CONet.
- Conclusion: OpenOccupancy is the first benchmark for surrounding semantic occupancy perception in driving scenarios.It introduces nuScenes-Occupancy by extending nuScenes with dense semantic occupancy annotations using the AAP pipeline.
- Conclusion: Camera-based and LiDAR-based baselines are complementary, while the multi-modal baseline relatively improves performance by 47% and 29%.The conclusion attributes these results to combining the two modalities within the benchmark baselines.
- Conclusion: CONet relatively improves the baseline by approximately 30% with minimal latency overhead.It is proposed to alleviate the computational burden of high-resolution occupancy predictions.