PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene as Occupancy

Chonghao Sima, Wenwen Tong, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, Hongyang Li

arXiv:2306.02851v3cs.CVcs.RO

TL;DR

The paper addresses limited scene representations and evaluation for vision-centric autonomous driving, where boxes lose fine geometry and prior occupancy work focuses mainly on scene completion. It proposes OccNet, a temporal cascade voxel pipeline with a general occupancy descriptor, and OpenOcc, a dense nuScenes-based benchmark. Across downstream tasks, the paper reports improved perception and planning, including a 15%-58% collision-rate reduction for motion planning.

  • Problem

    Vision-centric autonomous driving needs a geometry-aware representation that preserves detailed 3D scene structure, while prior occupancy work mainly focuses on scene completion and lacks broader full-stack evaluation.

  • Method

    OccNet reconstructs 3D occupancy from multi-view images with a cascade voxel decoder, temporal clues, and task-specific heads, while OpenOcc provides dense annotations for evaluation.

  • Results

    15%-58% collision-rate reduction is reported for motion planning versus policies based on BEV segmentation or 3D boxes, alongside gains across perception tasks.

  • Takeaways & Limitations

    3D occupancy functions as a general scene representation supporting 3D perception and planning beyond detection, with dense representations proving effective for vision-only models.

  • Takeaways & Limitations

    The current annotations remain based on an established dataset, leaving self-supervised learning to reduce human-annotation cost as future work.

Abstract

from arXiv · show

Human driver can easily describe the complex traffic scene by visual system. Such an ability of precise perception is essential for driver's planning. To achieve this, a geometry-aware representation that quantizes the physical 3D scene into structured grid map with semantic labels per cell, termed as 3D Occupancy, would be desirable. Compared to the form of bounding box, a key insight behind occupancy is that it could capture the fine-grained details of critical obstacles in the scene, and thereby facilitate subsequent tasks. Prior or concurrent literature mainly concentrate on a single scene completion task, where we might argue that the potential of this occupancy representation might obsess broader impact. In this paper, we propose OccNet, a multi-view vision-centric pipeline with a cascade and temporal voxel decoder to reconstruct 3D occupancy. At the core of OccNet is a general occupancy embedding to represent 3D physical world. Such a descriptor could be applied towards a wide span of driving tasks, including detection, segmentation and planning. To validate the effectiveness of this new representation and our proposed algorithm, we propose OpenOcc, the first dense high-quality 3D occupancy benchmark built on top of nuScenes. Empirical experiments show that there are evident performance gain across multiple tasks, e.g., motion planning could witness a collision rate reduction by 15%-58%, demonstrating the superiority of our method.

1. Introduction

The paper presents 3D occupancy as a geometry-aware scene representation for detailed perception and planning, then proposes OccNet and OpenOcc to support and evaluate this broader role.

  • Motivation: 3D occupancy quantizes scenes into structured cells with semantic labels, preserving irregular object and background geometry that 3D boxes oversimplify.Examples include protruding vehicle parts such as a construction truck’s crane arm.
  • Research gap: Prior work largely treats occupancy as scene completion, while temporal geometry, coarse-to-fine reconstruction, and full-stack evaluation remain insufficiently addressed.The paper identifies omitted temporal context, one-stage decoding, and the lack of a practical vision-centric benchmark as open issues.
  • Approach: OccNet reconstructs occupancy from multi-view images using a cascade voxel decoder with temporal clues and provides task-specific heads for perception and planning.Its occupancy descriptor supports 3D occupancy prediction, 3D detection, BEV segmentation, and motion planning.
  • Benchmark: OpenOcc is a dense, high-quality nuScenes-based benchmark with 16 semantic classes and directional flow annotations that account for object motion.The annotations cover foreground objects and background stuff and are designed to extend evaluation toward planning.
  • Results: 14% relative improvement is reported over TPVFormer in semantic scene completion, while OccNet-based pre-training raises detection performance by about 10 points on small-scale fine-tuning data.The reported detection comparison is against FCOS3D.
  • Results: 15%-58% collision-rate reduction is reported for motion planning compared with policies based on BEV segmentation or 3D boxes.The paper attributes these evaluations to occupancy predictions from OccNet.

2. Related Work

Related work spans box-based detection, LiDAR segmentation, reconstruction, and semantic scene completion, while the paper positions occupancy as a broader vision-centric descriptor.

  • 3D detection: 3D object detection commonly uses standardized 3D boxes, which structure downstream rules but oversimplify object shapes and focus on foreground objects.Occupancy instead represents objects with different shapes in a fine-grained voxel description.
  • LiDAR segmentation: LiDAR segmentation provides point-level understanding but requires costly point-cloud input and faces limited sensing range and sparse 3D descriptions.These properties make holistic 3D scene semantic understanding less suitable for such a pipeline.
  • 3D reconstruction and rendering: Prior 3D reconstruction and rendering work often targets a single object or scene, emphasizing geometry and appearance more than efficiency and semantic understanding.The paper notes that this setting does not directly fit autonomous-driving applications requiring strong generalization.
  • Semantic scene completion: Semantic scene completion evaluates 3D occupancy, but prior literature generally neglects temporal context and occupancy as a descriptor for tasks beyond detection.The paper also notes that coarse-to-fine decoding is ignored by one-stage methods.

3. Methodology

OccNet reconstructs a fine-grained 3D occupancy descriptor from multi-view images by combining temporal information with a cascade voxel decoder, then exposes it to multiple driving-task heads.

  • Framework: OccNet reconstructs occupancy features from images through separate reconstruction and exploitation stages connected by a unified occupancy descriptor.The descriptor supports downstream driving tasks rather than serving only scene completion.
  • Reconstruction Stage: The reconstruction stage fuses surrounding-image features with historical BEV features and current BEV queries to produce the current BEV feature.Its BEV encoder follows the BEVFormer structure.
  • Cascade Voxel Decoder: A cascade decoder progressively recovers voxel height information while increasing spatial resolution and reducing channel dimensions across stages.Intermediate voxel features are lifted from BEV features and refined through successive voxel decoders.
  • Voxel Attention: Each voxel decoder combines voxel-based temporal self-attention with voxel-based spatial cross-attention to refine the occupancy representation.Temporal refinement uses history voxel features, while spatial cross-attention interacts with multi-scale image features.
  • Efficient Attention: 3D Deformable Attention limits each voxel query to local voxels of interest, making 3D temporal attention computationally affordable.The method extends 2D deformable attention to 3D and uses sampled points with trilinear interpolation.
  • Task Exploitation: Task-specific heads predict voxel semantics with focal loss, occupied-voxel flow with L1 loss, and BEV or planning outputs for additional driving tasks.The heads cover semantic scene completion, BEV segmentation, and motion planning.

4. OpenOcc: 3D Occupancy Benchmark

OpenOcc is a multi-view 3D occupancy benchmark built on nuScenes with dense semantic and flow annotations. Its annotations support detailed geometry learning and evaluation of occupancy prediction against existing methods.

  • Benchmark Overview: OpenOcc is introduced as the first 3D occupancy benchmark built on nuScenes with surrounding camera views and corresponding occupancy and flow annotations.This expands beyond benchmarks using only front-camera input.
  • Dataset Scale: The benchmark contains 34,149 annotated frames from 700 training and 150 validation scenes, covering over 1.4 billion voxels across 16 classes.The classes include 10 foreground-object classes and 6 background-stuff classes.
  • Motion Annotations: OpenOcc adds directional flow annotations for foreground-object voxels, enabling object motion information to support motion planning.The flow annotations complement dense semantic occupancy labels.
  • Representation Quality: OpenOcc provides a denser, more complete scene representation than sparse occupancy alternatives while preserving local fine-grained geometry and flow information.The visual comparison contrasts OpenOcc with sparse occupancy and OccData.
  • Annotation Generation: The annotation process separates static background and moving foreground LiDAR points before accumulating them, avoiding errors from directly transforming intermediate-frame points.Unlabeled intermediate-frame points are also used to improve occupancy density.
  • Evaluation: For semantic scene completion, Table 3 compares RGB-input models using semantic mIoU and geometric IoUgeo, with OccNet reported to outperform previous state-of-the-art methods.The table distinguishes methods trained and evaluated on OpenOcc with an asterisk.

5. Experiments

Experiments evaluate OccNet on occupancy, perception, detection, segmentation, planning, efficiency, and irregular-object recognition. Results consistently support dense occupancy and occupancy-based representations across these tasks.

  • Semantic Scene Completion: OccNet outperforms BEVDet4D, BEVDepth, BEVDet, and TPVFormer on semantic scene completion, exceeding TPVFormer by 3.31 mIoU points.The reported scores against TPVFormer are 26.98 versus 23.67.
  • Occupancy for LiDAR Segmentation: 60.46 versus 65.50 mIoU makes camera-only OccNet comparable with LiDAR-based RangeNet++, while bicycle IoU is higher at 32.58 versus 21.30.The comparison is reported for LiDAR segmentation on the nuScenes validation set.
  • Downstream Tasks: Occupancy pretraining improves BEV segmentation IoU, while joint occupancy-detection training improves 3D detection performance.The experiments compare occupancy pretraining with detection pretraining and evaluate joint training on nuScenes.
  • Occupancy for Planning: 15%-58% collision-rate reduction is obtained from OccNet occupancy predictions in planning experiments.Using all 16 occupancy classes also improves L2 distance and helps avoid background-object collisions.
  • Model Efficiency: OccNet achieves the best mIoU and IoUgeo among the compared model structures while retaining efficiency and effectiveness.Efficiency and performance are evaluated in the semantic scene completion task on a V100 GPU.

6. Conclusion

The paper concludes that occupancy is an effective scene representation for evaluating multiple driving tasks, supported by the OpenOcc benchmark. It identifies annotation cost as a limitation and self-supervised learning as future work.

  • Conclusion: OccNet and OpenOcc validate the effectiveness of 3D occupancy as a scene representation across varied downstream driving tasks.The paper characterizes occupancy as superior to conventional alternatives in its initial conclusion.
  • Limitations and future work: Current annotations remain based on an established dataset, leaving self-supervised learning to reduce human-annotation cost as future work.The authors also state that they hope occupancy can become a foundation model for autonomous driving.

Appendix

The appendix places evaluation metrics, related work, visualizations, implementation and training details, ablations, and post-processing details in supplementary materials.

  • Appendix: Supplementary materials contain evaluation metrics, related work, visualizations, implementation and training details, ablations, and BEVNet, VoxelNet, and OpenOcc post-processing details.

A. Evaluation Metrics

The evaluation uses task-specific metrics for semantic scene completion, detection, and motion planning, including both geometric quality and planning safety.

  • Semantic Scene Completion: Semantic scene completion predicts semantic labels for 3D voxels and uses mean intersection-over-union across the 16 benchmark classes.IoUgeo additionally evaluates class-agnostic geometric reconstruction quality.
  • 3D Object Detection: The nuScenes detection evaluation reports NDS, mAP, ATE, ASE, AOE, AVE, and AAE.
  • Motion Planning: Planning evaluation measures L2 distance for trajectory regression accuracy and collision rate against other vehicles and pedestrians for action safety.
  • Representation Context: BEV segmentation requires explicit height information in challenging urban settings, whereas 3D occupancy represents scene geometry directly.

C. Implementation Detail of OccNet

OccNet combines multi-view image features, temporal BEV encoding, and cascaded voxel decoding to reconstruct 3D occupancy. Its implementation also defines single-scale VoxelNet and BEVNet comparison models.

  • Backbone and Multi-scale Features: ResNet101 with FPN extracts multi-scale features from multi-view images at 1/16, 1/32, and 1/64 resolutions.The selected backbone stages are S3, S4, and S5, with three aggregated feature levels.
  • BEV Encoder: The BEV encoder uses two layers with temporal self-attention and spatial cross-attention to refine a scene representation in BEV space.Its structure follows BEVFormer, using a spatial-temporal-transformer mechanism.
  • Feature Transformation in Voxel Decoder: The voxel decoder transforms feature dimensions between voxel scales with MLPs and projects FPN image features into the required channel dimensions.The transformations support spatial cross-attention for voxel features.
  • Training Strategy: OccNet is trained for 24 epochs with AdamW, a 2×10^-4 learning rate, batch size 1 per GPU, six images, and weight decay 1 × 10^-2.Perception tasks are trained together except BEV segmentation; other tasks are fine-tuned from frozen tasks.
  • Comparison Models: VoxelNet and BEVNet use single-scale feature maps, contrasting with OccNet’s cascaded feature-map design.VoxelNet directly constructs voxel features, while BEVNet reshapes BEV features into voxel features.

D. More Detail about OpenOcc

OpenOcc is generated by accumulating foreground and background points, refining annotations, removing noise, and postprocessing for scene completeness. The benchmark contains dense semantic occupancy labels and motion flow annotations.

  • Data Generation: Intermediate-frame object boxes are approximated by linear interpolation between adjacent key frames to accumulate dense foreground LiDAR points.The pipeline separates object points from background points before accumulation.
  • Data Generation: The occupancy data-generation pipeline progressively annotates unknown background points, removes remaining noise, and postprocesses the scene for completeness.The final output is a dense, high-quality 3D occupancy benchmark.
  • Dataset Statistics: OpenOcc annotates 16 classes across 34,149 frames from 700 training and 150 validation scenes, covering over 1.4 billion voxels.The benchmark has substantial class imbalance: ten foreground classes comprise 5.33% of labels, with bicycles at 0.02% and motorcycles at 0.03%.
  • Motion Annotations: Eight foreground classes receive directional flow annotations, with moving objects defined using a velocity threshold of vth = 0.2m/s.The reported moving-object proportion exceeds 50%, supporting motion-aware downstream use.
  • Class Distribution: Background stuff forms the majority of occupancy classes in OpenOcc.The class distribution is visualized in Figure 9.
  • Class Distribution: The benchmark reports occupancy with velocity for the eight movable foreground classes among ten foreground object classes.The excluded foreground classes are not considered movable for this analysis.

E. More Experiments

Additional experiments examine temporal context, occupancy-based planning metrics, pretraining, decoder structure, and voxel resolution. They generally favor more temporal frames, direct occupancy inputs, cascaded decoding, and finer voxels, while exposing a planning-pretraining boundary.

  • Temporal Self-Attention: Increasing temporal frames improves semantic scene completion and LiDAR segmentation until performance slows at a four-frame threshold.Insufficient historical frames hurt performance to some extent.
  • Planning Metrics: Occupancy inputs are more advantageous than alternative scene representations across most planning intervals under occupancy-based collision metrics.The evaluation counts collisions over foreground occupancy voxels and four background classes.
  • Planning Pretraining: Pretrained occupancy features do not directly improve planning, so the experiments recommend applying scene-completion occupancy results directly instead.This comparison replaces ST-P3’s perception module with pretrained OccNet and fine-tunes planning.
  • Semantic Scene Completion: OccNet outperforms BEVNet and VoxelNet in semantic scene completion, supporting cascaded voxel structure as a stronger occupancy descriptor.The ablation compares the three model designs directly.
  • Voxel Resolution: LiDAR segmentation performance increases as voxel resolution decreases, and camera-based OccNet can match LiDAR-based performance as ∆s →0.The evaluated resolutions are ∆s ∈ {1.0m, 0.5m, 0.25m}.

F. Visualization Results

Visualizations show OccNet’s occupancy predictions describing scene geometry and semantics in detail. Planning visualizations further contrast occupancy with bounding-box rasterization, while the tables report comparative model and resolution experiments.

  • Model Comparison: The semantic scene-completion ablation reports OccNet as superior to BEVNet and VoxelNet.This visualization section references the comparative experiment summarized in Table 14.
  • Resolution Comparison: The LiDAR-segmentation comparison reports best performance for the method using the smallest voxel resolution ∆s.The associated experiment evaluates OccNet with a ResNet50 backbone on the nuScenes validation set.
  • Planning Visualization: Rasterized occupancy enables a planned trajectory to maintain greater safety distance from a truck than rasterized bounding boxes.The caption attributes this to occupancy’s more accurate polygon representation.
  • Occupancy Prediction: OccNet visualizations present camera input alongside perspective and top views of predicted occupancy.The examples indicate that small or distant targets can be predicted well.
Loading 2306.02851v3…