PaperlayerView exact revision on arXiv

Source-linked AI summary

Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving

Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, Hang Zhao

arXiv:2304.14365v3cs.CV

TL;DR

Existing 3D perception methods rely on bounding boxes that miss geometric detail and predefined categories that omit general objects. Occ3D builds validated, visibility-aware occupancy labels and two benchmarks, then introduces CTF-Occ, which achieves superior performance on those benchmarks.

  • Problem

    3D bounding-box methods neglect fine geometry and struggle with out-of-vocabulary objects, motivating voxel-level prediction of occupancy and semantics.

  • Method

    Occ3D uses a three-stage label-generation pipeline and establishes Waymo- and nuScenes-derived benchmarks, while CTF-Occ aggregates 2D image features into 3D space through coarse-to-fine cross-attention.

  • Results

    The pipeline generates dense, visibility-aware annotations, and CTF-Occ achieves superior 3D occupancy prediction performance on Occ3D benchmarks.

  • Takeaways & Limitations

    Occ3D provides a released benchmark, label-generation protocol, and model to facilitate future research in 3D occupancy prediction.

  • Takeaways & Limitations

    Out-of-vocabulary objects are grouped into a general-object class, and finer human annotation is identified as a route to more detailed labels.

Abstract

from arXiv · show

Robotic perception requires the modeling of both 3D geometry and semantics. Existing methods typically focus on estimating 3D bounding boxes, neglecting finer geometric details and struggling to handle general, out-of-vocabulary objects. 3D occupancy prediction, which estimates the detailed occupancy states and semantics of a scene, is an emerging task to overcome these limitations. To support 3D occupancy prediction, we develop a label generation pipeline that produces dense, visibility-aware labels for any given scene. This pipeline comprises three stages: voxel densification, occlusion reasoning, and image-guided voxel refinement. We establish two benchmarks, derived from the Waymo Open Dataset and the nuScenes Dataset, namely Occ3D-Waymo and Occ3D-nuScenes benchmarks. Furthermore, we provide an extensive analysis of the proposed dataset with various baseline models. Lastly, we propose a new model, dubbed Coarse-to-Fine Occupancy (CTF-Occ) network, which demonstrates superior performance on the Occ3D benchmarks. The code, data, and benchmarks are released at https://tsinghua-mars-lab.github.io/Occ3D/.

1 Introduction

Occ3D addresses the limited geometry and category coverage of 3D bounding boxes by introducing dense, visibility-aware occupancy benchmarks and a coarse-to-fine model. Its pipeline handles sparsity, occlusion, and 3D-2D misalignment through validated label generation.

  • Motivation: 3D bounding boxes lose fine geometric details and omit uncommon objects outside predefined categories.These limitations motivate a representation covering detailed geometry and semantics within and beyond the ontology tree.
  • Dataset construction: The label-generation pipeline tackles sparsity, occlusion, and 3D-2D misalignment through voxel densification, occlusion reasoning, and image-guided voxel refinement.Each stage is validated with a 3D-2D consistency metric to produce dense, visibility-aware annotations.
  • Benchmarks: Occ3D provides Waymo- and nuScenes-derived benchmarks with surround-view images and high-resolution 3D voxel occupancy representations.The paper describes these benchmarks as covering more diverse scenarios than conventional datasets such as SemanticKITTI and KITTI-360.
  • Models and evaluation: The paper reproduces recent occupancy models on Occ3D and introduces CTF-Occ, a transformer-based coarse-to-fine network.CTF-Occ aggregates 2D image features into 3D space through cross-attention and achieves superior performance.

2 Related Work

The related work distinguishes 3D occupancy prediction from object detection, occupancy grid mapping, and semantic scene completion. The distinctions concern representation, sensing requirements, scene dynamics, and treatment of unobserved regions.

  • 3D detection: 3D object detection estimates object locations and dimensions within a predefined ontology, using LiDAR, cameras, or sensor fusion.Its compact bounding-box representation differs from voxel-level occupancy prediction.
  • Occupancy Grid Mapping: Occupancy Grid Mapping generates probabilistic maps from sequential noisy range measurements, assuming the scene remains static over time.It requires range sensors, unlike occupancy prediction for vision-only robotic systems in dynamic scenes.
  • Dataset comparison: The table compares occupancy datasets by dataset configuration, including surround-view inputs and camera, depth, and LiDAR modalities.The caption defines Surround as surround-view image inputs and C, D, and L as camera, depth, and LiDAR.
  • Semantic scene completion: Semantic Scene Completion infers dense semantic space from partial observations, focusing on occluded regions and typically static scenes.3D occupancy prediction does not intend to estimate invisible regions and operates in dynamic scenes.

3 Occ3D Dataset

Occ3D defines voxel-level occupancy prediction and provides two large-scale benchmarks with dense, visibility-aware labels. Its construction pipeline addresses sparsity, occlusion, and 3D-2D misalignment through densification, visibility reasoning, and image-guided refinement.

  • Task Definition: 3D occupancy prediction estimates each voxel’s occupancy state and semantic label from historical surround-view camera images.Voxel states are occupied, free, or unobserved; semantic labels identify categories for occupied voxels.
  • Dataset Overview: Occ3D-nuScenes contains 40,000 frames across 600 training, 150 validation, and 150 test scenes, with 16 common classes plus a general object class.
  • Dataset Overview: Occ3D-Waymo and Occ3D-nuScenes provide surround-view images and high-resolution 3D voxel occupancy representations across diverse scenarios.The benchmarks are derived from the Waymo Open Dataset, nuScenes, and Panoptic nuScenes datasets.
  • Dataset Construction Pipeline: The label pipeline combines voxel densification, occlusion reasoning, and image-guided voxel refinement to address sparsity, occlusion, and 3D-2D misalignment.Densification uses segmentation, separate static and dynamic aggregation, KNN label assignment, and mesh reconstruction.
  • Dataset Construction Pipeline: LiDAR and camera visibility masks mark voxels as occupied, free, observed, or unobserved through ray casting, restricting evaluation to jointly observed voxels.Image-guided refinement removes incorrectly occupied voxels when ray traversal reaches a voxel matching the pixel’s semantic label.

4 Quality Check

The benchmark quality check measures semantic consistency between projected 3D voxel labels and manually annotated 2D image pixels. It uses a LiDAR-defined image region, pixel-wise voxel queries, and standard segmentation metrics to evaluate pipeline design choices.

  • 3D-2D Consistency: Dataset quality is evaluated by checking semantic consistency between 3D voxel labels and corresponding 2D image pixels.Manually annotated 2D semantic masks provide the comparison labels.
  • 3D-2D Consistency: The 2D region of interest is limited to the maximum range covered by a single LiDAR frame.Projected LiDAR points determine the height boundary for selecting pixels in each image column.
  • 3D-2D Consistency: Corresponding 3D voxels are queried for each selected image pixel to avoid multi-pixel projection associations and complicated overlap-based occlusion decisions.
  • Metrics: Precision, Recall, IoU, and mIoU quantify agreement between image semantic labels and the semantic predictions of corresponding 3D voxels.
  • Quantitative Results: MFP improves recall over SFP but lowers precision, while voxelization and mesh reconstruction improve the precision-recall trade-off.The comparison evaluates successive auto-labeling pipeline choices on a subset of Occ3D-Waymo.

5 Coarse-to-Fine Occupancy Network

CTF-Occ predicts 3D occupancy by projecting multi-view image information into voxel space and refining representations progressively from coarse to fine. Its architecture combines an image backbone, a pyramid voxel encoder, and occupancy decoders including an implicit-resolution option.

  • Architecture: CTF-Occ aggregates multi-view 2D image features into 3D voxel space through cross-attention.
  • Coarse-to-Fine Voxel Encoder: Incremental token selection chooses foreground and uncertain voxel tokens for cross-attention, reducing computation while preserving accuracy.A binary classifier predicts whether each voxel is empty or non-empty at each pyramid level.
  • Architecture: The network consists of an image backbone, a coarse-to-fine voxel encoder, and an implicit occupancy decoder.
  • Coarse-to-Fine Voxel Encoder: Spatial cross-attention selects top-K voxel tokens and aggregates the corresponding image features at every pyramid level.
  • Occupancy Decoder: The occupancy decoder maps voxel features to semantic occupancy predictions and can produce arbitrary-resolution outputs using implicit neural representations.The implicit decoder uses a voxel feature vector and a 3D coordinate inside the voxel as inputs to an MLP.

6 Experiments

The experiments evaluate existing occupancy models and CTF-Occ on Occ3D-nuScenes and Occ3D-Waymo, using IoU and mIoU under the stated experimental setup. CTF-Occ outperforms prior methods, while ablations support its coarse-to-fine components and OHEM loss.

  • Experimental Setup: Experiments evaluate existing 3D occupancy prediction methods on the Occ3D-nuScenes and Occ3D-Waymo benchmarks.
  • Experimental Setup: Occ3D-Waymo contains 1,000 sequences, while Occ3D-nuScenes contains 700 training and 150 validation scenes; both use 0.4m voxels.The occupancy ranges differ between the two datasets, and performance is evaluated with IoU and mIoU.
  • Comparing with previous methods: CTF-Occ surpasses BEVFormer by 1.65 mIoU on Occ3D-nuScenes and performs better in all classes under IoU.The observations are consistent with those on Occ3D-Waymo.
  • Comparing with previous methods: CTF-Occ increases mIoU by 1.97 on Occ3D-Waymo, with gains of 2.88 IoU for traffic cones and 10.23 IoU for vehicles.The authors attribute the gains to preserving detailed geometry in 3D voxel space without height compression.
  • Ablation study: Both incremental token selection and OHEM loss improve performance, with OHEM plus top-k token selection producing the best ablation result.Without OHEM, mIoU is 14.06; random selection with OHEM reaches 16.62, while uncertain selection with OHEM reaches 17.37 mIoU.

7 Conclusion

The paper concludes by presenting Occ3D, its label-generation protocol, and CTF-Occ, while identifying calibration, dynamic-object, and category-coverage boundaries for further improvement.

  • Conclusion: Occ3D is presented as a large-scale, high-quality 3D occupancy benchmark, alongside a rigorous label-generation protocol and CTF-Occ.The benchmark, protocol, and model are publicly released.
  • Limitations: LiDAR-camera calibration errors can affect occupancy-label construction, and precise calibration is also required for multi-frame aggregation.
  • Limitations: Dynamic or deformable objects without suitable box annotations or rigid-body behavior can produce motion-blur problems during label generation.Running animals and people swinging their arms are given as examples.
  • Limitations: nuScenes and Waymo annotate limited categories, so out-of-vocabulary objects such as trash cans and traffic cones are treated as general objects.The authors state that further human annotation could provide finer-grained details.

A Occ3D Dataset

Occ3D is released with supporting resources and benchmarks, while its label-generation pipeline remains subject to reconstruction and sensor-related limitations.

  • The Occ3D resources include the dataset, benchmark, develop kit, data format, annotation instructions, and a release website.
  • Occ3D-nuScenes and Occ3D-Waymo are published under the MIT license for non-commercial research, while their source datasets retain separate licenses.
  • The data are stored through Google Drive links, with planned long-term maintenance and regular accessibility checks.
  • The benchmark results are provided for Occ3D-nuScenes, while label-generation code is slated for release upon acceptance.
  • Occ3D-Waymo contains 798 training, 202 validation, and 150 testing scenes across 200,000 frames; Occ3D-nuScenes contains 700, 150, and 150 scenes across 40,000 frames.
  • The label-generation pipeline does not achieve perfect reconstruction, relying on precise sensor calibration and excluding deformable objects.

C General Objects

Occ3D represents general or unknown objects with occupancy grids and semantics rather than restricting them to predefined detection categories, and illustrates this capability across varied objects.

  • General Objects (GOs) are unknown objects that 3D semantic occupancy prediction can represent beyond predefined detection categories.
  • Out-of-vocabulary objects are represented by voxels labeled as ("occupied", "unknown").
  • The benchmark uses a clustering algorithm to handle objects labeled as unknown.
  • Examples include dustbins, shopping carts, a sidewalk board, and a flying banner, with corresponding voxels inside each bounding box.

D Visibility

The visibility pipeline uses ray casting to assign voxel states from LiDAR and camera views, explicitly marking occluded or unavailable regions as unobserved and evaluating 3D–2D semantic consistency.

  • Ray casting: Ray casting supports both aggregated LiDAR and camera visibility calculations through initialization and incremental traversal stages.
  • Ray casting: The traversal initializes ray parameters and advances voxel by voxel until reaching the ray’s final voxel or leaving the specified range.
  • Ray casting: The ray-casting implementation nudges endpoints by EPS=1e-9 and stops traversal when the grid distance exceeds DISTANCE=0.5.
  • LiDAR visibility: Aggregated LiDAR visibility initializes voxels as unobserved with free labels, then accumulates occupied points and assigns their labels.
  • LiDAR visibility: LiDAR ray casting counts traversed voxels as free and point-containing voxels as occupied.
  • Camera visibility: Camera visibility projects virtual points from image pixels into ego coordinates and updates LiDAR-derived voxel states for camera visibility.
  • Visualization: Visibility masks mark blind spots and occluded regions behind parked vehicles, tree trunks, walls, and buildings as unobserved.
  • 3D-2D consistency: 3D voxel semantics generally align with manually annotated 2D semantic labels, although mislabeled tree trunks can affect consistency metrics.

F Datasheet

The datasheet describes Occ3D as a benchmark derived from nuScenes and Waymo, with voxel semantic labels, visibility masks, metadata, documented splits, and acknowledged data noise.

  • Occ3D was created specifically as a benchmark for 3D occupancy prediction from surround-view images.
  • The benchmark jointly estimates each voxel’s occupancy state and semantics to represent complex geometry and background structure beyond 3D bounding boxes.
  • The dataset is presented by Tsinghua MARS Lab and sponsored by Tsinghua University.
  • Occ3D provides 40,000 nuScenes samples and 200,000 Waymo samples through its GitHub repository.
  • Each instance contains RGB images, a LiDAR point cloud, 3D voxel semantic ground truth, LiDAR and camera visibility masks, and metadata.
  • The datasets use source-data splits, including 600/150/200 nuScenes train/validation/test sequences and 798/202/150 Waymo sequences.
  • Dataset noise arises from LiDAR noise and pose inaccuracies.
  • The data are derived from published nuScenes and Waymo datasets using the proposed auto-labeling method.

F.4 Uses

Occ3D is intended for research use and future extensions, including downstream autonomous-driving prediction and planning. Its distribution, licensing, maintenance, and use are governed by stated conditions, while the dataset’s known biases and incomplete real-world coverage constrain deployment.

  • Uses: Occ3D is intended to support downstream autonomous-driving prediction and planning research.
  • Use boundaries: Because of known dataset biases, the uncurated dataset should be used solely for research and not for production models.
  • Distribution: The dataset is open-source and available through the Occ3D GitHub repository.
  • Licensing: Occ3D is published under an MIT license for non-commercial research, while its source datasets retain their respective licenses.
  • Maintenance: Tsinghua MARS Lab will continue hosting and supporting Occ3D through its GitHub page.

Checklist

The checklist records affirmative documentation of the paper’s claims, limitations, ethics review, assets, and reproducibility materials. It also notes that experiments were not repeated to report error bars and that the dataset has safety-related scope limitations.

  • Claims and scope: The paper states that its abstract and introduction accurately reflect its contributions and scope.
  • Ethics and limitations: The authors report that they described limitations, discussed potential negative societal impacts, and followed ethics review guidelines.
  • Limitations: The dataset cannot cover all real-world objects, and algorithms developed from it may pose safety risks when applied to real roads.
  • Reproducibility: The authors state that code, data, training details, and compute information needed for reproducibility are provided in the paper or appendix.
  • Experimental reporting: Experiments were not repeated multiple times, so error bars with respect to random seeds were not reported.
  • Assets: The work uses public assets, cites their creators, and reports the licenses for the nuScenes and Waymo datasets.
Loading 2304.14365v3…