Source-linked AI summary
FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation
Zhiqi Li, Zhiding Yu, David Austin, Mingsheng Fang, Shiyi Lan, Jan Kautz, Jose M. Alvarez
TL;DR
The paper addresses camera-only 3D occupancy prediction, where every voxel must receive occupancy and semantic labels. FB-OCC combines forward-backward view transformation with joint depth-semantic pre-training, model scaling, and post-processing. It achieves a 54.19% mIoU on nuScenes and ranks first in the challenge track.
Problem
3D occupancy prediction requires estimating occupancy status and semantic class for every voxel using camera input during inference.
Method
FB-OCC builds on FB-BEV by combining forward voxel projection with backward BEV projection, joint depth-semantic pre-training, large-scale modeling, and weighted post-processing.
Results
54.19% mIoU on the nuScenes test set, achieved with a seven-model ensemble, ranked first in the challenge track.
Takeaways & Limitations
The report identifies visual foundation models and large-scale pre-training as effective components for 3D occupancy prediction.
Abstract
from arXiv · showhide
This technical report summarizes the winning solution for the 3D Occupancy Prediction Challenge, which is held in conjunction with the CVPR 2023 Workshop on End-to-End Autonomous Driving and CVPR 23 Workshop on Vision-Centric Autonomous Driving Workshop. Our proposed solution FB-OCC builds upon FB-BEV, a cutting-edge camera-based bird's-eye view perception design using forward-backward projection. On top of FB-BEV, we further study novel designs and optimization tailored to the 3D occupancy prediction task, including joint depth-semantic pre-training, joint voxel-BEV representation, model scaling up, and effective post-processing strategies. These designs and optimization result in a state-of-the-art mIoU score of 54.19% on the nuScenes dataset, ranking the 1st place in the challenge track. Code and models will be released at: https://github.com/NVlabs/FB-BEV.
1. Introduction
3D occupancy prediction assigns occupancy status and semantic class to every voxel, supporting autonomous-vehicle perception and planning. The challenge focuses on camera-only inference while enabling open-source data and models for exploring data-driven algorithms and large-scale models.
- 3D occupancy prediction estimates each voxel’s occupancy status and semantic class in a 3D voxel space.
- Camera-only inference is required in the challenge track.
- Open-source datasets and models enable exploration of data-driven algorithms and large-scale models.
- The report highlights limited knowledge about how large-scale models and pre-training affect 3D occupancy prediction.
2. Method
FB-OCC combines forward projection into 3D voxels with backward projection over BEV features, then enhances occupancy prediction through pre-training, scaling, and post-processing. Its pipeline uses joint depth-semantic supervision, temporal and geometric augmentation, and weighted ensembling.
- 2.1. Model design: FB-BEV combines forward and backward view transformation, using forward projection for initial voxels and backward projection to enrich BEV-derived geometry.The fused voxel and optimized BEV representations feed the occupancy prediction head.
- 2.1. Model design: Forward projection models detailed 3D voxel representations, while backward projection optimizes sparse representations using computationally cheaper BEV features.Forward projection follows LSS principles and uses depth supervision; backward projection draws inspiration from BEVFormer.
- 2.1. Model design: The final 3D voxel representation combines the original voxel features with expanded optimized BEV features before occupancy prediction.The voxel encoder and occupancy prediction head process this fused representation.
- 2.2. Scaling up and pre-training: The method scales to the 1B-parameter InternImage-H backbone while using additional Object365 data to address overfitting from limited nuScenes samples.The report identifies large-scale models and pre-training as important factors in its solution.
- 2.2. Scaling up and pre-training: Joint depth estimation and semantic segmentation pre-training uses ground-truth depth maps and automatically generated 2D semantic masks.SAM generates masks from box prompts for thing categories and LiDAR-point prompts for stuff categories.
- 2.3. Post-processing: Test-time processing averages eight flipped predictions, replaces selected static distant voxels using prior frames, and combines model outputs with learned voxel weights.Ensemble weights multiply overall-result mIoU weights by category-specific IoU weights and are searched automatically with NNI.
3. Experiments
Experiments evaluate camera-only 3D occupancy prediction on Occ3D-nuScenes using mIoU, ablations of training and temporal strategies, and models at increasing scales. The final seven-model ensemble achieves 54.19% mIoU on the test set.
- Dataset and metrics: The Occ3D-nuScenes dataset covers 18 voxel classes, including free space, and provides camera-visibility masks.Annotations cover [-40m, -40m, -1m, 40m, 40m, 5.4m] at 0.4m voxel resolution.
- Dataset and metrics: mIoU averages class-wise intersection-over-union using true positives, false positives, and false negatives across all classes.The metric is the primary challenge evaluation measure.
- Ablation study: The ablation progresses from a vanilla baseline through depth supervision, invisible-voxel masking, bug fixes, 16-frame temporal input, joint depth-semantic pre-training, Dice loss, 3D alignment, and test-time augmentation.These milestones correspond to Versions A through H.
- Model scaling: Larger models use larger backbones and image inputs, progressing through VoVNet-99, ViT-L with ViT-Adapter, and a more-than-1-billion-parameter InternImage-H model.The scaling configurations are summarized in Table 2.
- Final submission: 54.19% mIoU is achieved on the test set by combining seven models with different backbones through an ensemble.The final submission uses the ensemble for its best accuracy.
4. Conclusion
The report presents FB-OCC as a winning 3D occupancy prediction solution, combining state-of-the-art model design with visual foundation models and large-scale pre-training.
- Conclusion: FB-OCC demonstrates state-of-the-art model design for BEV perception and the effectiveness of visual foundation models and large-scale pre-training for 3D occupancy prediction.The report describes this as its concluding contribution.