FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation

• 3D occupancy prediction estimates each voxel’s occupancy status and semantic class in a 3D voxel space.Source: §1 · p. 1
• Camera-only inference is required in the challenge track.Source: §1 · p. 1
• Open-source datasets and models enable exploration of data-driven algorithms and large-scale models.Source: §1 · p. 1
• The report highlights limited knowledge about how large-scale models and pre-training affect 3D occupancy prediction.Source: §1 · p. 1

2.1. Model design

• FB-BEV combines forward and backward view transformation, using forward projection for initial voxels and backward projection to enrich BEV-derived geometry.The fused voxel and optimized BEV representations feed the occupancy prediction head.Source: §2.1 · p. 1, Figure 1 · p. 3
• Forward projection models detailed 3D voxel representations, while backward projection optimizes sparse representations using computationally cheaper BEV features.Forward projection follows LSS principles and uses depth supervision; backward projection draws inspiration from BEVFormer.Source: §2.1 · pp. 1–2
• The final 3D voxel representation combines the original voxel features with expanded optimized BEV features before occupancy prediction.The voxel encoder and occupancy prediction head process this fused representation.Source: §2.1 · pp. 1–2, Figure 2 · p. 3

2.2. Scaling up and pre-training

• The method scales to the 1B-parameter InternImage-H backbone while using additional Object365 data to address overfitting from limited nuScenes samples.The report identifies large-scale models and pre-training as important factors in its solution.Source: §2.2 · p. 2
• Joint depth estimation and semantic segmentation pre-training uses ground-truth depth maps and automatically generated 2D semantic masks.SAM generates masks from box prompts for thing categories and LiDAR-point prompts for stuff categories.Source: §2.2 · p. 2, Figure 3 · p. 3

2.3. Post-processing

• Test-time processing averages eight flipped predictions, replaces selected static distant voxels using prior frames, and combines model outputs with learned voxel weights.Ensemble weights multiply overall-result mIoU weights by category-specific IoU weights and are searched automatically with NNI.Source: §2.3.1 · p. 2, §2.3.2 · p. 3

Dataset and metrics

• The Occ3D-nuScenes dataset covers 18 voxel classes, including free space, and provides camera-visibility masks.Annotations cover [-40m, -40m, -1m, 40m, 40m, 5.4m] at 0.4m voxel resolution.Source: §3.1 · p. 3
• mIoU averages class-wise intersection-over-union using true positives, false positives, and false negatives across all classes.The metric is the primary challenge evaluation measure.Source: §3.1 · p. 3

Ablation study

• The ablation progresses from a vanilla baseline through depth supervision, invisible-voxel masking, bug fixes, 16-frame temporal input, joint depth-semantic pre-training, Dice loss, 3D alignment, and test-time augmentation.These milestones correspond to Versions A through H.Source: §3.3 · p. 4

Model scaling

• Larger models use larger backbones and image inputs, progressing through VoVNet-99, ViT-L with ViT-Adapter, and a more-than-1-billion-parameter InternImage-H model.The scaling configurations are summarized in Table 2.Source: §3.4 · p. 4, Table 2 · p. 4

Final submission

• 54.19% mIoU is achieved on the test set by combining seven models with different backbones through an ensemble.The final submission uses the ensemble for its best accuracy.Source: §3.5 · p. 4

Conclusion

• FB-OCC demonstrates state-of-the-art model design for BEV perception and the effectiveness of visual foundation models and large-scale pre-training for 3D occupancy prediction.The report describes this as its concluding contribution.Source: §4 · p. 4