PaperlayerView exact revision on arXiv

Source-linked AI summary

OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction

Yunpeng Zhang, Zheng Zhu, Dalong Du

arXiv:2304.05316v1cs.CV

TL;DR

Vision-based occupancy prediction needs fine-grained 3D geometry and semantics beyond condensed BEV features, while existing 3D processing struggles with sparse voxel features and efficiency. OccFormer uses local and global transformer pathways with preserve-pooling and class-guided sampling, achieving state-of-the-art results on SemanticKITTI semantic scene completion and nuScenes camera-based LiDAR segmentation.

  • Problem

    3D semantic occupancy provides fine-grained geometry and semantics, but existing methods face sparse-feature, receptive-field, spatial-invariance, and parameter-efficiency limitations.

  • Method

    OccFormer decomposes 3D encoding into local and global transformer pathways and adapts Mask2Former with preserve-pooling and class-guided sampling for occupancy decoding.

  • Results

    OccFormer achieves state-of-the-art performance for semantic scene completion on SemanticKITTI and camera-based LiDAR segmentation on nuScenes.

  • Takeaways & Limitations

    The dual-path encoder captures local semantic structures and global scene layouts while the decoder adaptations address occupancy sparsity and class imbalance.

Abstract

from arXiv · show

The vision-based perception for autonomous driving has undergone a transformation from the bird-eye-view (BEV) representations to the 3D semantic occupancy. Compared with the BEV planes, the 3D semantic occupancy further provides structural information along the vertical direction. This paper presents OccFormer, a dual-path transformer network to effectively process the 3D volume for semantic occupancy prediction. OccFormer achieves a long-range, dynamic, and efficient encoding of the camera-generated 3D voxel features. It is obtained by decomposing the heavy 3D processing into the local and global transformer pathways along the horizontal plane. For the occupancy decoder, we adapt the vanilla Mask2Former for 3D semantic occupancy by proposing preserve-pooling and class-guided sampling, which notably mitigate the sparsity and class imbalance. Experimental results demonstrate that OccFormer significantly outperforms existing methods for semantic scene completion on SemanticKITTI dataset and for LiDAR semantic segmentation on nuScenes dataset. Code is available at \url{https://github.com/zhangyp15/OccFormer}.

1. Introduction

Vision-based 3D perception is moving beyond efficient but information-limited BEV representations toward fine-grained 3D semantic occupancy. OccFormer addresses the resulting 3D encoding and decoding challenges with dual-path transformers and an adapted Mask2Former decoder.

  • 3D semantic occupancy reconstructs the surrounding environment with fine-grained geometry and semantics from multi-view images.
  • BEV representations offer an effective performance–efficiency trade-off but cannot fully recover holistic 3D structure for variably shaped obstacles.
  • 3D convolution has fixed receptive fields, spatial invariance, and high parameter costs that limit its handling of sparse, discontinuous voxel features.
  • OccFormer decomposes 3D processing into local windowed attention on BEV slices and global attention on collapsed BEV features, then adaptively fuses both paths.
  • 1.24% mIoU improvement over MonoScene on SemanticKITTI corresponds to an 11% relative improvement and first place among monocular test methods.

2. Related Work

Related work includes BEV-based vision pipelines, LiDAR-dependent semantic scene completion, monocular 3D UNet methods, tri-perspective representations, and efficiency-oriented 3D architectures. OccFormer re-advocates dense 3D features with a transformer encoder-decoder.

  • Vision-based 3D perception commonly projects image features into voxelized BEV representations for downstream tasks.
  • Most semantic scene completion methods rely on geometrical inputs with explicit depth, while MonoScene introduced monocular semantic scene completion using sight projection and a 3D UNet.
  • TPVFormer uses tri-perspective views for semantic occupancy, but its tri-plane representation can lose fine-grained semantic information.
  • OccFormer re-advocates dense 3D features and proposes a transformer-based encoder-decoder network for semantic occupancy prediction.
  • Efficiency-oriented SSC methods reduce 3D computation through sparse grouped convolutions, consecutive 1D convolutions, anisotropic kernels, or 2D UNets.

3. Approach

OccFormer lifts multi-scale image features into 3D volumes, encodes them with complementary local and global transformer paths, and decodes occupancy through mask classification. Its design addresses 3D processing efficiency, sparse occupancy supervision, and class imbalance while outperforming existing monocular methods on SemanticKITTI.

  • 3.1. Overview: OccFormer extracts multi-scale image features, lifts them into a 3D feature volume, and predicts occupancy through a transformer-based encoder-decoder.The image encoder produces fused perspective features, while the image-to-3D transformation uses depth distributions, an outer product, and voxel pooling.
  • 3.2. Dual-path Transformer Encoder: The dual-path encoder applies shared windowed attention to horizontal slices for local structures and processes collapsed BEV features for global semantic layouts.The two outputs are adaptively fused to produce the output 3D feature volume.
  • 3.2. Dual-path Transformer Encoder: The dual-path transformer uses mostly 2D reasoning and shared modules, requiring fewer parameters and less computation than classic 3D convolutions.The encoder still aggregates local semantic structures and global semantic layouts for semantic reasoning.
  • 3.3. Transformer Occupancy Decoder: The occupancy decoder adapts Mask2Former by combining multi-scale deformable attention, iterative query updates, per-voxel embeddings, and binary 3D mask prediction.Each query predicts semantic logits and a mask embedding, which forms a voxel mask through a dot product with per-voxel embeddings and a sigmoid.
  • 3.3.3. Preserve-Pooling: Preserve-pooling replaces trilinear interpolation because the trivial trilinear adaptation does not adequately handle sparse 3D semantic occupancy masks.The supplied passages identify the limitation of trilinear interpolation but do not provide the full preserve-pooling mechanism.
  • Experiments: OccFormer outperforms existing monocular methods on SemanticKITTI semantic scene completion in both SC IoU and SSC mIoU.Table 1 reports this comparison on the SemanticKITTI test set.
  • 3.3.4. Class-Guided Sampling: Class-guided sampling uses inverse class frequencies to increase supervision for rare occupancy classes, despite minor degradation on larger classes.The method computes class frequencies, reciprocal weights, normalization, and a power-law sampling weight controlled by β.

3.4. Loss Functions

OccFormer trains its occupancy model with matched predicted and ground-truth segments, using sampled positions and class-plus-mask costs. It also supervises the intermediate depth distribution with projected LiDAR points.

  • Hungarian bipartite matching pairs predicted and ground-truth segments using class and binary mask losses over sampled positions.
  • Projected LiDAR points supervise the intermediate depth distribution used for view transformation.

4. Experiments

Experiments evaluate OccFormer for monocular semantic scene completion on SemanticKITTI and vision-based LiDAR segmentation on nuScenes. The results show strong benchmark performance, while ablations and qualitative comparisons support the dual-path encoder and decoder designs.

  • Datasets: SemanticKITTI evaluates monocular semantic scene completion using the left camera, with voxelized ground truth and 21 annotated classes.The occupancy grid is 256 × 256 × 32, with 0.2m × 0.2m × 0.2m voxels.
  • Datasets: nuScenes evaluates LiDAR semantic segmentation with sparse LiDAR point supervision, using the camera-based occupancy model to query pointwise logits.
  • Main Results: 1.06 mIoU OccFormer gain over TPVFormer is reported for SemanticKITTI semantic scene completion, with comparable scene-completion IoU on validation data.
  • Main Results: OccFormer outperforms the only vision-based nuScenes competitor and achieves comparable performance with LiDAR-based methods for LiDAR semantic segmentation.One model performs both LiDAR segmentation and semantic occupancy prediction, unlike TPVFormer’s separately trained models.
  • Ablation Studies: Both local and global encoder paths improve performance, and their dual-path combination offers a better trade-off than vanilla 3D convolution and 3D windowed attention.
  • Ablation Studies: Multi-scale 3D deformable attention outperforms a 3D-adapted FPN as the pixel decoder.
  • Ablation Studies: Max-pooling improves naive Mask2Former adaptation by about 0.5 mIoU, while class-guided sampling outperforms uniform sampling for sparse 3D supervision.
  • Qualitative Results: Qualitative results show more complete scene layouts, recovered object structures, and better neighboring-class interactions than MonoScene and TPVFormer.OccFormer’s occupancy predictions are described as more contiguous, complete, and realistic in the nuScenes comparison.

5. Conclusion

OccFormer is presented as a dual-path transformer network for camera-based 3D semantic occupancy prediction. Its encoder and occupancy decoder address 3D feature processing, sparsity, and class imbalance, with state-of-the-art reported benchmark performance.

  • OccFormer processes camera-generated 3D voxel features with local and global transformer pathways that capture fine-grained details and scene-level layouts.
  • Preserve-pooling and class-guided sampling improve occupancy prediction under inherent sparsity and class imbalance.
  • OccFormer achieves state-of-the-art semantic scene completion on SemanticKITTI test data and camera-based LiDAR segmentation on nuScenes test data.
  • The nuScenes validation table reports that OccFormer surpasses TPVFormer and first exceeds 70% mIoU using only multi-view images.

A.1. LiDAR Segmentation Results

OccFormer achieves strong nuScenes LiDAR segmentation performance using only multi-view images, while jointly predicting semantic occupancy helps address inaccurate 3D positioning.

  • OccFormer is the first method to achieve 70%+ mIoU for nuScenes LiDAR segmentation using only multi-view images.
  • OccFormer can notably outperform TPVFormer with smaller input sizes and a ResNet-50 backbone, although its advantage decreases with larger configurations.The reduced advantage may reflect saturation of vision-based methods.
  • Jointly solving LiDAR segmentation and semantic occupancy mitigates inaccurate 3D positions observed in TPVFormer's occupancy predictions.

A.2. More Ablation Studies

Ablations show that dual-path encoder components and training augmentations materially affect occupancy performance, particularly through fusion, attention, long-range context, and 3D transformations.

  • Encoder Modules: Removing soft fusion of dual-path outputs drops SSC mIoU from 13.46 to 12.73.
  • Encoder Modules: Removing global-path windowed attention degrades performance by around 0.5 mIoU.The global-path weights are shared with the local path.
  • Encoder Modules: The global-path bottleneck ASPP extracts long-range information for scene-level semantic layouts.
  • Augmentations: Augmentation techniques are essential for reducing over-fitting and improving performance because the attention mechanism has strong capacity.
  • Augmentations: Disabling joint 3D augmentation causes best performance at epoch 9 despite a 30-epoch training schedule.

A.3. Analysis

Class-guided sampling strengthens supervision for rare classes, producing substantial gains for several infrequent object categories despite small declines on larger classes.

  • Class-guided sampling greatly improves supervision signals for rare classes.
  • Class-guided sampling notably boosts truck, person, bicyclist, and traffic-sign performance while slightly degrading road, sidewalk, and parking.
  • Different sampling patterns provide an approach for model ensemble.

B. More Visualizations

OccFormer predicts dense semantic occupancy from multi-view images and sparse LiDAR supervision, covering background structure and accurately locating foreground objects.

  • OccFormer predicts dense vegetation, driveable-surface, and building occupancy from multi-view images trained with sparse LiDAR points.
  • Cars, pedestrians, and trucks can be located accurately in the predicted 3D semantic occupancy.
  • The predicted 3D semantic occupancy provides comprehensive and fine-grained scene understanding.
Loading 2304.05316v1…