PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugin

Zichen Yu, Changyong Shu, Jiajun Deng, Kangjie Lu, Zongdai Liu, Jiangyong Yu, Dawei Yang, Hui Li, Yan Chen

arXiv:2311.12058v1cs.CV

TL;DR

Voxel-level occupancy methods face substantial computation demands that hinder deployment, while occupancy prediction addresses object-detection gaps. FlashOCC replaces 3D convolutions with 2D BEV processing and Channel-to-Height reshaping; experiments report advantages over prior state-of-the-art methods in precision, runtime, and memory efficiency.

  • Problem

    Voxel-level representations introduce substantial computation that challenges deployment, while 3D object detection is limited to predefined classes and misses intricate geometry.

    Source: §1 · p. 2

  • Method

    FlashOCC replaces voxel-level 3D convolutions with 2D BEV processing and reshapes BEV outputs into 3D occupancy logits using Channel-to-Height.

    Source: §1 · p. 2

  • Results

    Experiments across diverse voxel-level methods report FlashOCC’s superiority over previous state-of-the-art approaches in precision, time consumption, memory efficiency, and deployment-friendliness.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that FlashOCC provides a plug-and-play approach to fast, memory-efficient occupancy prediction with deployment potential.

    Source: §5 · p. 8

    The LS view transformer assumes uniformly distributed depth, which can misalign features and cause false detections along camera rays.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Given the capability of mitigating the long-tail deficiencies and intricate-shaped absence prevalent in 3D object detection, occupancy prediction has become a pivotal component in autonomous driving systems. However, the procession of three-dimensional voxel-level representations inevitably introduces large overhead in both memory and computation, obstructing the deployment of to-date occupancy prediction approaches. In contrast to the trend of making the model larger and more complicated, we argue that a desirable framework should be deployment-friendly to diverse chips while maintaining high precision. To this end, we propose a plug-and-play paradigm, namely FlashOCC, to consolidate rapid and memory-efficient occupancy prediction while maintaining high precision. Particularly, our FlashOCC makes two improvements based on the contemporary voxel-level occupancy prediction approaches. Firstly, the features are kept in the BEV, enabling the employment of efficient 2D convolutional layers for feature extraction. Secondly, a channel-to-height transformation is introduced to lift the output logits from the BEV into the 3D space. We apply the FlashOCC to diverse occupancy prediction baselines on the challenging Occ3D-nuScenes benchmarks and conduct extensive experiments to validate the effectiveness. The results substantiate the superiority of our plug-and-play paradigm over previous state-of-the-art methods in terms of precision, runtime efficiency, and memory costs, demonstrating its potential for deployment. The code will be made available.

1. Introduction

Occupancy prediction addresses object-detection gaps by labeling voxels, but voxel-level processing is computationally demanding. FlashOcc replaces 3D convolutions with 2D BEV processing and a Channel-to-Height transform.

1. Introduction

  • Occupancy prediction assigns semantic classes to 3D voxels, addressing detection’s predefined-class limits and its difficulty capturing intricate shapes.It can label objects outside predefined categories as general objects.Source: §1 · p. 2
  • Voxel-level representations can be accurate, but their 3D convolutions and other operators create computation and on-chip deployment challenges.Sparse and tri-perspective representations conserve memory but do not fundamentally resolve deployment and computation challenges.Source: §1 · p. 2
  • FlashOcc replaces 3D convolutions with 2D convolutions and reshapes BEV features into 3D occupancy logits through Channel-to-Height.The approach is presented as a general, plug-and-play enhancement to existing models.Source: §1 · p. 2

2. Related Work

Prior occupancy work uses voxel-level representations for detailed 3D structure, while BEV features offer a more efficient alternative. FlashOcc applies a sub-pixel-inspired Channel-to-Height transform to BEV features for occupancy prediction.

2. Related Work

  • Voxel-based occupancy work includes sparse and dense perception; voxelization represents each voxel with a vector, but its computational and deployment costs motivate alternatives.OGMs aimed to extract 3D scene structure for planning and navigation; dense perception shares similarities with semantic scene completion.Source: §2 · pp. 2–3
  • BEV methods represent an entire pillar with a grid feature, reducing height-dimension representation and avoiding 3D convolutions.BEV features can implicitly capture height information, including uneven roads and suspended objects.Source: §2 · p. 3
  • FlashOcc reshapes flattened BEV features into occupancy logits with Channel-to-Height, applying the sub-pixel paradigm to occupancy without computational 3D convolutions.The paper describes this as the first use of the sub-pixel paradigm for occupancy using BEV features exclusively.Source: §2 · p. 3

3. Framework

FlashOcc is a plug-and-play framework for real-time surround-view occupancy prediction, designed to avoid costly voxel-level feature processing. It combines image-to-BEV processing with an occupancy head and an optional temporal fusion module.

3. Framework

  • FlashOcc takes surround-view images to dense occupancy prediction and comprises image encoding, view transformation, BEV encoding, occupancy prediction, and optional temporal fusion.The framework is intended to enhance existing models and avoid costly voxel-level feature processing.Source: §3 · p. 3

3.1. Image Encoder

  • The image encoder extracts multi-scale image features with a backbone and fuses them in a neck; the paradigm permits alternative backbone and neck architectures.Examples include ResNet or Swin Transformer backbones and FPN-LSS, NAS-FPN, or BiFPN necks.Source: §3.1 · p. 3

3.2. View Transformer

  • The view transformer maps perceptive-view image features into BEV; LSS projects using dense depth and camera parameters, then pools vertically, while LS assumes uniform depth.The LS assumption can cause feature misalignment and false detections along camera rays, despite reduced computational complexity.Source: §3.2 · pp. 3–4
  • Figure 2 marks replaceable modules with a dashed box, uses icons for 2D, BEV, and voxel features, and shows optional temporal fusion in light blue controlled by a red switch.The caption says the figure is best viewed in color and with zoom functionality.Source: Figure 2 · p. 4

3.3. BEV Encoder

  • The BEV encoder refines coarse BEV features with a backbone and neck; feature diffusion addresses missing center features for LSS or aliasing artifacts for LS.Two multi-scale features are integrated to improve representation quality.Source: §3.3 · p. 4

3.4. Occupancy Prediction Module

  • The occupancy head’s BEV feature is reshaped along channels into 3D occupancy logits, with C = C* × Z.The stated output shape is B × C* × Z × W × H, where C* is the class count and Z the vertical dimension.Source: §3.4 · p. 4

3.5. Temporal Fusion Module

  • The optional temporal fusion module aligns historical BEV features using ego information, then fuses them with current inputs to represent dynamic objects or attributes.It contains spatio-temporal alignment and feature fusion components.Source: §3.5 · p. 4

4. Experiment

The experiments evaluate FlashOcc on Occ3D-nuScenes and test its accuracy, generalization, temporal performance, and resource use across occupancy models. Comparisons report gains over existing methods alongside lower computation and memory costs.

4.2. Comparison with State-of-the-art Methods

  • FlashOcc improves BEVDetOcc by 1.3 mIoU and UniOcc by 0.3 mIoU, while FO(BEVDetOcc) exceeds PanoOcc by 1.1 mIoU.The comparison uses the Occ3D-nuScenes validation set.Source: Table 1 · p. 5, §4.2 · p. 6

4.1. Experimental Setup

  • Occ3D-nuScenes provides 700 training scenes and 150 validation scenes, with 0.4 m voxels across 17 categories and mIoU as the reported metric.The stated spatial range is −40 m to 40 m on X and Y, and −1 m to 5.4 m on Z.Source: §4.1 · p. 5
  • Experiments apply FlashOcc to BEVDetOcc, UniOcc, and FBOcc, following their original training protocols with AdamW and a 1e-4 learning rate.BEVDetOcc and UniOcc train for 24 epochs; FBOcc trains for 20.Source: §4.1 · p. 6

4.3. Ablation Study

  • Against the 3D voxel-level approach, M0 loses 0.6 mIoU while increasing speed from 92.1 FPS to 210.6 FPS; M1 gains 0.8 mIoU at 60.6 Hz.The comparison replaces 3D convolutions with 2D counterparts and applies Channel-to-Height at the model output.Source: Table 3 · p. 7, §4.3 · p. 7
  • Across model substitutions, FO(BEVDetOcc) gains 1.7 mIoU, FO(UniOcc) loses 0.2 mIoU, and FO(FBOcc) gains 0.1 mIoU against their originals.The substitutions replace 3D convolutions with 2D convolutions and use Channel-to-Height output logits.Source: Table 4 · p. 7, §4.3 · p. 7
  • With temporal modules, FlashOcc gains 5.4 mIoU versus 4.5 for BEVDetOcc; it improves UniOcc by 6.1 mIoU and FBOcc by 2.6 mIoU overall.The reported FlashOcc improvements over the corresponding baselines are 1.7 mIoU for temporal BEVDetOcc, 0.5 for non-temporal UniOcc, and 0.1 for temporal FBOcc.Source: Table 5 · p. 7, §4.3 · pp. 7–8

5. Conclusion

FlashOcc is a plug-and-play approach for fast, memory-efficient occupancy prediction, replacing voxel-level 3D convolutions with 2D convolutions and a Channel-to-Height transformation of flattened BEV features. Experiments report advantages over previous state-of-the-art methods, and visualizations show that FlashOcc preserves height information.

  • Experiments across diverse voxel-level occupancy methods report superiority over previous state-of-the-art approaches in precision, time consumption, memory efficiency, and deployment-friendliness.Source: §5 · p. 8
  • FlashOcc replaces 3D convolutions with 2D convolutions and reshapes flattened BEV features into occupancy logits through Channel-to-Height transformation.Source: §5 · p. 8
  • The authors describe this as the first use of the sub-pixel Channel-to-Height paradigm for occupancy prediction using BEV-level features exclusively, avoiding computational 3D convolutions and transformer modules.Source: §5 · p. 8
  • Visualization results demonstrate that FlashOcc preserves height information.Source: §5 · p. 8
  • Future work will explore integrating FlashOcc into autonomous-driving perception pipelines for efficient on-chip deployment.Source: §5 · p. 8
Loading 2311.12058v1…