PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, Song Han

arXiv:2205.13542v3cs.CV

TL;DR

Point-level fusion can discard most camera semantic density when projecting camera features onto LiDAR points, limiting semantic-oriented perception. BEVFusion instead unifies camera and LiDAR features in a shared BEV space, accelerates BEV pooling by more than 40×, and achieves state-of-the-art results across detection and map segmentation with lower computation cost.

  • Problem

    Point-level fusion projects camera features onto LiDAR points, matching less than 5% of camera features and severely hurting semantic-oriented tasks such as BEV map segmentation.

  • Method

    BEVFusion unifies multimodal features in a shared BEV representation that preserves geometric structure and semantic density, using optimized BEV pooling and task-specific heads.

  • Results

    BEVFusion achieves state-of-the-art performance on nuScenes 3D detection and BEV map segmentation, including 1.3% higher test-split mAP and NDS than TransFusion with 1.9× lower MACs.

  • Takeaways & Limitations

    BEVFusion provides a simple, efficient baseline for generic multi-task multi-sensor fusion beyond point-level fusion.

Abstract

from arXiv · show

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we break this deeply-rooted convention with BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than 40x. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on nuScenes, achieving 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9x lower computation cost. Code to reproduce our results is available at https://github.com/mit-han-lab/bevfusion.

I. INTRODUCTION

BEVFusion addresses the semantic loss of camera-to-LiDAR fusion by unifying camera and LiDAR features in BEV space. It accelerates this transformation and supports multi-task perception with strong detection and segmentation results.

  • Multi-sensor fusion combines cameras’ semantic information with LiDARs’ spatial information for accurate and reliable perception.
  • BEVFusion unifies multi-modal features in a shared BEV representation that preserves geometric structure and semantic density while supporting task-agnostic learning.
  • More than 40× speedup in view transformation comes from precomputation and interval reduction that address BEV pooling’s runtime bottleneck.BEV pooling alone accounts for more than 80% of model runtime.
  • BEVFusion achieves state-of-the-art 3D detection on nuScenes and Waymo and larger gains on BEV map segmentation than existing fusion methods.It achieves 6% higher mIoU than camera-only models and 13.6% higher mIoU than LiDAR-only models.
  • The framework delivers these results with 1.9× lower computation cost.

II. RELATED WORK

Prior 3D perception work develops LiDAR-based, camera-based, and proposal- or point-level fusion methods. BEVFusion instead performs generic sensor fusion in a shared BEV space.

  • LiDAR-Based 3D Perception: LiDAR-based detectors use PointNets, SparseConvNet, anchor-free designs, or two-stage RCNN architectures for 3D object detection.
  • Camera-Based 3D Perception: Camera-based methods extend image detectors, improve depth modeling, or detect objects in 3D space with object queries.
  • Multi-Sensor Fusion: Multi-sensor fusion methods include proposal-level approaches that extract image features for 3D proposals or object queries, alongside point-level fusion.
  • BEVFusion: BEVFusion performs sensor fusion in shared BEV space and treats foreground, background, geometric, and semantic information equally.It is presented as a generic multi-task multi-sensor perception framework.

III. METHOD

BEVFusion extracts modality-specific features, transforms them into a unified BEV representation, and processes the fused features with a convolution-based BEV encoder and task-specific heads.

  • Modality-specific encoders first extract features from the different sensor inputs.
  • View transformations convert multi-modal features into a unified BEV representation that preserves geometric and semantic information.
  • BEV pooling is accelerated with precomputation and interval reduction to address the view-transformation efficiency bottleneck.
  • A convolution-based BEV encoder processes the unified features before task-specific heads produce outputs for different 3D perception tasks.

A. Unified Representation

BEVFusion adopts BEV as a shared representation because camera and LiDAR views differ, while alternative projections lose geometric structure or camera semantic density.

  • Camera-to-BEV transformation is the key fusion step, and efficient BEV pooling reduces its reported latency by about 40×.
  • Camera and LiDAR features occupy different views, making direct feature fusion difficult because corresponding tensor elements may represent different spatial locations.
  • LiDAR-to-camera projection is geometrically lossy because neighboring depth-map pixels can be far apart in 3D space.This weakens camera-view processing for geometry-focused tasks such as 3D object detection.
  • Camera-to-LiDAR fusion retains less than 5% of camera features for a 32-beam LiDAR, hurting semantic-oriented tasks such as BEV map segmentation.
  • BEV preserves LiDAR geometric structure and camera semantic density while providing an output space compatible with almost all perception tasks.

B. Efficient Camera-to-BEV Transformation

BEVFusion converts camera features into BEV by predicting discrete depth distributions and pooling ray-sampled features into BEV grids. Optimized interval reduction removes the main aggregation bottleneck, reducing camera-to-BEV latency substantially.

  • Camera-to-BEV transformation: Each camera feature pixel is scattered along D discrete ray points, weighted by predicted depth probabilities, then aggregated within r × r BEV grids.The resulting feature point cloud has size NHWD and is flattened along the z-axis during pooling.
  • Efficiency bottleneck: Around 2 million camera feature points can be generated per frame, making BEV pooling far denser than LiDAR feature processing.For the stated workload, six cameras use 32×88 feature maps and 118 discretized depth samples.
  • Interval reduction: A specialized GPU kernel parallelizes aggregation directly over BEV grids, avoiding tree reduction and unused partial sums.The kernel assigns one GPU thread to each grid and avoids writing partial sums to DRAM.
  • Interval reduction: 40× faster camera-to-BEV transformation reduces latency from more than 500ms to 12ms.Interval reduction alone lowers feature aggregation latency from 500ms to 2ms.

C. Fully-Convolutional Fusion

After conversion to a shared BEV space, BEVFusion fuses camera and LiDAR features with an elementwise operator and uses convolutional encoding to address local spatial misalignment. More accurate depth estimation is identified as a potential future improvement.

  • Feature fusion: Camera and LiDAR BEV features are fused in the shared representation space with an elementwise operator such as concatenation.The shared BEV space enables direct multimodal fusion after view transformation.
  • Convolutional fusion: A convolution-based BEV encoder with residual blocks compensates for local misalignments caused by inaccurate depth estimation.The encoder is applied after feature fusion.
  • Scope and limitation: More accurate depth estimation, including ground-truth depth supervision, is left for future work.The paper presents this as a potential way to improve the view transformer.

D. Multi-Task Heads

BEVFusion applies task-specific heads to the fused BEV feature map, supporting both 3D object detection and multilabel BEV map segmentation.

  • Task-specific heads: Multiple task-specific heads operate on the fused BEV feature map for different 3D perception tasks.The framework uses a class-specific center heatmap and regression heads for detection, plus separate binary heads for segmentation classes.
  • 3D object detection: The detection head predicts object centers, size, rotation, and velocity.It uses a class-specific center heatmap together with several regression heads.
  • BEV map segmentation: Map segmentation is formulated as one binary semantic segmentation problem per class because categories may overlap.The segmentation head is trained with standard focal loss.

IV. EXPERIMENTS

BEVFusion is evaluated on camera-LiDAR 3D detection and BEV map segmentation, covering geometric- and semantic-oriented tasks. It achieves strong segmentation gains and is designed to extend to additional sensors and perception tasks.

  • Evaluation scope: BEVFusion evaluates camera-LiDAR fusion on 3D object detection and BEV map segmentation.These tasks cover both geometric- and semantic-oriented perception.
  • Extensibility: The framework can be extended to radars, event-based cameras, 3D object tracking, and motion forecasting.The paper presents these as potential additional sensor types and perception tasks.
  • BEV map segmentation: 13.6% higher BEV map segmentation performance is reported against state-of-the-art multi-sensor fusion methods on nuScenes validation.Improvements are consistent across different categories.
  • Robustness: BEVFusion is reported to improve single-modality models by +10.7 in rainy scenes and +12.8 in nighttime scenes.The table describes these gains as robustness under different lighting and weather conditions.

A. 3D Object Detection

BEVFusion achieves state-of-the-art 3D detection on nuScenes and Waymo while reducing computation and improving measured efficiency over prior methods.

  • BEVFusion achieves state-of-the-art nuScenes detection with 8.4 FPS inference on a desktop GPU.
  • 1.3% higher test-split mAP and NDS, with 1.9× fewer MACs and 1.3× lower measured latency than TransFusion.
  • 3.8% higher test-set mAP, 1.6× speedup, and 1.5× MACs reduction compared with PointPainting and MVP.
  • BEVFusion achieves state-of-the-art performance on the Waymo open dataset and outperforms DeepFusion using only one model with test-time augmentation.

B. BEV Map Segmentation

BEVFusion provides especially strong gains on semantic-centric BEV map segmentation, where point-level fusion methods are poorly suited to preserving dense camera semantics.

  • BEVFusion achieves an even larger performance boost on the semantic-centric BEV map segmentation task.
  • 8–13% higher performance lets camera-only BEVFusion outperform LiDAR-only baselines.
  • 6 mIoU improvement over monocular BEVFusion and more than 13% improvement over state-of-the-art sensor-fusion methods are achieved in the multi-modality setting.
  • PointPainting and MVP are less helpful for map components because they focus on foreground LiDAR points and foreground 3D objects.

V. ANALYSIS

BEVFusion remains effective across adverse weather, lighting, object scale, distance, and LiDAR sparsity, with larger gains where individual sensors are weaker.

  • Weather and Lighting: 10.7 mAP improvement in rainy weather narrows the performance gap between sunny and rainy scenarios for LiDAR-based detection.
  • Weather and Lighting: 12.8 mIoU improvement from multi-modal BEVFusion in daytime-versus-nighttime analysis demonstrates the value of geometric clues when cameras fail.
  • Sizes and Distances: BEVFusion improves over its LiDAR-only counterpart for both small and large objects, with greater gains for smaller and more distant objects.
  • Sparser LiDARs: 12% improvement in the 1-beam LiDAR scenario accompanies a 1.6× reduction in MACs compared with MVP.
  • BEVFusion accelerates camera-to-BEV transformation by more than 40 times and uses 1.5–1.9× less computation than existing solutions.
Loading 2205.13542v3…