PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, Zhi Tang

arXiv:2205.13790v3cs.CV

TL;DR

Existing LiDAR-camera fusion methods depend on LiDAR point clouds to query image features, making them vulnerable to LiDAR malfunction in realistic autonomous driving. BEVFusion separates camera and LiDAR streams in a shared BEV space before fusion, and it outperforms prior methods under normal and robustness settings, with reported robust gains of 15.7% to 28.9% mAP.

  • Problem

    Current LiDAR-camera fusion methods intrinsically depend on raw LiDAR point clouds, so LiDAR malfunction can prevent meaningful predictions and limit realistic deployment.

  • Method

    BEVFusion encodes camera and LiDAR inputs through independent streams into a shared BEV space, then fuses the resulting features for task-specific prediction.

  • Results

    15.7% to 28.9% mAP: BEVFusion surpasses state-of-the-art fusion methods under robustness training settings simulating LiDAR malfunctions.

  • Takeaways & Limitations

    The framework provides robust LiDAR-camera fusion across normal and malfunction settings and can generalize to multiple modern architectures.

  • Takeaways & Limitations

    The study uses a simple camera stream and leaves temporal multi-view input and intermediate LiDAR-camera feature alignment for future work.

Abstract

from arXiv · show

Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discovered that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion, whose camera stream does not depend on the input of LiDAR data, thus addressing the downside of previous methods. We empirically show that our framework surpasses the state-of-the-art methods under the normal training settings. Under the robustness training settings that simulate various LiDAR malfunctions, our framework significantly surpasses the state-of-the-art methods by 15.7% to 28.9% mAP. To the best of our knowledge, we are the first to handle realistic LiDAR malfunction and can be deployed to realistic scenarios without any post-processing procedure. The code is available at https://github.com/ADLab-AutoDrive/BEVFusion.

1 Introduction

Existing LiDAR-camera fusion methods use LiDAR inputs to query image features, so LiDAR malfunctions can prevent meaningful predictions. BEVFusion separates camera and LiDAR processing in BEV space, achieving strong accuracy and robustness.

  • Motivation: Current fusion methods query image features using LiDAR points or proposals, making their predictions intrinsically dependent on raw LiDAR point clouds.Point-level methods project image features onto raw points, while feature-level methods project LiDAR features or proposals into camera views.
  • Motivation: LiDAR malfunctions such as low reflection, transfer glitches, or incomplete field of view can make existing fusion methods fail to produce meaningful results.This dependency limits applicability in realistic autonomous-driving systems.
  • BEVFusion: BEVFusion uses independent camera and LiDAR streams that encode both modalities into a shared BEV space before fusion and task-specific prediction.The camera stream projects multi-view image features into 3D ego-car coordinates, while LiDAR models encode point clouds into BEV features.
  • Results: 69.2% mAP: BEVFusion exceeds TransFusion’s 68.9% mAP under normal nuScenes training settings.It also improves PointPillars and CenterPoint by 18.4% and 7.1% mAP, respectively.
  • Results: 15.7% ∼28.9% mAP: BEVFusion surpasses state-of-the-art fusion methods under robustness training settings simulating LiDAR malfunctions.The contribution summary reports gains under both normal and robust settings.

2 Related Works

Prior 3D detection work spans camera-only, LiDAR-only, and LiDAR-camera fusion approaches. BEVFusion addresses a central fusion limitation by disentangling the camera branch from LiDAR point clouds while remaining generalizable across architectures.

  • Camera-only: Camera-only 3D detection has progressed from monocular methods toward multi-view approaches enabled by datasets such as nuScenes and Waymo.Lift-Splat-Shoot transforms multi-perspective image features into 3D ego-car coordinates using estimated depth.
  • LiDAR-only: LiDAR-only methods operate directly on raw point clouds or transform them into voxel and pillar feature spaces.Some models combine multiple LiDAR feature modalities to increase representation power.
  • LiDAR-camera fusion: LiDAR-camera fusion methods query image features through projected LiDAR points or features, then combine information from both modalities.These methods are categorized as point-level or feature-level fusion according to their fusion mechanism.
  • LiDAR-camera fusion: Existing fusion mechanisms heavily rely on LiDAR point clouds and can fail when LiDAR input is missing, hindering deployment in realistic settings.BEVFusion addresses this issue by disentangling the camera branch from LiDAR point clouds.
  • Scope: BEVFusion differs from one concurrent multimodal method by supporting generalization to arbitrary architectures rather than one specific detection head.Other camera-radar fusion approaches are outside this work’s scope.

3 BEVFusion: A General Framework for LiDAR-Camera Fusion

BEVFusion disentangles camera and LiDAR processing by encoding both modalities independently into a shared BEV space, then dynamically fusing their features. The framework supports adaptable camera and LiDAR streams and common detection-head architectures.

  • Framework overview: BEVFusion introduces independent camera and LiDAR streams that encode raw sensor inputs into features within the same BEV space.The framework then incorporates these modality-specific features through a dynamic fusion module.
  • Camera stream architecture: The camera stream adapts Lift-Splat-Shoot with an image-view encoder, view projector, and BEV encoder to produce camera BEV features.The image-view encoder uses Dual-Swin-Tiny with an FPN, while the view projector converts image features into 3D ego-car coordinates using predicted depth and camera extrinsics.
  • LiDAR stream architecture: The LiDAR stream can use any network that transforms LiDAR points into BEV features, including PointPillars, CenterPoint, and TransFusion.These streams typically voxelize raw points and use sparse 3D convolutions to produce BEV representations.
  • Dynamic fusion module: The dynamic fusion module concatenates camera and LiDAR BEV features, applies static channel-spatial fusion, then uses adaptive channel attention to select important features.The static fusion function uses a 3×3 convolution, while adaptive selection uses global average pooling, a linear transform, and a sigmoid function.
  • Detection heads: The fused BEV representation enables BEVFusion to use anchor-based, anchor-free-based, and transform-based detection heads.This design supports the framework's use with multiple common detection-head categories.

4 Experiments

Experiments on nuScenes evaluate BEVFusion’s generalization, state-of-the-art performance, robustness to LiDAR and camera malfunctions, and component effectiveness.

  • Generalization and state of the art: The LiDAR stream consistently outperforms the camera stream, reflecting LiDAR’s robust local boundary and surface-normal features for bounding-box prediction.These single-stream results are reported because BEVFusion uses a two-stage training scheme.
  • Robustness experiments: Under limited LiDAR FOV, BEVFusion improves PointPillars by 24.4% and 25.1% mAP and improves TransFusion-L by over 18.6% mAP and 5.3% NDS.The reported FOV ranges are (−π/2, π/2) and (−π/3, π/3) for PointPillars; performance degrades as LiDAR FOV decreases.
  • Robustness experiments: With object-level LiDAR-point dropping and robustness finetuning, BEVFusion improves PointPillars, CenterPoint, and TransFusion-L by 28.9%, 22.7%, and 15.7% mAP.The augmentation drops objects or their interior LiDAR points with the probabilities described in the robustness protocol.
  • Robustness experiments: BEVFusion remains superior under front-camera loss, preservation of only the front camera, and 50% stuck camera frames.The comparison includes camera-only and other LiDAR-camera fusion methods.
  • Ablation studies: Ablations show that the camera stream reaches 22.9% mAP and 31.1% NDS, while dynamic fusion raises PointPillars, CenterPoint, and TransFusion mAP by 16.5%, 5.9%, and 2.4%.Adaptive feature selection adds further mAP gains of 1.9%, 1.2%, and 0.6%, respectively.

5 Conclusion

BEVFusion is a LiDAR-camera fusion framework that disentangles modality dependence through separate camera and LiDAR streams in a shared BEV space. Experiments demonstrate robustness and generalization against camera and LiDAR malfunctions.

  • BEVFusion uses separate streams to encode raw camera and LiDAR inputs into a shared BEV space before feature fusion.The fused features can be passed into modern task prediction heads.
  • The framework disentangles the LiDAR-camera fusion dependency present in previous methods.
  • Experiments demonstrate strong robustness and generalization against various camera and LiDAR malfunctions.

Broader Impacts Statement and Limitations

The paper studies robust LiDAR-camera fusion for 3D object detection and notes safety risks from models biased toward training data. It uses a simple camera stream and leaves temporal multi-view input and intermediate-feature alignment for future work.

  • Models biased toward training data may pose safety threats when deployed in practice.
  • The study uses a simple camera stream as its baseline and encourages extensions with temporal multi-view input and intermediate LiDAR-camera feature alignment.
  • Extending the method toward such systems is left for future work.

Checklist

The checklist records ethics, reproducibility, asset, and broader-impact disclosures, including code availability, training details, fixed-seed experiments, and stated limitations.

  • [Yes] The authors state that the main claims accurately reflect the paper’s contributions and scope.
  • [Yes] The paper describes its limitations and discusses potential negative societal impacts.
  • [Yes] The checklist states that personally identifiable information is blurred or absent from the publicly released data.
  • [N/A] Error bars are not reported because experiments are described as stable across multiple runs using a fixed seed.
  • The supplementary document provides implementation details, robustness analyses, distance-based performance gains, latency and memory measurements, and failure-case visualizations.

A Network architectures

The adaptive module operates on multi-scale FPN features, bringing them to a common resolution for subsequent processing. The section also specifies the detection metrics used to evaluate the system.

  • Network architecture: The adaptive module is applied on top of a standard Feature Pyramid Network and processes multi-scale features F2 through F5.The feature resolutions range from H/4×W/4×C to H/32×W/32×C.
  • Network architecture: The module upsamples multi-scale FPN features to the common shape H/4×W/4×C.The described operations include torch.nn.Upsample and torch.nn.AdaptiveAvgPool2d.
  • Evaluation metrics: The evaluation reports mAP, ATE, ASE, AOE, AVE, AAE, and NDS for 3D object detection.mAP averages BEV-center-distance results across four thresholds and ten object classes, while NDS consolidates mAP with other indicators.

B.1 Implementation details.

BEVFusion training uses separate modality streams before fusion, with robustness experiments reported under varied malfunction settings and a specified fine-tuning schedule.

  • Training procedure: Training begins by separately training the LiDAR and camera streams before fusion.The streams use their respective modality inputs and official MMDetection3D settings.
  • Robustness evaluation: Robustness experiments cover different lighting conditions and both camera and LiDAR malfunctions.These evaluations are reported in Tables 10 and 9, respectively.
  • Fine-tuning schedule: Fusion detectors are fine-tuned for 12 epochs with learning-rate reductions by 10× at epochs 8 and 11.The initial learning rate is 1e−4 and the batch size is 32.

C.1 Robustness of Dynamic Fusion Module

The Dynamic Fusion Module improves robustness to LiDAR failures and maintains effectiveness under camera, LiDAR, and lighting perturbations.

  • LiDAR malfunction: 15.5% mAP improvement is obtained for the LiDAR stream with simple channel& spatial fusion when object reflection points are missing.Adaptive feature selection is also evaluated under the same robustness setting.
  • Joint malfunction: BEVFusion remains effective under a certain degree of simultaneous camera and LiDAR malfunction.The authors qualify this robustness by noting that objects unseen by both sensors cannot be identified.
  • Camera conditions: BEVFusion shows the best robustness under different lighting conditions compared with CenterPoint and TransFusion.The evaluation uses the nuScenes validation set.

D Performance for different distance regions

BEVFusion improves both camera and LiDAR streams across distance ranges and reports latency, failure-case behavior, and distant-region comparisons with TransFusion.

  • Distance-based comparison: Table 12 compares distant-region performance between BEVFusion and TransFusion.The comparison includes original-paper and re-implemented TransFusion results.
  • Distance-based performance: 10%-35.4%, 14.8%-50.2%, and 16.3-44.3% mAP gains improve the camera stream for regions under 15m, 15-30m, and over 30m.These gains are reported across the three distance ranges.
  • Distance-based performance: 1%-4.6%, 3.3%-7.2%, and 5.8%-9.3% mAP gains improve the LiDAR stream across the same three distance ranges.The ranges are under 15m, 15-30m, and over 30m, respectively.
  • Latency: The 2D->3D projector in the camera stream costs more than 957 ms, making it the latency bottleneck rather than the fusion module.The authors note that concurrent processing could improve deployment latency.
  • Failure cases: When one stream fails and the other succeeds, BEVFusion can balance the streams and detect objects; when both fail, it also fails.The failure cases use BEVFusion with PointPillars as the LiDAR stream.
Loading 2205.13790v3…