PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection

Ziying Song, Lei Yang, Shaoqing Xu, Lin Liu, Dongyang Xu, Caiyan Jia, Feiyang Jia, Li Wang

arXiv:2403.11848v4cs.CV

TL;DR

Calibration errors can misalign LiDAR and camera features in BEV-based 3D detection. GraphBEV uses neighbor-aware depth and learnable offsets to address local and global misalignment, outperforming BEVFusion on nuScenes validation and under noisy misalignment.

  • Problem

    Calibration errors between LiDAR and camera sensors cause feature misalignment, while BEVFusion overlooks local depth and global BEV-feature misalignment.

    Source: §1 · pp. 2–3

  • Method

    GraphBEV combines graph-based neighbor-aware depth features in LocalAlign with learnable offsets in GlobalAlign to address local and global feature misalignment.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    GraphBEV achieves 70.1% mAP and 72.9% NDS on nuScenes validation, surpasses BEVFusion by +1.6% mAP and +1.5% NDS, and exceeds it by over 8.3% in noisy settings.

    Source: §1 · p. 3, 3D Object Detection · p. 10

  • Takeaways & Limitations

    GraphBEV bolsters the real-world applicability of BEV-based methods by addressing sensor calibration errors.

    Source: §1 · p. 3

Abstract

from arXiv · show

Integrating LiDAR and camera information into Bird's-Eye-View (BEV) representation has emerged as a crucial aspect of 3D object detection in autonomous driving. However, existing methods are susceptible to the inaccurate calibration relationship between LiDAR and the camera sensor. Such inaccuracies result in errors in depth estimation for the camera branch, ultimately causing misalignment between LiDAR and camera BEV features. In this work, we propose a robust fusion framework called Graph BEV. Addressing errors caused by inaccurate point cloud projection, we introduce a Local Align module that employs neighbor-aware depth features via Graph matching. Additionally, we propose a Global Align module to rectify the misalignment between LiDAR and camera BEV features. Our Graph BEV framework achieves state-of-the-art performance, with an mAP of 70.1\%, surpassing BEV Fusion by 1.6\% on the nuscenes validation set. Importantly, our Graph BEV outperforms BEV Fusion by 8.3\% under conditions with misalignment noise.

1 Introduction

Calibration errors between LiDAR and camera can misalign BEV features, while existing fusion methods overlook local depth and global feature misalignment. GraphBEV addresses both with LocalAlign and GlobalAlign and outperforms BEVFusion in clean and noisy settings.

1 Introduction

  • GraphBEV surpasses BEVFusion by 1.6% mAP on nuScenes validation and over 8.3% in noisy misalignment settings.The noisy-setting comparison uses the nuScenes dataset provided by Dong et al..Source: Figure 1 · p. 2, §1 · p. 3
  • Prior approaches use cross-attention, projection offsets, neighboring projections, or object-centric fusion, but MetaBEV aligns only during BEV fusion.The passage notes that MetaBEV overlooks depth-estimation errors in view transformation.Source: §1 · p. 2
  • BEVFusion overlooks local misalignment during LiDAR-to-camera depth projection and global misalignment during BEV fusion.Its direct concatenation of BEV representations followed by basic convolution does not address depth-inaccuracy-driven feature misalignment.Source: §1 · pp. 2–3
  • LocalAlign addresses local misalignment from imprecise projected depth, while GlobalAlign dynamically generates offsets to align LiDAR and camera BEV features.GraphBEV uses graph-based neighbor depth information and dual depth encoding in LocalAlign.Source: §1 · p. 3

2 Related Work

Related work covers LiDAR point-cloud representations, camera-based detection and view transformation, and multimodal fusion strategies. BEV methods unify LiDAR and camera representations but are commonly evaluated on clean datasets.

2.1 LiDAR-based 3D Object Detection

  • LiDAR detectors use point-based, voxel-based, or point-voxel representations, with point-voxel methods combining representations at increased computational demand.Voxel-based methods commonly use 3D sparse convolutions, while PointPillars encodes pillars with a 2D backbone.Source: §2.1 · pp. 3–4

2.2 Camera-based 3D Object Detection

  • Camera-based detection progressed from 2D detectors with 3D box-regression heads to LSS-style BEV lifting and transformer-based query methods.LSS-based approaches use LiDAR-to-camera projections for depth supervision before unifying multi-view features onto BEV.Source: §2.2 · p. 4

2.3 Multi-modal 3D Object Detection

  • Multimodal detection fuses sensor features at point, feature, or BEV level, but BEV methods are typically tested on clean datasets that overlook real-world misalignment.BEV-based methods unify LiDAR and camera representations in BEV space.Source: §2.3 · p. 4

3 Method

GraphBEV pairs LocalAlign, which addresses projection-driven local misalignment during camera-to-BEV transformation, with GlobalAlign for misalignment between LiDAR and camera BEV features.

3 Method

  • GraphBEV combines modality-specific encoders, LocalAlign for camera-to-BEV projection, GlobalAlign for cross-modal BEV alignment, and a 3D detection head.Source: §3 · p. 5

3.1 LocalAlign

  • Calibration errors can assign neighboring pixels’ depths incorrectly; LocalAlign uses KD-Tree neighbors and dual depth encoding to enrich camera features before BEV transformation.Source: §3.1 · pp. 6–7

3.2 GlobalAlign

  • GlobalAlign adds camera-BEV offset noise during training and learns offsets to spatially adjust features; inference uses learned offsets without added noise.Source: §3.2 · pp. 8–9, Figure 4 · p. 8

4 Experiments

Experiments evaluate 3D detection on nuScenes and simulate feature-misalignment scenarios.

Dataset and Metric.

Implementations.

  • The implementation builds on BEVFusion and OpenPCDet, using SECOND for LiDAR features and a Swin Transformer with FPN for camera features.Source: Implementations · p. 10
  • Training uses augmentation and CBGS resampling; inference removes test-time augmentation, and latency is measured on the same A100 workstation.Source: Implementations · p. 10

3D Object Detection.

GraphBEV improves nuScenes 3D detection over BEVFusion and retains stronger performance under misalignment, while also showing gains across map segmentation, weather, distance, and object-size conditions.

3D Object Detection.

  • 70.1% mAP and 72.9% NDS give GraphBEV the best nuScenes validation performance; it beats BEVFusion by 1.6% in mAP and 1.5% in NDS.The comparison covers nuScenes validation and test results.Source: 3D Object Detection · p. 10, Table 1 · p. 11
  • GraphBEV improves Barrier by 4.0%, Bike by 2.2%, and Pedestrian by 2.7% over BEVFusion, and exceeds ObjectFusion by 0.7% mAP and 0.3% NDS.On the nuScenes test set, it surpasses BEVFusion by 1.5% mAP and 0.7% NDS.Source: 3D Object Detection · p. 11

BEV map segmentation.

  • GraphBEV improves IoU for five BEV map classes—drivable area, pedestrian crossing, walkway, stop line, and car park—with a minor decrease for divider.Evaluation uses the same training strategy as BEVFusion and covers the [-50m, 50m] × [-50m, 50m] region around the ego car.Source: Table 2 · p. 12, BEV map segmentation · p. 12

4.3 Ablation Study

  • Training on clean nuScenes and evaluating with validation-only misalignment shows a larger GraphBEV advantage under noisy conditions and a smaller clean-to-noisy metric drop than BEVFusion.Misalignment is introduced into the validation set, not the training or testing sets.Source: §4.3 · p. 12

Roles of Different Modules in GraphBEV for Feature Alignment.

  • Adding LocalAlign, GlobalAlign, or both improves alignment performance, while either single module adds minimal latency relative to BEVFusion and remains faster than TransFusion.The combined modules perform strongly under both clean and noisy conditions.Source: Roles of Different Modules in GraphBEV… · p. 12, Table 3 · p. 13

Effect of the Hyperparameters Kgraph for Feature Misalignment.

Robustness to Weather Conditions.

  • GraphBEV outperforms BEVFusion across weather conditions, especially at night, with gains reported in sunny and adverse weather conditions.The analysis partitions nuScenes validation scenes into sunny, rainy, day, and night conditions.Source: Table 5 · p. 13, Robustness to Weather Conditions · p. 14

Robustness to Ego Distances and Object Sizes.

  • GraphBEV improves performance over BEVFusion across all ego-distance and object-size groups, with notable gains for distant and small objects.Distances are grouped as near (0–20m), middle (20–30m), and far (>30m); object sizes use three equal-proportion levels.Source: Table 6 · p. 13, Robustness to Ego Distances and Object… · p. 14

5 Conclusion

GraphBEV addresses local and global feature misalignment with separate alignment modules. It outperforms BEVFusion on nuScenes validation, especially under noisy misalignment.

5 Conclusion

  • GraphBEV significantly outperforms BEVFusion on nuScenes validation, especially in noisy misalignment settings.The conclusion gives no numeric margin.Source: §5 · p. 14
  • LocalAlign uses graph matching to incorporate neighbor-aware depth features and mitigate misalignment from inaccurately projected LiDAR depth.Source: §5 · p. 14
  • GlobalAlign simulates offset noise and uses learnable offsets to align global LiDAR and camera BEV features.Source: §5 · p. 14

Related papers

Loading 2403.11848v4…