Source-linked AI summary
RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection
Zhiwei Lin, Zhe Liu, Zhongyu Xia, Xinhao Wang, Yongtao Wang, Shengxiang Qi, Yang Dong, Nan Dong, Le Zhang, Ce Zhu
TL;DR
Camera-only 3D detection is economical but struggles with precise depth and robustness, motivating fusion with millimeter-wave radar. RCBEVDet extracts radar BEV features and fuses them with camera features, achieving state-of-the-art radar-camera results while maintaining real-time inference speed.
Problem
Multi-view cameras provide economical semantic information but lack precise depth and robustness in adverse weather or low-light conditions, motivating radar-camera fusion.
Method
RCBEVDet combines RadarBEVNet for radar BEV feature extraction with a Cross-Attention Multi-layer Fusion module for deformable radar-camera BEV alignment and fusion.
Results
RCBEVDet achieves new state-of-the-art radar-camera fusion results on nuScenes and VoD while maintaining real-time inference speed.
Takeaways & Limitations
Radar-camera fusion improves 3D detection over camera-based methods while preserving real-time operation.
Abstract
from arXiv · showhide
Three-dimensional object detection is one of the key tasks in autonomous driving. To reduce costs in practice, low-cost multi-view cameras for 3D object detection are proposed to replace the expansive LiDAR sensors. However, relying solely on cameras is difficult to achieve highly accurate and robust 3D object detection. An effective solution to this issue is combining multi-view cameras with the economical millimeter-wave radar sensor to achieve more reliable multi-modal 3D object detection. In this paper, we introduce RCBEVDet, a radar-camera fusion 3D object detection method in the bird's eye view (BEV). Specifically, we first design RadarBEVNet for radar BEV feature extraction. RadarBEVNet consists of a dual-stream radar backbone and a Radar Cross-Section (RCS) aware BEV encoder. In the dual-stream radar backbone, a point-based encoder and a transformer-based encoder are proposed to extract radar features, with an injection and extraction module to facilitate communication between the two encoders. The RCS-aware BEV encoder takes RCS as the object size prior to scattering the point feature in BEV. Besides, we present the Cross-Attention Multi-layer Fusion module to automatically align the multi-modal BEV feature from radar and camera with the deformable attention mechanism, and then fuse the feature with channel and spatial fusion layers. Experimental results show that RCBEVDet achieves new state-of-the-art radar-camera fusion results on nuScenes and view-of-delft (VoD) 3D object detection benchmarks. Furthermore, RCBEVDet achieves better 3D detection results than all real-time camera-only and radar-camera 3D object detectors with a faster inference speed at 21~28 FPS. The source code will be released at https://github.com/VDIGPKU/RCBEVDet.
1. Introduction
Camera-only 3D detection is economical and semantically rich but lacks precise depth and robustness in adverse conditions. RCBEVDet addresses these limitations by extracting radar BEV features and dynamically aligning them with camera features.
- Multi-view cameras provide detailed color, texture, and high-resolution semantic information for economical 3D object detection.
- Camera-only detection remains limited because cameras lack precise depth and may fail in adverse weather or low-light conditions.
- Millimeter-wave radar complements cameras through high-precision distance measurement, velocity estimation, and reliability across weather and lighting conditions.
- Existing radar-camera fusion methods commonly use simple feature fusion and LiDAR-oriented encoders, leaving spatial misalignment insufficiently addressed.
- RCBEVDet introduces RadarBEVNet and CAMF to extract radar features and robustly align and fuse radar-camera BEV features.RadarBEVNet combines dual-stream encoding with an RCS-aware BEV encoder, while CAMF uses deformable cross-attention.
2. Related Work
Prior work develops camera-based BEV representations and radar-camera fusion strategies, but radar remains challenging because it is sparse and semantically limited. RCBEVDet instead combines a dedicated radar BEV extractor with robust cross-modal feature alignment and fusion.
- Camera-based 3D Object Detection: Camera-based 3D detection methods infer depth, use geometry and shape priors, design specialized losses, or jointly optimize detection and reconstruction.
- Camera-based 3D Object Detection: Geometry-based methods transform multi-view image features into voxel or BEV representations, with BEVDet building detection on LSS viewpoint transformation.
- Camera-based 3D Object Detection: Transformer-based methods map perspective features to voxel or BEV representations through attention, including multi-scale deformable attention and temporal object queries.
- Radar-camera 3D Object Detection: Radar is inexpensive, long-range, and provides Doppler velocity measurements, but its sparsity and limited semantic information make radar-only detection difficult.
- Radar-camera 3D Object Detection: Radar-camera methods use multi-level, proposal-level, association-based, or radar-depth fusion strategies to combine complementary sensor information.
- Radar-camera 3D Object Detection: RCBEVDet introduces RadarBEVNet and cross-attention multi-layer fusion for efficient radar BEV extraction and robust multimodal feature alignment.
3. Method
RCBEVDet processes camera and radar inputs in BEV using RadarBEVNet and Cross-Attention Multi-layer Fusion. RadarBEVNet extracts radar features with dual streams and RCS-aware scattering, while deformable cross-attention aligns modalities before channel and spatial fusion.
- Overall pipeline: RCBEVDet transforms multi-view image features and aligned radar point clouds into BEV features, then fuses them for 3D object detection.The image branch uses view transformation, while the radar branch uses RadarBEVNet.
- RadarBEVNet: RadarBEVNet combines point-based local feature learning with transformer-based global information modeling.The dual-stream backbone uses separate point-based and transformer-based backbones.
- RadarBEVNet: The Injection and Extraction module uses cross-attention to exchange features between the point-based and transformer-based backbones.It is applied at each block of the two backbones.
- RadarBEVNet: RCS-aware scattering uses radar cross-section as an object-size prior to distribute one radar-point feature across multiple nearby BEV pixels.Features scattered to overlapping pixels are aggregated by summation, and an RCS-derived Gaussian-like weight map is concatenated with the radar BEV feature.
- Cross-Attention Multi-layer Fusion: Deformable cross-attention dynamically aligns radar and camera BEV features, reducing complexity from O(H^2W^2C) to O(HWC^2K).The method exchanges the radar and camera roles for a second deformable cross-attention update.
- Cross-Attention Multi-layer Fusion: After alignment, channel and spatial fusion concatenate the BEV features and process them with residual CBR blocks.The CBR blocks comprise convolution, batch normalization, and ReLU operations.
4. Experiments
RCBEVDet is evaluated on nuScenes and VoD using real-time radar-camera detection settings, component ablations, and sensor-failure tests. The experiments report strong benchmark performance, consistent component gains, and improved robustness under sensor failures.
- NuScenes Results: RCBEVDet surpasses previous radar-camera fusion methods with faster inference across backbone settings on nuScenes.With ResNet-50, it reduces mAVE by 14.7% versus SOLOFusion and 37.5% versus CRN.
- VoD Results: 0.34 mAP separates RCBEVDet from RCFusion across the full VoD area, while the region of interest reaches 69.80 mAP.These results are reported on the VoD validation set.
- Main Components: Each principal component consistently improves 3D detection performance when added to the BEVDepth baseline.The ablation progressively adds components to compose RCBEVDet.
- RadarBEVNet and CAMF: RadarBEVNet adds 2.1 NDS and 3.0 mAP over the baseline radar backbone, while CAMF raises performance from 55.7 NDS to 56.4 NDS.The RCS-aware BEV encoder contributes 1.4 NDS and 1.9 mAP, and the injection-extraction module contributes 0.6 NDS and 0.8 mAP.
- CAMF: Deformable cross-attention improves performance from 55.7 to 56.1 NDS and from 45.3 to 45.5 mAP.Channel and spatial fusion adds 0.3 NDS and 0.1 mAP over one-layer BEVFusion fusion.
- Robustness: RCBEVDet outperforms CRN and BEVFusion in Car mAP across every tested sensor-failure case.The experiment randomly drops image or radar inputs and uses dropout training as augmentation.
5. Conclusion
RCBEVDet combines RadarBEVNet for radar BEV feature extraction with CAMF for dynamic radar-camera feature alignment and fusion. It achieves state-of-the-art radar-camera results on nuScenes and VoD while maintaining real-time inference speed.
- Conclusion: RCBEVDet uses a dual-stream radar backbone and RCS-aware BEV encoder for efficient radar BEV feature extraction.Its CAMF module dynamically aligns radar-camera BEV features for robust 3D object detection.
- Conclusion: RCBEVDet achieves new state-of-the-art radar-camera fusion results on nuScenes and VoD 3D object detection benchmarks.The conclusion also reports significant gains over camera-based methods with real-time inference speed.