PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the View Disparity Between Radar and Camera Features for Multi-modal Fusion 3D Object Detection

Taohua Zhou, Yining Shi, Junjie Chen, Kun Jiang, Mengmeng Yang, Diange Yang

arXiv:2208.12079v2cs.CV

TL;DR

Radar and camera features have different views, complicating their fusion for 3D detection. RCBEV transforms image features into BEV and combines them with temporally encoded radar features through point-fusion and ROI-fusion. On nuScenes, it reports the highest camera-radar testing mAP and NDS scores among the compared methods.

  • Problem

    Radar and camera features occupy different views, and direct projection can lose high-dimensional sensory information needed for 3D fusion.

    Source: §I · p. 1

  • Method

    RCBEV transforms image features to BEV and uses a temporal-spatial radar encoder with point-fusion and ROI-fusion for feature-level radar-camera detection.

    Source: §I · p. 2

  • Results

    On the nuScenes testing set, RCBEV records 40.6% mAP and 48.6% NDS, the highest scores among camera-radar fusion works.

    Source: C. Overall result analysis · p. 7

  • Takeaways & Limitations

    On nuScenes, the authors report that RCBEV achieves the best detection performance among camera-radar fusion methods.

    Source: §VI · p. 11

Abstract

from arXiv · show

Environmental perception with the multi-modal fusion of radar and camera is crucial in autonomous driving to increase accuracy, completeness, and robustness. This paper focuses on utilizing millimeter-wave (MMW) radar and camera sensor fusion for 3D object detection. A novel method that realizes the feature-level fusion under the bird's-eye view (BEV) for a better feature representation is proposed. Firstly, radar points are augmented with temporal accumulation and sent to a spatial-temporal encoder for radar feature extraction. Meanwhile, multi-scale image 2D features which adapt to various spatial scales are obtained by image backbone and neck model. Then, image features are transformed to BEV with the designed view transformer. In addition, this work fuses the multi-modal features with a two-stage fusion model called point-fusion and ROI-fusion, respectively. Finally, a detection head regresses objects category and 3D locations. Experimental results demonstrate that the proposed method realizes the state-of-the-art (SOTA) performance under the most crucial detection metrics-mean average precision (mAP) and nuScenes detection score (NDS) on the challenging nuScenes dataset.

I. INTRODUCTION

Radar and cameras provide complementary information, but their different views make feature-level fusion difficult. RCBEV addresses that disparity with BEV cross-view features and a two-stage fusion design; the introduction also reviews single-sensor 3D detection approaches.

I. INTRODUCTION

  • Radar supplies position and velocity and is robust to adverse weather, while cameras provide semantic information but are sensitive to light and occlusion.Radar point clouds also lack semantic information and can contain environmental clutter.Source: §I · p. 1
  • Radar-camera fusion schemes include output fusion, raw-data fusion, and feature fusion; feature fusion uses deep networks to combine intermediate features for 2D, 2.5D, or 3D detection.The introduction notes that raw-data fusion by projecting radar points into the image plane is more suited to 2D tasks than 3D detection.Source: §I · p. 1
  • Directly projecting front-view camera features onto radar’s 3D space ignores view disparity and can lose high-dimensional sensory information; direct IPM can distort or lose image features.The authors therefore question raw-data-stage spatial-temporal synchronization as the most suitable approach.Source: §I · p. 1
  • RCBEV bridges radar-camera view disparity with BEV cross-view feature-level fusion, a temporal-spatial radar encoder, point-fusion and ROI-fusion, and an anchor-free 3D detection head.Its radar branch addresses sparsity and clutter, while radar geometry informs high-dimensional heatmaps for spatial regression.Source: §I · p. 2

A. Single-sensor 3D Object Detection

  • LiDAR-based detectors include two-stage, single-stage, and anchor-free designs, while camera-only 3D detection uses geometry priors, pseudo-LiDAR, 3D regression branches, object queries, or BEV view transformers.The paper notes that camera-only 3D detection is difficult because images lack depth; camera methods can also be affected by occlusion, truncation, and height-estimation errors.Source: A. Single-sensor 3D Object Detection · p. 2

B. Fusion-based Object Detection

Radar-camera fusion research spans early, feature, and late approaches, with radar sparsity and cross-modal representation remaining challenges. RCBEV combines transformed image features with temporally encoded radar features and fuses them in two stages in BEV.

B. Fusion-based Object Detection

  • Radar-camera fusion research includes early, feature, and late fusion, while radar sparsity, noise, and efficient cross-modal feature representation remain challenges.Prior radar-camera work also applies radar augmentation to 2D detection and uses feature fusion for 3D detection.Source: B. Fusion-based Object Detection · p. 2

III. SYSTEM ARCHITECTURE

  • RCBEV lifts multi-scale image features with depth estimates into BEV, extracts radar features, fuses modalities at point-fusion and ROI-fusion, then regresses categories and 3D boxes.Radar-derived Gaussian heatmaps represent object geometry constraints in the unified BEV space.Source: §III · pp. 2–3, §IV · p. 3

A. Radar processing

  • The radar branch comprises preprocessing, a spatial-temporal feature extractor, and an object heatmap generator for fusion-based detection.The paper identifies sparsity, clutter, and suitable feature structures for fusion as motivations for these modules.Source: A. Radar processing · p. 3
  • Radar sweeps are aligned to the current reference frame, accumulated over a time window, and filtered using point-confidence criteria to address sparsity and clutter.The preprocessing formulation includes sequential accumulation and clutter filtering; N is the multi-sweep accumulation window size.Source: A. Radar processing · pp. 3–4
  • The radar encoder uses position, ego-motion-compensated Doppler velocity, and RCS as five-dimensional inputs, then applies spatial encoding and ConvLSTM temporal refinement.The temporal encoder refines features because slight misalignments can remain between sweeps; its output is sent to the image-branch fusion process.Source: A. Radar processing · pp. 4–5

B. Image processing

RCBEV extracts multi-scale image features, transforms them from the front view into BEV, and further processes them there. Depth estimation and camera calibration support the transformation and synchronization with radar features.

B. Image processing

  • The image branch comprises image-view feature extraction, cross-view transformation, and BEV feature extraction.Fig. 3 illustrates this image data processing sequence.Source: B. Image processing · p. 5, Figure 3 · p. 5
  • Multi-scale image features are extracted by the image-view feature extractor using a backbone and neck model.The BEV feature extractor then uses ResNet and FPN layers to refine features for top-down representation.Source: B. Image processing · p. 5
  • The cross-view transformer lifts image-view features using depth estimation, then splats them into the BEV reference space.Camera extrinsics and intrinsics are used in the transformation, where image and radar features are synchronized.Source: B. Image processing · p. 5

C. Fusion module

The fusion module combines radar and image features in BEV through point-fusion followed by ROI-fusion. The second stage uses predicted and radar heatmaps to augment features for detection.

C. Fusion module

  • The two fusion stages are point-fusion and ROI-fusion, ordered by their forward-propagation sequence.The stages are designed to enable sufficient interaction between modalities.Source: C. Fusion module · p. 5
  • Point-fusion aligns image and radar feature-map resolutions, concatenates them, and applies convolution to fuse them.The image and radar features are both treated as high-dimensional point-cloud encodings; separate downsampling and upsampling address their differing fields of view and sparsity.Source: C. Fusion module · p. 5
  • The first-stage output predicts category-specific object heatmaps, while radar inputs generate six-channel Gaussian heatmaps from position, velocity, intensity, and false-alarm probability.The radar heatmaps use radar positions as means and measurement-dependent standard deviations.Source: C. Fusion module · p. 5
  • After obtaining HP F and Hradar, ROI-fusion multiplies the two heatmaps and applies a 1 × 1 convolution to obtain HRF, which has the same resolution as HP F.Source: C. Fusion module · p. 6

D. Detection head design

The anchor-free detection head jointly optimizes classification and 3D regression losses under BEV. It predicts object categories and bounding-box attributes through heatmaps and regression outputs.

D. Detection head design

  • The anchor-free head uses a shared convolution layer to calculate regression losses and obtain final outputs, with losses jointly optimized under BEV.Total training loss combines classification loss and weighted location-regression loss.Source: D. Detection head design · p. 6
  • Focal loss handles the sparse foreground and background imbalance in BEV classification, with ground truth splatted onto class heatmaps.The prediction heatmaps are produced from feature heatmaps through another convolution layer.Source: D. Detection head design · p. 6
  • Regression uses L1 losses for position offsets, dimensions, and velocities, softmax for orientation, and binary cross-entropy for attributes.Orientation is encoded using two bins and eight scalars.Source: D. Detection head design · p. 6

A. Dataset and Evaluation metrics

The evaluation uses nuScenes, a large-scale autonomous-driving dataset spanning 1,000 scenes in Boston and Singapore. Detection is assessed with mAP, true-positive error metrics, and NDS.

A. Dataset and Evaluation metrics

  • nuScenes contains 1,000 driving scenes from Boston and Singapore, including rainy and nighttime scenarios.The dataset records approximately 1.4M camera images, 1.4M radar sweeps, and 1.4M object boxes in 40k keyframes.Source: A. Dataset and Evaluation metrics · p. 6
  • nuScenes mAP measures precision and recall using ground-plane 2D center-distance matching rather than intersection over union.The matching thresholds are D = {0.5, 1, 2, 4} across ten classes.Source: A. Dataset and Evaluation metrics · pp. 6–7
  • NDS summarizes mAP and regression quality, while true-positive metrics assess translation, scale, orientation, velocity, and attribute errors.The listed true-positive metrics are ATE, ASE, AOE, AVE, and AAE.Source: A. Dataset and Evaluation metrics · p. 7

B. Implementation details

RCBEV was implemented and trained with a specified detection platform, hardware, optimizer, and sampling strategy. Its setup also specifies sensor configuration, image resolution, radar accumulation, augmentation, and voxel size.

B. Implementation details

  • RCBEV was implemented on MMDetection3D and trained for 20 epochs on four NVIDIA GeForce GTX 3090 GPUs with AdamW, an initial learning rate of 2e−4, and batch size 32.CBGS was used during training, and the vision backbone was pre-trained on nuScenes.Source: B. Implementation details · p. 7
  • Training and testing used six cameras and five radars, with images resized to 704 × 256 pixels and ten radar sweeps accumulated to increase point-cloud density.The image resizing was chosen to balance precision and computational efficiency.Source: B. Implementation details · p. 7
  • Radar augmentation used flips, rotations from −0.3925 to 0.3925 rad, and scaling from 0.95 to 1.05; voxel size was [0.1, 0.1, 0.2] meters.The voxel dimensions correspond to the x, y, and z directions.Source: B. Implementation details · p. 7

C. Overall result analysis

RCBEV is evaluated against camera-only and radar-camera methods on nuScenes, with reported gains in overall detection, per-class performance, and adverse conditions. Qualitative examples show detections across multiple views and varied scenes.

C. Overall result analysis

  • Fig. 5 presents detections in multi-image views and BEV for city-road, parking-lot, rainy, and night scenes, with box colors indicating object categories.The images show qualitative results rather than quantitative comparisons.Source: C. Overall result analysis · p. 7, Figure 5 · p. 8
  • RCBEV4d exceeds BEVDet4d by 4.3% mAP and 2.1% NDS; RCBEV records nuScenes test highs of 40.6% mAP and 48.6% NDS.On validation, RCBEV exceeds CenterFusion by 4.5% mAP and 2.9% NDS, while RCBEV4d exceeds it by 4.9% mAP and 4.4% NDS.Source: C. Overall result analysis · p. 7, Table III · p. 8

D. Evaluation of special cases

  • Compared with the camera-only method, RCBEV improves accuracy for car, truck, bus, construction vehicle, pedestrian, motorcycle, traffic cone, and barrier.The authors attribute the gains to radar data providing additional object geometry constraints.Source: D. Evaluation of special cases · p. 7, Table IV · p. 9
  • RCBEV improves mAP and NDS in sunny, rainy, daytime, and night conditions; radar's stable operation in adverse weather is reported to improve fusion-model robustness.The authors identify night and rainy weather as challenging for camera-only models due to camera-lens properties.Source: D. Evaluation of special cases · pp. 7, 9, Table V · p. 9

E. Ablation study

The ablations show that image and radar design choices each affect RCBEV’s fusion performance. They identify beneficial settings for backbone, resolution, radar information, sweep length, and radar encoding, with image size and sweep length showing trade-offs.

E. Ablation study

  • Swin-Transformer and ResNet backbones both improve in NDS and mAP after radar fusion, compared with camera-only results using the same backbone.The reported NDS and mAP increases are (7.9%, 6.9%) and (12.3%, 8.4%), respectively.Source: E. Ablation study · p. 9, Table VI · p. 10
  • At 1056×384, RCBEV reaches 50.3% NDS and 40.5% mAP, the best performance among the tested image resolutions.Beyond the ideal range, larger images make training more difficult and harm detection precision.Source: E. Ablation study · p. 9, Table VII · p. 10
  • Combining radar velocity and intensity information gives the best performance, while velocity information reduces velocity-estimation error.The variants alter the properties used alongside position information in the radar spatial encoder.Source: E. Ablation study · p. 9
  • More radar sweeps improve overall performance, especially velocity estimation, but a three-sweep window gives the highest mAP.With three sweeps, RCBEV exceeds CenterFusion by 4.7% mAP and 0.7% NDS.Source: E. Ablation study · p. 9
  • Both radar point encoder variants outperform camera-only methods, and the voxel encoder outperforms the pillar encoder.The comparisons concern different radar point encoder forms.Source: E. Ablation study · p. 9
  • Both radar- and camera-related components influence fusion performance, and suitable feature extractors and parameters can outperform single-sensor models.The ablation section compares design variants across the image and radar branches.Source: E. Ablation study · pp. 9–10

VI. CONCLUSION

RCBEV fuses radar and camera features under a top-down view to bridge their view disparity, using a radar temporal-spatial encoder and two-stage fusion. On nuScenes, the authors report improved detection over camera-only baselines and the best performance among camera-radar fusion methods; the work remains focused on dynamic-object perception.

VI. CONCLUSION

  • RCBEV bridges radar-camera view disparity by fusing features under the top-down view, with a temporal-spatial radar encoder and two-stage fusion.The encoder addresses radar sparsity and clutter, while the fusion stages support information interaction.Source: §VI · p. 10
  • On nuScenes, RCBEV improves detection over camera-only methods and achieves the best reported performance among camera-radar fusion methods.The conclusion describes this as validated on the large-scale nuScenes benchmark.Source: §VI · p. 11
  • The study covers dynamic-object perception, not static elements such as lane lines and lamp poles.The authors identify unified perception of dynamic and static elements as future work.Source: §VI · p. 11
Loading 2208.12079v2…