PaperlayerView exact revision on arXiv

Source-linked AI summary

Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review

Shanliang Yao, Runwei Guan, Xiaoyu Huang, Zhuoxiao Li, Xiangyu Sha, Yong Yue, Eng Gee Lim, Hyungjoon Seo, Ka Lok Man, Xiaohui Zhu, Yutao Yue

arXiv:2304.10410v2cs.CVcs.AIcs.RO

TL;DR

Radar-camera fusion research is constrained by sparse, uncertain radar data, limited joint datasets, and imperfect radar-image association. This review organizes sensor principles, representations, datasets, fusion methodologies, challenges, and research directions for object detection and semantic segmentation, reporting improved accuracy and robustness from fusion.

  • Problem

    Radar-camera fusion research faces sparse, low-resolution, uncertain radar data, insufficient joint datasets, and poor transferability of LiDAR-based methods.

  • Method

    The review analyzes radar and camera principles, signal processing and representations, fusion datasets, and methodologies across why, what, where, when, and how to fuse.

  • Results

    Radar-camera fusion improves detection accuracy and robustness, including more than 12% improvement in overall nuScenes Detection Score (NDS) when radar depth, rotation, and velocity complement image features.

  • Takeaways & Limitations

    The review provides a guideline for radar-camera fusion research and an interactive website for retrieving and comparing datasets and fusion methods.

  • Takeaways & Limitations

    Practical deployment remains constrained by limited edge-device resources and the need for low-latency, high-performance fusion, with only one work reporting fusion output speed.

Abstract

from arXiv · show

Driven by deep learning techniques, perception technology in autonomous driving has developed rapidly in recent years, enabling vehicles to accurately detect and interpret surrounding environment for safe and efficient navigation. To achieve accurate and robust perception capabilities, autonomous vehicles are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. This review aims to provide a comprehensive guideline for radar-camera fusion, particularly concentrating on perception tasks related to object detection and semantic segmentation.Based on the principles of the radar and camera sensors, we delve into the data processing process and representations, followed by an in-depth analysis and summary of radar-camera fusion datasets. In the review of methodologies in radar-camera fusion, we address interrogative questions, including "why to fuse", "what to fuse", "where to fuse", "when to fuse", and "how to fuse", subsequently discussing various challenges and potential research directions within this domain. To ease the retrieval and comparison of datasets and fusion methods, we also provide an interactive website: https://radar-camera-fusion.github.io.

I. INTRODUCTION

Radar-camera fusion addresses limited radar-camera research coverage by combining complementary sensing capabilities for robust autonomous-driving perception. This review surveys datasets, methods, challenges, and research directions for object detection and semantic segmentation.

  • Motivation: Radar remains effective in challenging lighting and weather, while cameras provide rich semantic information and radar provides range and velocity.Their integration supports perception of outlines, colors, textures, ranges, and velocities throughout the day.
  • Challenges: Radar-camera fusion research is constrained by low-resolution, sparse, uncertain, and cluttered radar outputs and insufficient multimodal datasets.Directly adapting LiDAR-based algorithms also performs poorly because radar point clouds are substantially sparser.
  • Research gap: Existing surveys primarily emphasize LiDAR-camera fusion, while the only survey focused on radar-camera object detection omits datasets and semantic segmentation.This leaves limited consolidated coverage of the radar-camera fusion field.
  • Contributions: The review is the first survey focused on radar-camera fusion for autonomous-driving object detection and semantic segmentation.It covers the period 2019–2023 and examines why, what, where, when, and how to fuse.
  • Contributions: The review analyzes critical challenges, proposes potential research directions, and provides an interactive website for retrieving and comparing fusion datasets and methods.The website is intended to support dataset and method comparison.

II. BACKGROUND

The background introduces radar and camera fusion through sensor principles, radar representations, and core autonomous-driving perception tasks. Radar processing transforms raw signals into increasingly structured forms, each supporting different analysis needs.

  • Perception Tasks: The review uses sensor principles and representations to motivate radar-camera fusion and introduce object detection and semantic segmentation.These tasks are presented as fundamental perception functions for understanding surrounding environments.
  • Sensor Characteristics: Radar measures target range and velocity using time-of-flight and Doppler principles, while newer 4D radar additionally measures elevation.4D radar represents range, Doppler velocity, azimuth angle, and elevation angle.
  • Sensor Characteristics: Radar offers all-weather operation and motion information but has low angular resolution, sparse point clouds, and difficulty distinguishing closely located objects.A pedestrian may produce only a few points and a car roughly a dozen points.
  • Data Representations: Radar data processing converts raw ADC signals into SCA tensors, image-like radar tensors, point clouds, and micro-Doppler signatures.3D FFT produces RA, RD, AD, or RAD tensors, while peak detection yields sparse point-like representations.
  • Data Representations: Radar point clouds provide intuitive spatial visualization, whereas micro-Doppler signatures capture fine motion patterns useful for category and behavior recognition.A walking pedestrian produces periodic Doppler-frequency variation that can support gait and gesture recognition.

B. Camera Sensors

Camera sensors provide rich appearance information for object detection, while radar contributes complementary sensing under adverse conditions. Camera- and radar-based detectors use architectures adapted to image-like tensors or point-cloud representations.

  • Camera Sensors: Cameras capture colors, shapes, and textures that neural networks use to identify vehicles, pedestrians, bicycles, traffic lights, and other obstacles.Camera imaging depends on incident light and can be adversely affected by insufficient light, extreme weather, droplets, or dust.
  • Sensor Comparison: Radar and camera sensors cannot substitute for each other, so mutual integration combines complementary information and can preserve operation when one sensor fails.The review connects fusion with scenario understanding and autonomous-driving system reliability.
  • Object Detection: Camera-based object detectors include two-stage and one-stage CNN methods, with transformer-based detectors emerging through self-attention and contextual modeling.Representative transformer detectors include DETR, Deformable DETR, RT-DETR, Swin, and YOLOS.
  • Object Detection: Radar object detection applies image-based networks to radar tensors and point-based or grid-based networks to radar point clouds.Point-based approaches include PointNet and PointNet++, while grid-based methods map clouds into image planes or voxel grids.
  • Object Detection: Real-time radar-tensor processing is challenged by high dimensionality, noise, interference, and clutter.These factors complicate direct deployment of radar-processing algorithms.

2) Semantic Segmentation:

Semantic segmentation assigns predefined semantic labels to image pixels or point-cloud points. The review surveys camera- and radar-based segmentation, associated network families, and radar-camera datasets spanning detection and segmentation tasks.

  • Semantic Segmentation: Semantic segmentation labels each pixel or point with a category from a predefined label set.It applies to image-based datasets and point-based datasets.
  • Camera-based Semantic Segmentation: Camera-based segmentation supports free-space, lane, and obstacle segmentation through CNN, dilated-convolution, and transformer-based architectures.Dilated convolutions enlarge receptive fields while avoiding repeated pooling and associated resolution loss.
  • Radar-based Semantic Segmentation: Radar-based segmentation covers vehicles, pedestrians, freespace, and static objects using architectures adapted to radar tensors and point clouds.PointNet and PointNet++ are used for point-wise and grid-based radar segmentation approaches.
  • Datasets: The review categorizes datasets into 2D and 3D detection and semantic-segmentation groups and provides statistics on radar representations and dataset sizes.Table II supports retrieval and comparison of the summarized datasets.
  • Datasets: Most radar-camera datasets target object detection, while fewer support semantic segmentation; CARRADA, RadarScenes, and RADIal support both tasks.nuScenes supports detection, tracking, prediction, and localization among multiple tasks.

B. Sensing Modalities

Camera data are essentially 2D images, whereas radar data take ADC-signal, tensor, or point-cloud forms at different processing stages. These representations support varying semantic detail, intuitiveness, and object-category annotation.

  • Camera outputs are single images or videos that fundamentally provide 2D image data.
  • Radar representations comprise ADC signals, radar tensors, and point clouds, reflecting successive data-processing stages.
  • Only RaDICaL and RADIal provide raw ADC signals; RADIal additionally supplies radar tensors and point clouds.
  • Radar tensors span 2D, 3D, and 4D forms, including range-azimuth, range-Doppler, and higher-dimensional representations.
  • Point clouds are lighter and more intuitive than radar tensors, with conventional 3D and emerging 4D radar datasets represented.
  • Pedestrians, bicycles, and cars are the most common categories, while some datasets add fine-grained classes and object attributes.

D. Dataset Size

Radar-camera fusion datasets vary widely in scale, recording format, environments, and adverse-condition coverage. They include real and simulated data, with some datasets specifically targeting poor visibility and seasonal variation.

  • Dataset sizes range from 500 to 1.4 million frames, with nuScenes largest and Astyx smallest among the cited examples.
  • Several datasets contribute hundreds of thousands of frames, including CRUW, CARRADA, RADIATE, AIODrive, SeeingThroughFog, RADDet, and RADIal.
  • RadarScenes, CRUW, and CARRADA also provide videos that researchers can split into keyframes for further study.
  • Datasets cover urban streets, country roads, highways, and parking lots, while some use simulated radar or multi-sensor driving scenarios.
  • Coverage includes fog, snow, rain, night, darkness, bright light, blur, and repeated seasonal routes for studying low-visibility or localization conditions.
  • The review organizes fusion methodologies around what, where, when, and how to fuse, alongside dataset and method comparisons.

B. What to Fuse

Radar-camera fusion varies by input representation, coordinate perspective, and fusion stage. The review distinguishes front-view and bird’s-eye-view transformations and object-, data-, feature-, and hybrid-level fusion.

  • What to fuse: Camera inputs are typically RGB or infrared 2D images, while radar inputs include processed representations such as point clouds and tensors.
  • Where to fuse: Front-view fusion projects radar point clouds or tensors onto the image plane to generate proposals or feature maps.
  • Where to fuse: Front-view projection can suffer radar azimuth-resolution and camera-calibration errors, causing radar points to deviate from objects and complicating matching.
  • Where to fuse: Bird’s-eye-view fusion converts radar data or camera images into BEV coordinates, but grid discretization can lose information needed for bounding-box refinement.
  • When to fuse: Fusion stages are categorized as object-level, data-level, feature-level, and hybrid-level, according to when modalities combine in the network.
  • When to fuse: Object-level fusion matches independently detected radar and camera objects using similarity measures, transformations, or matching algorithms.
  • When to fuse: Object-level fusion is flexible and modular but depends heavily on individual-module outputs and discards rich intermediate features.

2) Data-level Fusion:

Data-, feature-, and hybrid-level methods combine radar and camera information at different network stages. Their trade-offs involve representation completeness, alignment sensitivity, camera reliability, and preservation of multi-stage information.

  • Data-level Fusion: Data-level fusion combines raw or preprocessed radar and camera data early to learn a joint representation from their complete characteristics.
  • Data-level Fusion: Data-level methods are sensitive to temporal and spatial misalignment, require precise external calibration, and complicate radar-to-image matching.
  • Feature-level Fusion: Feature-level fusion combines modality-specific features extracted by separate branches, using operations such as concatenation, addition, multiplication, and attention.
  • Feature-level Fusion: Feature extractors can be designed for each modality and networks can learn complementary cross-modal features, but this does not address unreliable cameras.
  • Hybrid-level Fusion: Hybrid-level fusion combines proposals and features across stages, including radar proposal refinement and attention-based feature fusion.
  • Hybrid-level Fusion: Hybrid-level fusion can yield more accurate proposals and preserve information across data levels when modalities contribute unequally.
  • How to fuse: Fusion design also requires temporal and spatial alignment and comparison of addition, mean, multiplication, concatenation, and attention operations.

1) Data Alignment:

Radar-camera alignment addresses temporal synchronization and spatial calibration before fusion, while fusion operations and network structures determine how aligned modalities are combined.

  • Temporal Alignment: Temporal alignment synchronizes sensor sequences despite measurement latency, clock drift, and setup differences.Methods estimate latency and drift periodically or predict future latency using Kalman filters or Bayesian estimation.
  • Temporal Alignment: Temporal-offset calibration can align corresponding radar-camera observations to estimate timestamp differences.Examples align frames in which a vehicle crosses a detection line or use real-time preprocessing.
  • Spatial Alignment: Spatial alignment maps 3D or 2D radar point clouds to camera pixels through radar-camera transformations.Joint calibration strategies differ according to whether they require a specially designed target.
  • Spatial Alignment: Target-based calibration uses known radar and camera target locations to estimate rigid transformations, commonly using triangular corner reflectors for radar.The reflector provides positional information through specific radar cross-section values.
  • Spatial Alignment: Target-less calibration improves portability by avoiding checkerboards but remains sensitive to environmental uncertainty when matching cross-sensor features.Approaches use radar velocity and camera pose or machine learning to predict calibration parameters.
  • Fusion Architecture: Dual-branch fusion networks separately process radar and camera inputs before fusing raw data, feature maps, or final objects.The review classifies networks as point-based or tensor-based according to radar representation.

1) Point-based Networks:

Point-based networks process radar point clouds through projection, pseudo-image, voxel, or BEV representations, while tensor-based methods preserve radar tensors and use cross-modal supervision or geometric projection.

  • Point-based Networks: Point-based networks take radar point clouds as input and divide processing into projection-based, pseudo-image-based, voxel-based, and BEV-based methods.These categories differ in how radar point clouds are represented before fusion.
  • Projection-based Methods: Projection-based methods map radar points onto the image plane and use radar images or additional branches with standard deep networks.Reported approaches use range and range-rate images, multi-stage fusion, and network adaptation.
  • Pseudo-Image-based Methods: Pseudo-image methods transform radar point clouds into image-like inputs and apply spatial attention, entropy-based fusion, or frustum association.These techniques are used to improve detection accuracy and robustness.
  • Voxel-based Methods: Voxel-based methods directly extract features from 3D radar points using voxelization, sparse convolution, and occupancy grids.The review reports improved vehicle detection, especially at longer distances and when camera data is limited.
  • BEV-based Methods: BEV-based methods combine top-down representations with transformers, proposal refinement, cross-modal interaction, semantic alignment, and attention.These methods are reported to achieve high performance in 3D radar-camera fusion tasks.
  • Tensor-based Networks: Tensor-based fusion addresses information loss after CFAR detection by combining radar tensors with camera images.The review divides these methods into cross-supervised-based and projection-based approaches.
  • Cross-Supervised-based Methods: Cross-supervised methods use camera information to generate radar labels, supporting radar object detection and segmentation.RODNet cross-supervises 3D localization during training, while related methods report performance improvements.
  • Projection-based Methods: Projection-based tensor fusion uses coordinate conversion, homography, and ray-constrained cross-attention to integrate radar and camera features.CramNet estimates depth along pixel rays and refines camera-pixel 3D locations.

G. Model Evaluations

Radar-camera fusion evaluations use conventional detection and segmentation metrics alongside nuScenes error metrics and NDS. Reported results show substantial gains across nuScenes and VoD, with BEV, transformer, and attention designs prominent among high-performing methods.

  • Evaluation Metrics: Common evaluation metrics include precision, recall, AP, AR, mAP, and mIoU, but they measure prediction accuracy on a given test dataset.
  • Evaluation Metrics: nuScenes adds mATE, mASE, mAOE, mAVE, and mAAE, while NDS combines mAP with translation, scale, orientation, velocity, and attribute errors.
  • nuScenes Results: 32.6% mAP and 44.9% NDS were achieved by CenterFusion on the complete nuScenes test set.Compared with CenterNet, CenterFusion delivered relative increases of 38.1% in NDS and 62.1% in velocity error metrics.
  • nuScenes Results: 40.6% mAP and 48.6% NDS were reported for RCBEV, while CRAFT reached 41.1% mAP and 52.3% NDS on nuScenes.CRAFT attributes most performance gain to improved localization and velocity estimation from its spatio-contextual fusion transformer.
  • nuScenes Results: 57.5% mAP and 62.4% NDS made CRN the top-performing radar-camera detector on nuScenes in the review.Its Radar-assisted View Transformation uses sparse radar points to address missing spatial information in image features.
  • 4D Radar Results: 56.31% mAP on VoD and 36.32% mAP on TJ4DRadSet made LXL the leading 4D radar-camera method on those datasets.
  • Overall Progress: Across datasets, mAP improved by 24.9% on complete nuScenes and 18.31% on VoD, while NDS increased by 17.5% on complete nuScenes.The review identifies transformers, attention mechanisms, and BEV features as crucial contributing factors.
  • Semantic Segmentation: Radar can improve semantic-segmentation accuracy and reliability by providing complementary depth information and 3D object localization.Radar localization can validate the 2D output produced by semantic segmentation algorithms.

V. CHALLENGES AND RESEARCH DIRECTIONS

Radar-camera fusion must balance heterogeneous modalities while addressing radar sparsity, inaccurate localization, noise, and annotation needs. The review highlights denser 4D radar, uncertainty modeling, and learned denoising as research directions.

  • General Challenges: Balancing modality performance is a central challenge because fusion quality can affect downstream localization, prediction, planning, and control.
  • Data Quality: Radar point clouds are disordered relative to structured image information, creating data-quality challenges categorized as sparsity, inaccuracy, and noise.
  • Sparsity: Combining radar frames from 0.25 seconds to 1 second densifies point clouds but introduces system delays.4D radar is highlighted as a direction because it can produce denser point clouds and preserve elevation information.
  • Sparsity: 31.9% to 38.0% mAP improvement was reported when additional elevation data from 4D radar increased object-detection performance on VoD.
  • Inaccuracy: Radar points may lie at object corners or outside objects rather than at their centers, limiting direct geometric correspondence.Proposed responses include marking points as circles and generating anchors with varied sizes and aspect ratios.
  • Inaccuracy: Column and pillar expansion improves vertical association, but fixed expansion sizes may need to vary across object types.One cited pillar setting is [0.2, 0.2, 1.5] meters along [x, y, z].
  • Noise: Radar returns can contain ghost objects, ground detections, and multi-radar interference that generate fake targets and limit detection or segmentation accuracy.
  • Noise: CFAR and peak-detection denoising methods have poor generalization, motivating deep-learning approaches for radar tensor and point-cloud noise mitigation.High-quality annotations and modeling radar uncertainty remain necessary for effective noise removal.

2) Data Diversity:

Radar-camera fusion is constrained by limited and incomplete data diversity, calibration and labeling burdens, and radar-specific sparsity. The review highlights richer representations, attention, graph methods, neural processing, and data-efficiency techniques as responses.

  • CRUW contains only 400k frames and 260k objects, illustrating the small scale of radar-camera datasets compared with ImageNet.
  • Complex weather, road conditions, and 360-degree sensing require multiple sensors, making comprehensive multimodal dataset collection time-consuming and labor-intensive.
  • Simulated datasets support varied sensors and dangerous scenarios, but cannot completely replace real-world data; integrating both remains an open research area.
  • Calibration remains difficult because radar returns are inaccurate and vulnerable, while vehicle vibration and sampling differences create spatial and temporal inconsistencies.
  • Camera-aided radar labeling can reduce annotation effort, but radar targets may not align with image ground truth and noisy points remain challenging to filter.
  • Using about 40% of the training data achieved the same classification results as complete supervision in reported active-learning experiments.
  • Attention and graph neural networks aggregate global or relational information to address sparse radar features and improve object proposals or parameter regression.
  • Neural processing of radar information instead of traditional FFT operations may reduce computational requirements and simplify embedded data flow.

2) Data Association:

Radar-camera data association is difficult because heterogeneous, sparse, inaccurate radar returns do not align reliably with image objects. The review discusses learned, geometric, polar, and confidence-based association strategies, alongside multimodal training and augmentation challenges.

  • 2) Data Association:: Directly projecting sparse and noisy radar data onto image planes often produces poor object-level or data-level associations.
  • 2) Data Association:: Radar Proposal Refinement, CenterFusion frustum association, and AssociationNet learn or refine correspondences between radar detections and camera proposals.
  • 2) Data Association:: RC-PDA filters occluded returns with confidence measures, while Soft Polar Association matches radar points around image proposals in polar coordinates.
  • 3) Data Augmentation:: Radar-only augmentation methods include rotation, scaling, flipping, shifting, translating, interpolating, and mixing.
  • 3) Data Augmentation:: Joint radar-camera augmentation is necessary because independently modifying aligned modalities can destroy physical radar features and cause incorrect inferences.
  • 4) Multimodal Training:: Multimodal networks may overfit or ignore one branch when training hyper-parameters favor another modality.
  • 4) Multimodal Training:: Per-modality losses, weighted loss blending, and pretrained unimodal encoders are proposed to balance multimodal optimization.

5) Model Robustness:

Radar-camera fusion robustness remains limited by sensor degradation, unseen scenarios, inconsistent evaluation, interpretability gaps, edge-device constraints, and the preliminary state of multitask learning. The review identifies uncertainty modeling, generative models, standardized evaluation, acceleration, and multitask learning as directions.

  • 5) Model Robustness:: Most reviewed methods prioritize public-dataset accuracy, while few address sensor failure when only one modality remains available.
  • 5) Model Robustness:: Bayesian neural networks are suggested for estimating higher uncertainty on unseen objects and driving scenarios.
  • 5) Model Robustness:: Generative models may detect sensor defects or novel scenarios while preserving fundamental radar effects and real-time capability.
  • 6) Evaluation:: Different nuScenes subsets, sensor selections, metrics, and thresholds prevent direct comparisons across methods.
  • 6) Evaluation:: PDQ and open-set metrics are proposed as alternatives for evaluating uncertainty-related robustness and unknown objects.
  • 6) Evaluation:: Visual analytics for interpreting radar-camera fusion networks remains an open and challenging question.
  • 7) Model Deployment:: Only one cited work reports fusion speed on an edge device, reaching 11 Hz on an NVIDIA Jetson AGX TX2.
  • 8) Multitask Learning:: Multitask learning can share knowledge and reduce parameters and computations, but radar-camera multitask research remains preliminary.
Loading 2304.10410v2…