PaperlayerView exact revision on arXiv

Source-linked AI summary

MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review

Zhiqing Wei, Fengkai Zhang, Shuo Chang, Yangyang Liu, Huici Wu, Zhiyong Feng

arXiv:2108.03004v3cs.CV

TL;DR

Accurate object detection in complex autonomous-driving scenarios motivates combining complementary radar and vision sensing. This survey organizes radar–vision fusion by pipeline stages and fusion levels, reviews related technologies, and identifies 3D detection and broader multimodal fusion as future directions.

  • Problem

    Vision can degrade under occlusion, extreme weather, and untrained distortions, while mmWave radar lacks object outlines and struggles with relatively stationary targets.

  • Method

    The survey reviews sensor selection, calibration, and fusion, classifying fusion methods into data-level, decision-level, and feature-level approaches.

  • Results

    The review identifies radar–vision fusion as a mainstream obstacle-detection solution and discusses 3D detection, lidar–vision fusion, and multimodal fusion as promising technologies.

  • Takeaways & Limitations

    Radar–vision fusion uses complementary sensor capabilities to improve autonomous-vehicle perception, while future work emphasizes more accurate 3D detection and additional sensing information.

Abstract

from arXiv · show

With autonomous driving developing in a booming stage, accurate object detection in complex scenarios attract wide attention to ensure the safety of autonomous driving. Millimeter wave (mmWave) radar and vision fusion is a mainstream solution for accurate obstacle detection. This article presents a detailed survey on mmWave radar and vision fusion based obstacle detection methods. First, we introduce the tasks, evaluation criteria, and datasets of object detection for autonomous driving. The process of mmWave radar and vision fusion is then divided into three parts: sensor deployment, sensor calibration, and sensor fusion, which are reviewed comprehensively. Specifically, we classify the fusion methods into data level, decision level, and feature level fusion methods. In addition, we introduce three-dimensional(3D) object detection, the fusion of lidar and vision in autonomous driving and multimodal information fusion, which are promising for the future. Finally, we summarize this article.

GLOSSARY

Autonomous driving requires accurate object detection in complex scenarios, while vision and mmWave radar each have limitations that motivate their fusion. The fusion process comprises sensor selection, calibration, and information fusion.

  • Accurate object detection is a major challenge for high-level autonomous vehicles because obstacle detection is central to anti-collision safety.
  • Object detection research includes traditional handcrafted-feature methods and deep-learning detectors categorized as one-stage or two-stage algorithms.Examples include HOG and DPM, and deep-learning methods such as YOLO, SSD, RetinaNet, and R-CNN variants.
  • Vision-based detection is challenged by varied object scales, occlusion, blurred appearance, extreme weather, and poor generalization to untrained distortions.
  • mmWave radar is less affected by extreme weather and measures distance and velocity vectors, but it lacks object outlines and struggles with relatively stationary targets.
  • The fusion pipeline consists of sensor selection, sensor calibration, and sensor fusion to combine complementary sensing information.Calibration includes coordinate transformation, invalid-information filtering, and error calibration.

Error calibration

The article surveys autonomous-driving object detection and sensor-fusion topics in an organized progression. It moves from tasks, criteria, and datasets through sensor configuration, calibration, fusion, future trends, and conclusions.

  • Earlier surveys primarily addressed visual object detection and did not cover radar–vision fusion for vehicle detection.
  • The article introduces object-detection tasks, evaluation criteria, and public datasets before reviewing sensor configuration, calibration technologies, and fusion schemes.
  • The organization includes vehicle detection based on sensor fusion, with subsections on sensing information processing, object detection frameworks, and radar–vision feature fusion.

A. Tasks

Autonomous-driving object detection covers 2D and 3D localization, uses precision–recall-based evaluation, and relies on representative datasets with different sensing modalities and annotations.

  • 2D object detection uses image bounding boxes to classify and position obstacles within the image.
  • 3D object detection additionally determines objects’ posture and position in the real world using 3D bounding boxes.
  • AP and AR evaluate detection accuracy and regression, while mAP averages AP across object classes.The PR curve plots recall horizontally and precision vertically.
  • On KITTI, 2D localization is evaluated using IoU thresholds and mAP computed from 11 equally spaced recall thresholds.
  • KITTI requires predicted 3D vehicle boxes to overlap real boxes by more than 70%, compared with 50% for pedestrians and bicycles.
  • Representative autonomous-driving datasets provide differing combinations of images, lidar, point clouds, semantic annotations, and 3D bounding boxes.ApolloScape provides pixel-level semantic annotations, while KITTI provides raw images and accurate labeled 3D bounding boxes.

3) Cityscapes:

The review presents major autonomous-driving datasets and sensor-deployment practices, emphasizing radar–camera fusion as the mainstream obstacle-detection configuration.

  • 3) Cityscapes:: Cityscapes provides 5,000 high-quality pixel-level annotated urban driving images across 19 categories, plus 20,000 rough-annotated images.Its splits include 2,975 training, 500 validation, and 1,525 test images from over 50 cities.
  • 4) Waymo Open Dataset:: Waymo contains multimodal data from more than 10 million autonomous-driving miles across 25 cities.It includes lidar point clouds, vision images, over 12 million 3D annotations, and 1.2 million 2D annotations.
  • 5) nuScenes:: nuScenes includes camera, lidar, and radar data with 23 object classes and eight attributes.The review identifies it as the only dataset among those discussed containing radar data.
  • A. Sensor Configuration: Most surveyed manufacturers combine radar and cameras, while some additionally fuse lidar, making radar–vision fusion a mainstream sensing solution.The review attributes this adoption to complementary sensor characteristics.

1) mmWave Radar and Lidar:

MmWave radar and lidar offer different strengths: radar supports long-range, velocity-aware, all-weather sensing, whereas lidar provides denser and more accurate spatial and semantic measurements.

  • 1) mmWave Radar and Lidar:: MmWave radar detects obstacles within 250 m, compared with lidar’s detection range within 150 m.The review identifies radar’s longer range as important for autonomous-driving security.
  • 1) mmWave Radar and Lidar:: MmWave radar measures target relative velocity through the Doppler effect with 0.1m/s resolution.This velocity information is described as critical for vehicle decision-making.
  • 1) mmWave Radar and Lidar:: Lidar offers higher angle resolution and detection accuracy than mmWave radar, whose measurements are sparser and lack semantic information.Lidar measurements support 3D environmental modeling and target position and velocity prediction.
  • 1) mmWave Radar and Lidar:: Lidar’s cost and sensitivity to heavy fog, rain, and snow constrain its large-scale deployment.Bad weather increases laser attenuation and reduces propagation distance and performance.
  • 1) mmWave Radar and Lidar:: Radar and cameras have complementary capabilities: radar estimates distance and radial speed, while cameras provide color and classification but estimate speed and distance less effectively.The review presents their fusion as a way to improve perception accuracy and object-detection capability.

IV. SENSOR CALIBRATION

Sensor calibration aligns radar detections with image objects despite differences in sensor location and sampling, using coordinate, cross-sensor verification, or vision-based methods.

  • IV. SENSOR CALIBRATION: Calibration is necessary because different sensors may produce mismatched information for the same target.The mismatch arises from differences in spatial location and sampling frequency.
  • IV. SENSOR CALIBRATION: Coordinate calibration matches radar points to objects in images through coordinate transformation, sensor verification, or vision-based methods.These categories organize the reviewed calibration approaches.
  • Coordinate transformation method: Coordinate transformation unifies radar and vision information under one coordinate system through matrix operations.One reviewed approach also uses thread synchronization to address time inconsistency from different sampling rates.
  • Sensor verification method: Sensor verification generates or searches for targets with one sensor and validates them using information from the other sensor.The reviewed methods first use radar target lists or transformed radar coordinates, then compare against image information.
  • Vision based method: Vision-based calibration detects image targets or candidate regions and checks whether radar points correspond to them.Examples use motion stereo or adaptive background subtraction.
  • IV. SENSOR CALIBRATION: Open-source datasets generally remove the need for error calibration, whereas self-made datasets require radar filtering and error correction.Filtering removes noise or invalid points; correction methods address calibration and sensor-measurement errors.

V. VEHICLE DETECTION BASED ON SENSOR FUSION

The review divides radar–vision fusion into data-, decision-, and feature-level approaches, with data-level methods using radar-guided image regions to reduce search cost but risking missed objects.

  • V. VEHICLE DETECTION BASED ON SENSOR FUSION: Data-level fusion combines raw radar and camera data, decision-level fusion combines detection results, and feature-level fusion combines extracted radar and image features.The review compares these three fusion levels in Table IV.
  • A. Data Level Fusion: Data-level fusion generates image regions of interest from radar points, extracts corresponding image regions, then applies feature extraction and classification.This approach was mature but was not the current mainstream research trend.
  • A. Data Level Fusion: Data-level fusion narrows the object-detection search space and saves computational resources, but missing radar points can cause image regions to be ignored.The review identifies this as a hidden security danger.
  • A. Data Level Fusion: Radar-generated regions of interest can improve processing efficiency by avoiding sliding-window and multi-scale image searches.The initial region size can be determined by obstacle distance from the mmWave radar.
  • A. Data Level Fusion: Reviewed data-level pipelines preprocess images, extract features, and classify objects using methods including contrast normalization, symmetry, shadows, gradient information, Adaboost, and SVM.Radar points may serve as reference centers for image segmentation and boundary determination.
  • A. Data Level Fusion: Object classification methods in the reviewed systems include Adaboost, SVM, and combinations of ROI imagery with Doppler-spectrum information.These methods select or recognize final vehicle boxes in the image.

B. Decision Level Fusion

Decision-level fusion combines radar and vision detections to exploit radar’s longitudinal distance and vision’s horizontal field of view. The reviewed systems process radar-derived object lists alongside image-based detections, including CNN methods such as YOLO V2 and YOLO V3.

  • Decision level fusion: Decision-level fusion combines radar and vision detections to use their complementary sensing strengths.Radar contributes longitudinal distance, while vision contributes a wider horizontal field of view.
  • Sensing information processing: Radar processing generates object lists containing target speed and distance, while vision processing applies feature-based or CNN object detectors to images.
  • Radar information: Radar systems can estimate target distance using FFT and angular position using digital wavefront reconstruction and beamforming.
  • Image object detection: Vision-based systems reviewed include gradient-histogram and MeanShift methods for pedestrian or depth-image detection.
  • Image object detection: YOLO V2 was reported as faster and more accurate than Faster R-CNN and SSD for vehicle detection, while YOLO V3 was used for obstacle detection.

2) Decision Fusion:

Decision fusion combines sensor-level detection results through probabilistic, filtering, evidential, or radar-validation approaches, whereas feature fusion adds radar branches to CNN detectors. The reviewed CNN frameworks differ in architecture, feature representation, and fusion placement.

  • Decision Fusion: Decision fusion combines detection results using Bayesian theory, Kalman filtering, Dempster-Shafer theory, radar validation, or motion stereo refinement.
  • Decision Fusion: Bayesian approaches model multisensor fusion probabilistically and can modularize fusion algorithms for improved reusability when sensors are added.
  • Decision Fusion: Kalman-based methods support fusion in 2D image and 3D spaces while accounting for sensing results with different qualities.
  • Decision Fusion: Dempster-Shafer fusion treats sensor detection lists as evidence grids and identifies targets after sequential evidence fusion and clustering.
  • Decision Fusion: Radar validation can mark vision detections as strong or weak hypotheses based on target-list matches without discarding unmatched detections.
  • Decision Fusion: Feature-level CNN systems commonly add radar input branches, transform radar data into image-like representations, and fuse radar and vision features within detectors.

2) Radar Feature Extraction:

Radar feature extraction converts sparse radar measurements into image-like matrices so they can be integrated with vision features. Representations include multichannel radar images, sparse radar images, and camera-like images generated from radar data.

  • Radar Feature Extraction: Radar feature extraction transforms radar information into image-like matrix information because radar cannot be fused directly with image information.
  • Radar Feature Extraction: Projected radar images use multiple channels to encode physical quantities such as distance, longitudinal speed, and lateral speed.
  • Radar Feature Extraction: CMGGAN generates artificial camera-like environmental images from measured radar data, while radar sparse images use a 416 × 416 three-channel format aligned with vision images.

3) Feature Fusion:

Feature fusion combines radar and vision representations inside detection networks, commonly through concatenation, element-wise operations, or spatial attention. The review identifies sparse radar information, limited performance gains, and the need for stronger fusion and 3D detection as continuing challenges and directions.

  • Feature Fusion: Basic feature fusion uses concatenation or element-wise addition to combine radar and image feature matrices.Concatenation creates a multichannel matrix, whereas addition combines the matrices element by element.
  • Feature Fusion: SAF produced the best performance among SAF, element-wise addition, multiplication, and concatenation, and also improved Faster R-CNN generalization.
  • Challenges: Most reviewed results concern 2D detection, while 3D detection remains less accurate and is a major autonomous-driving object-detection challenge.
  • Challenges: MmWave radar features remain sparse, limiting information relative to vision and motivating more efficient fusion methods and stronger integration of mutual information.
  • Future Trends: Future directions include 3D object detection, adding sensors such as lidar, and multimodal information fusion.

2) Lidar in Autonomous Driving:

The review presents lidar–vision fusion as a promising autonomous-driving direction alongside mmWave radar–vision fusion, covering detection, classification, and road detection. It also frames radar–vision fusion as multimodal information fusion, while noting that post-processed radar data may omit information available from original detections.

  • Lidar in Autonomous Driving:: Lidar–vision fusion is promising because lidar and mmWave radar have distinct advantages, while lidar and vision complement each other.
  • Object Detection: Lidar–vision object-detection methods can generate image proposals from lidar point clouds, refine bounding boxes, and then perform detection through shared proposal and detection networks.
  • Object Classification: Lidar–vision classification can upsample lidar information into depth data and fuse it with images for Deep-CNN training.
  • Road Detection: Lidar–vision road detection projects point clouds onto images, creates spatially encoded dense images, and trains fully convolutional networks.
  • Road Detection: 96.03% MaxF was achieved on KITTI for the reviewed lidar–vision road-detection method.
  • Multimodal Information Fusion: Radar–vision fusion can be viewed as multimodal information fusion, potentially enabling richer sensing information when original radar detections are fused with images.
  • Multimodal Information Fusion: A central multimodal-fusion challenge is combining modality-specific information and noise while mining relations that support understanding of the same object or scene.
Loading 2108.03004v3…