PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal 3D Object Detection in Autonomous Driving: a Survey

Yingjie Wang, Qiuyu Mao, Hanqi Zhu, Jiajun Deng, Yu Zhang, Jianmin Ji, Houqiang Li, Yanyong Zhang

arXiv:2106.12735v3cs.CV

TL;DR

Real-world autonomous-driving perception remains challenging, and combining camera and LiDAR data is difficult because their representations and acquisition characteristics differ. This survey synthesizes sensors, datasets, and multi-modal 3D detection methods through fusion stage, input, and granularity, and discusses open challenges and possible solutions. The review reports that fusion-based methods dominate the nuScenes leaderboard and that fusion inputs evolve toward representations that reduce information loss.

  • Problem

    Real-world autonomous-driving perception remains unsatisfactory, while effective fusion of heterogeneous sensor data lacks in-depth design guidelines.

  • Method

    The survey reviews sensors, datasets, and camera–LiDAR fusion networks, organizing methods by fusion stage, fusion input, and fusion granularity.

  • Results

    The top 8 methods on the nuScenes leaderboard are all fusion-based, while fusion methods’ inputs evolve from projected or voxelized point clouds toward raw points and richer image representations.

  • Takeaways & Limitations

    The survey provides a structured comparison of fusion designs and identifies open challenges with possible solutions for future multi-modal 3D detection research.

  • Takeaways & Limitations

    The survey notes that later fusion is easier to implement but loses access to information in raw data or earlier feature maps, making the optimal stage difficult to identify.

Abstract

from arXiv · show

In this survey, we first introduce the background of popular sensors used for self-driving, their data properties, and the corresponding object detection algorithms. Next, we discuss existing datasets that can be used for evaluating multi-modal 3D object detection algorithms. Then we present a review of multi-modal fusion based 3D detection networks, taking a close look at their fusion stage, fusion input and fusion granularity, and how these design choices evolve with time and technology. After the review, we discuss open challenges as well as possible solutions. We hope that this survey can help researchers to get familiar with the field and embark on investigations in the area of multi-modal 3D object detection.

1 Introduction

Autonomous-driving perception remains insufficient for challenging urban environments, motivating multi-modal 3D detection that combines complementary camera and LiDAR data. This survey organizes the field around fusion stage, input, and granularity while reviewing datasets, methods, and open challenges.

  • Urban deployment remains technologically challenging because real-world perception performance is still unsatisfactory.
  • 3D detection extends 2D detection by predicting objects’ three-dimensional location, shape, and rotational angles for autonomous vehicles.
  • Single sensors have complementary weaknesses: cameras lack accurate depth, while LiDAR resolution decreases at long distances.
  • For objects beyond 60m in KITTI, there are usually fewer than 10 LiDAR points but more than 400 image pixels.
  • Key challenges include sensor calibration, information loss during fusion, and consistent data augmentation across modalities.
  • The survey examines multi-modal fusion through three design choices: fusion stage, fusion input, and fusion granularity.

2 Background

This section surveys camera-, LiDAR-, and mmWave-radar-based 3D detection, emphasizing their data properties, detection approaches, and complementary strengths and limitations. These trade-offs motivate multi-modal fusion, while also making fusion design challenging because sensor streams can interfere with one another.

  • 3D Object Detection: 3D object detection predicts each object's 3D bounding box, including center coordinates, dimensions, and orientation.
  • Camera-based Detection: Camera-based methods provide dense visual information at relatively low cost, but monocular cameras lack depth and camera performance degrades under poor lighting, smoke, or fog.Stereo cameras estimate depth more accurately, but substantially increase processing complexity and cost.
  • LiDAR-based Detection: Point-voxel hybrid detectors combine point and voxel features, using their complementary representations for accurate 3D object detection.Examples include STD, PV-RCNN, and M3DETR, which combine these representations at different stages or scales.
  • LiDAR-based Detection: LiDAR provides strong 3D geometry and better adaptation to strong light, but its point clouds are sparse, its sensors are costly, and performance can fail in extremely severe weather.LiDAR detection methods include view-based, point-based, voxel-based, and point-voxel hybrid representations.
  • Other Sensors: mmWave radar is more robust to severe weather and supplies object velocity, but its points are noisier and less accurate than LiDAR points and labeled data is limited.Consequently, radar is more often fused with cameras or LiDAR than used alone.

3 Datasets and Metrics

This section surveys datasets and evaluation metrics for multi-modal 3D object detection, emphasizing sensor configurations, annotations, class coverage, and dataset scale. It also compares metric designs and observes that fusion-based methods are especially prominent on nuScenes.

  • Datasets: Large-scale datasets provide the labeled data needed for 3D detection in changing autonomous-driving environments.The survey discusses KITTI, nuScenes, Waymo, and other datasets for evaluating multi-modal detectors.
  • KITTI: KITTI provides stereo color images, LiDAR point clouds, GPS coordinates, and IMU data for tasks including 3D object detection.Its detection benchmark includes car, pedestrian, and cyclist categories and uses IoU-based true-positive matching.
  • nuScenes: nuScenes combines six cameras, a 32-beam LiDAR, and five radar sensors across 700 training, 150 validation, and 150 test scenes.It annotates 23 classes in a 360-degree field of view and evaluates detection using center-distance-based AP.
  • nuScenes: nuScenes NDS combines mAP with average translation, scale, orientation, velocity, and attribute errors for a broader performance description.These errors are denoted ATE, ASE, AOE, AVE, and AAE, respectively.
  • Waymo Open Dataset: Waymo contains 798 training and 202 validation scenes collected with five LiDARs and five pinhole cameras, covering four object categories.Its APH metric incorporates heading accuracy into AP, and the dataset supports domain adaptation.
  • Discussion: Autonomous-driving datasets range from about 15,000 to over 230,000 frames but remain smaller than major 2D image datasets, with limited and imbalanced classes.The survey also compares sensor counts, annotations, class coverage, and locations across popular multi-modal datasets.
  • Discussion: The top eight nuScenes leaderboard methods are fusion-based, compared with four of the top ten on KITTI and two of the top ten on Waymo.The survey relates this pattern to differing LiDAR resolutions and suggests multi-modal methods are more necessary for relatively sparse point clouds.

4 Deep Learning Based Multi-Modal 3D Detection Networks

The survey organizes multi-modal 3D detection networks around three fusion questions: where fusion occurs, what representations are fused, and how fusion is combined. Because fusion inputs are especially diverse, they form the basis for five network categories.

  • Design considerations: Fusion stage asks where in the detection pipeline multi-modal data are combined.The review considers input, feature-extraction, and prediction stages.
  • Design considerations: Fusion input asks what data representations enter the fusion module, while fusion granularity asks how those data are combined.These considerations distinguish representations from the level at which modalities interact.
  • Design considerations: The survey reviews possible options for all three design considerations and tracks how recent networks address them.Figure 8 summarizes the available options.
  • Fusion input: Fusion inputs exhibit the greatest diversity and reflect characteristic network designs, motivating five categories of fusion-based detectors.The survey reviews the corresponding fusion schemes in detail.

4.1 Fusion Stage: where to fuse?

Fusion stage determines where modalities interact: early, late, or cascade fusion. These choices trade intermediate-feature interaction, reuse of modality-specific networks, alignment requirements, and dependence on upstream proposals.

  • Early fusion: Early fusion combines modalities during input or feature extraction before prediction, enabling direct intermediate-feature interactions.The fused feature is then used for classification and regression, and early fusion is the most widely used stage.
  • Late fusion: Late fusion keeps modality branches separate and combines decision-level outputs during prediction.It can reuse existing modality-specific networks and avoids directly aligning different modalities.
  • Late fusion: Late fusion can outperform single-modality detectors but cannot exploit rich intermediate features.Pang et al. use geometric and semantic consistency between 2D and 3D predictions to learn probabilistic dependencies.
  • Cascade fusion: Cascade fusion feeds one branch’s prediction into another branch’s input, such as lifting 2D proposals into 3D frustums.The resulting frustum proposal defines a 3D search space for collecting points and performing 3D classification and localization.
  • Cascade fusion: Cascade methods can be limited by the accuracy of the upstream 2D detector, motivating subsequent efforts to improve proposal accuracy.F-PointNet introduced the cascading structure described here.
  • Discussion: Most fusion-based algorithms employ early fusion, which is the survey’s primary focus.The survey summarizes the advantages and disadvantages of all three fusion stages and lists representative methods for each.

4.2 Fusion Input: what to fuse?

Fusion inputs determine what representations enter the multi-modal module, ranging from raw data to intermediate features and branch outputs. For LiDAR-camera fusion, surveyed designs increasingly combine richer image semantics with less information-losing point-cloud representations and multiple inputs.

  • Fusion stage: Fusion architectures span early, late, and cascade stages, respectively combining modality features before prediction, predictions after each branch, or one branch’s prediction with another branch’s input.These stages answer the survey’s question of where to fuse.
  • Fusion input choices: Fusion inputs specify the representation provided to the fusion module, which may be raw data, intermediate features, or branch-level outputs.The survey emphasizes fusion input as a central design choice because representation diversity strongly affects detection performance.
  • Image-branch inputs: Image-side inputs commonly include feature maps, segmentation masks, and pseudo-LiDAR point clouds.Feature maps provide richer appearance cues, masks offer compact semantic summaries, and pseudo-LiDAR lifts image pixels into 3D.
  • Point-cloud inputs: Point-cloud inputs include raw point clouds, voxel grids, and projections onto bird’s-eye-view or range-view planes.Voxel grids support parallel processing but introduce information loss, empty voxels, and costly 3D convolutions.
  • Typical combinations: The most popular LiDAR-camera combinations are point-cloud BEV/RV with image feature maps, voxelized points with image feature maps, raw points with image feature maps, and raw points with image masks.The survey also reviews methods using multiple input types from the image or point-cloud modality.
  • Development trends: Point-cloud representations evolve from projections and voxel grids toward raw points, while image inputs evolve from lower-level feature maps toward higher-level segmentation results.The survey links these trends to reducing point-cloud information loss and exploiting richer image information; growing computing capability enables multiple inputs from one modality.

4.3 Fusion Granularity: how to fuse?

Fusion granularity describes how finely the image and point-cloud streams are combined. RoI-, voxel-, and point-level designs trade computational cost against correspondence precision and information exchange, with methods generally moving toward finer or combined granularities.

  • Granularity choices: Fusion granularity has three common levels: RoI-level, voxel-level, and point-level, with finer granularity usually requiring more computation but often improving performance.The choice therefore affects both fusion complexity and effectiveness.
  • RoI-level: RoI-level fusion pools features from selected object regions into equal-length vectors, typically after 3D proposal generation.It fuses selected regions rather than dense feature-map locations.
  • Voxel-level: Voxel-level fusion projects voxel centers into the image and appends corresponding image features, providing finer correspondence than RoI-level fusion.Dense image information can compensate for empty voxels caused by LiDAR sparsity.
  • Point-level: Point-level fusion projects every 3D point to image pixels and decorates each point or voxel with corresponding image features or masks.This early fusion creates dense cross-modal context and avoids the blurring associated with coarser correspondence, but occlusion can provide invalid image information and memory use is higher than voxel-level fusion.
  • Development trend: Across the surveyed timeline, fusion granularity becomes finer, and some recent methods combine multiple granularity levels for further refinement.The trend is documented in the survey’s comparison of method years and granularities.

4.4 LiDAR-Camera Fusion: summary and development

LiDAR-camera fusion has moved beyond hard point-to-pixel associations toward BEV-based designs. The survey reports that BEV fusion is more robust to degraded image quality and sensor misalignment.

  • Limitations of earlier designs: RoI-level fusion lacks deep feature interaction, whereas voxel- and point-level methods enable deeper exchange but can be affected by sensor misalignment.The latter methods establish hard point-to-pixel associations through calibration matrices.
  • BEV-level fusion: BEV-based LiDAR-camera fusion avoids hard 3D-2D association and has been shown robust against degenerated image quality and sensor misalignment.This development establishes a newer BEV-level fusion paradigm.

4.5 Fusion with Other Sensors

The survey also covers fusion involving mmWave radar and multiple same-kind sensors. Radar contributes long-range and motion-sensitive information, while LiDAR-radar methods can still degrade in adverse weather and multi-LiDAR systems increase point-cloud information.

  • mmWave radar: mmWave radar is increasingly used because of its long range, low cost, and sensitivity to motion.The survey briefly reviews both radar-camera and LiDAR-radar fusion.
  • Radar-camera fusion: Radar-camera methods project radar detections into images or use them to generate 3D proposals for joint detection and depth estimation.CenterFusion is cited as a method exploiting both radar and camera data for 3D object detection.
  • LiDAR-radar fusion: RadarNet combines early joint representation learning with decision fusion using radar radial velocity, but shows significant degradation in rare critical adverse weather conditions.Subsequent work exploits complementary radar that is less affected by adverse weather.
  • Same-kind sensor fusion: HorizonLiDAR3D fuses point clouds from five LiDAR sensors through simple concatenation to augment point-cloud information.This illustrates fusion across sensors of the same kind.

5 Open Challenges and Possible Solutions

Multi-modal 3D detection must address calibration, information loss, augmentation consistency, data scarcity, and the limitations of available sensing and datasets. The survey identifies representation- and learning-based directions for improving fusion.

  • 5.1 Open Challenge I: Multi-Sensor Calibration: Sensor calibration is required to align LiDAR point clouds and camera pixels, but vehicle motion can change extrinsic parameters and accumulate error.Targetless automatic calibration reduces manual effort; integrating sensors in a suite can limit relative displacement.
  • 5.2 Open Challenge II: Information Loss during Fusion: Fusion inevitably loses information through projection, quantization, feature blurring, and representation choices, making fusion stage, input, and granularity important design variables.Later fusion is easier to implement but cannot exploit information in raw data or earlier feature maps.
  • 5.2 Open Challenge II: Information Loss during Fusion: BEV representations compress point clouds vertically, while range views suffer scale variation; combining point clouds with voxel grids and learning temporal fusion are proposed directions.The survey suggests learning 4D spatio-temporal information across sensors and time.
  • 5.2 Open Challenge II: Information Loss during Fusion: Coarse fusion granularity can blur features, motivating learnable calibration offsets that aggregate neighboring spatial information.The proposed direction aims to maximize the effect of data fusion.
  • 5.3 Other Challenges: Multi-modal augmentation must preserve fine-grained correspondences between points and pixels, because independently augmenting modalities can create misalignment.Consistent cutting and pasting of point-cloud and image patches is proposed to prevent this problem.
  • 5.3 Other Challenges: Camera-only 3D detection lacks accurate geometry, while LiDAR-only detection has low resolution at long distances; far KITTI objects have fewer than 10 LiDAR points but over 400 image pixels.The survey also identifies dataset size, class imbalance, labeling errors, and synthetic-to-real domain gaps as bottlenecks.

6 Conclusion

The survey reviews multi-modal 3D object detection, especially camera–LiDAR fusion, by organizing methods around fusion stage, input, and granularity. It also summarizes sensors and datasets before discussing open challenges and potential solutions.

  • 6 Conclusion: The survey reviews camera–LiDAR multi-modal 3D detection networks and compares popular sensors, their properties, and single-modal detection limitations.It focuses especially on networks that fuse camera images and LiDAR point clouds.
  • 6 Conclusion: Methods are systematically categorized by fusion stage, fusion input, and fusion granularity, describing where, what, and at what level sensor streams are combined.The survey also discusses how these design considerations evolve with time and technology in the broader paper context.
  • 6 Conclusion: The paper summarizes autonomous-driving datasets and concludes with open challenges and potential solutions for multi-modal 3D object detection.
Loading 2106.12735v3…