PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

Yaodong Cui, Ren Chen, Wenbo Chu, Long Chen, Daxin Tian, Ying Li, Dongpu Cao

arXiv:2004.05224v2cs.CVcs.LGcs.RO

TL;DR

Deep-learning camera-LiDAR fusion lacks a focused critical review despite its relevance to autonomous-driving perception. This paper surveys fusion methods across major perception tasks, organizes them by fusion methodology, compares public-dataset results, and identifies overlooked deployment challenges and research directions.

  • Problem

    Deep-learning-based camera-LiDAR fusion had not received a focused critical review, despite the need for robust autonomous-driving perception.

  • Method

    The paper reviews image and point-cloud processing and camera-LiDAR fusion across depth completion, detection, segmentation, tracking, calibration, and fusion levels.

  • Results

    The survey provides methodology-based reviews, publicly available dataset comparisons, model advantages and disadvantages, and summaries of trends and open challenges.

  • Takeaways & Limitations

    The paper highlights open-set detection, sensor-agnostic frameworks, reliability, and all-weather operation as important directions for real-world autonomous-driving deployment.

Abstract

from arXiv · show

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this paper devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.

I. INTRODUCTION

Camera-LiDAR fusion addresses the need for accurate real-time 3D environmental understanding by combining complementary image and point-cloud information. The survey focuses specifically on deep-learning-based fusion methods, organizing prior work by methodology and reviewing multiple perception tasks and open deployment challenges.

  • Autonomous-driving applications remain constrained by the complexity of dynamic environments, sensor cost, and the need for accurate real-time 3D understanding.
  • Sensor fusion combines complementary sensor characteristics to enhance perception and reduce cost.
  • Cameras provide low-cost visual perception, whereas LiDAR supplies 3D geometry that monocular cameras cannot reliably provide.
  • Images are regular, ordered, and discrete, while point clouds are irregular, orderless, and continuous, requiring different feature-extraction methodologies.
  • The survey is the first reported review focused on deep-learning-based image and point-cloud fusion for autonomous driving across six perception topics.
  • It organizes methods by fusion methodology, compares state-of-the-art approaches from 2014–2020, and highlights open questions including open-set detection and sensor-agnostic frameworks.

B. Deep Learning on Point Cloud

Point-cloud deep learning must handle sparse, irregular, orderless 3D data while preserving geometric structure. The reviewed approaches use volumetric, tree, multi-view, graph, and direct point representations, with point-wise convolutions explicitly modeling spatial relationships.

  • Point clouds encode 3D structure and per-point features but are sparse, irregular, orderless, and continuous, challenging conventional deep-learning models.
  • Volumetric representations enable standard 3D convolutions but lose spatial resolution and fine-grained geometry during voxelization.
  • Tree-based representations adapt resolution to point density, reducing unnecessary computation and memory for low-density regions.
  • Multi-view methods project point clouds into 2D planes for standard convolutions, while graph methods perform convolution-like operations over neighboring points or graph spectra.
  • Direct point representations avoid intermediate transformations, using shared MLPs or specialized point convolutions to process point clouds.
  • PointNet aligns clouds, extracts permutation-invariant point features with shared MLPs and symmetric aggregation, and scales linearly with input size.
  • Point-wise convolutions explicitly model spatial relationships by learning continuous weight functions over neighboring points and their features.

III. DEPTH COMPLETION

Depth completion transforms sparse, irregular LiDAR depth into dense depth maps, often using high-resolution images to guide up-sampling. The review covers supervised and self-supervised formulations, model timelines, and KITTI benchmark comparisons.

  • Depth completion upsamples sparse, irregular depth into dense, regular depth to support downstream perception.
  • The task is formulated as minimizing a loss between the network prediction f(x; w) and ground truth G over network parameters w.
  • The review presents a depth-completion timeline and comparative results on the KITTI depth completion benchmark.
  • High-resolution RGB images provide reference information for depth up-sampling because dense color information contains relevant 3D geometry.
  • Supervised RGB-D autoencoders require pixel-level depth ground truth, while self-supervised models can train from image sequences and sparse depth using sparse-depth, photometric, and smoothness constraints.
  • CSPN uses image-dependent affinity matrices for real-time sharp depth completion, while CSPN++ reduces computation through dynamic kernels and iteration selection.

2) Feature-level fusion:

Feature-level camera-LiDAR fusion combines image and sparse-depth processing within learned network representations. Reviewed approaches span parallel encoders, shared decoders, multistage guidance, confidence-aware fusion, and stereo-LiDAR schemes, while dataset scale remains a constraint.

  • 2) Feature-level fusion:: Parallel image and sparse-depth encoders can fuse their representations in a shared decoder for depth completion.Jaritz et al. use NASNet-based encoders, while unguided depth completion processes both modalities in parallel.
  • 2) Feature-level fusion:: Multistage fusion injects image features into sparse-depth features at varying network depths to guide depth up-sampling.GuideNet achieves top performance on the KITTI depth completion benchmark among the reviewed approaches.
  • 2) Feature-level fusion:: Confidence-aware global and local branches combine RGB-D and depth data, using confidence maps to guide fusion.Van Gansbeke et al. combine signal-level and feature-level fusion in an image-guided depth completion network.
  • 2) Feature-level fusion:: Stereo-LiDAR fusion combines dense stereo geometry with sparse, accurate LiDAR depth to produce more accurate dense depth.Stereo cameras remain less ideal in limited-range, high-occlusion, and texture-less environments.
  • 2) Feature-level fusion:: A two-stage CNN can first fuse LiDAR with stereo disparity, then combine the fused disparity with RGB features for final prediction.Park et al. reconstruct the 3D scene from the resulting high-precision disparity, but large-scale annotated stereo-LiDAR datasets are a bottleneck.
  • 2) Feature-level fusion:: Self-supervised and unsupervised training schemes address the shortage of dense stereo-LiDAR annotations through photometric, smoothness, sparse-depth, and geometric losses.LidarStereoNet additionally uses a feedback loop for robustness to noisy point clouds and sensor misalignment.

A. 2D Proposal Based Sequential Models

2D proposal-based sequential models use image detections to seed 3D object detection, while later methods refine proposals or fuse image and point-cloud features. Alternative 3D proposal pipelines use BEV or voxelized representations to reduce search complexity, each with distinct information trade-offs.

  • A. 2D Proposal Based Sequential Models: F-PointNets project 2D image boxes into 3D frustums and process the resulting proposals with a PointNet-based 3D detector.RoarNet instead generates smaller cylinder proposals and uses a neural network for proposal refinement.
  • A. 2D Proposal Based Sequential Models: IPOD replaces region-wise image proposals with point-wise proposals by using 2D semantic segmentation to filter background points.Its foreground point cloud preserves contextual and fine-grained location information, and PointsIoU accelerates training and inference.
  • A. 2D Proposal Based Sequential Models: PointFusion combines image and point-cloud features per proposal after projecting 2D boxes to select corresponding points.This feature-level fusion supports 3D box regression, although its proposal stage remains amodal.
  • A. 2D Proposal Based Sequential Models: Pixel-wise fusion projects 3D geometry into image format or appends it as image channels, but image-plane outputs are poorly suited to 3D localization.PointPainting instead appends high-level image semantics to points, enabling multiple point-cloud detectors while coupling their retraining to image-model changes.
  • A. 2D Proposal Based Sequential Models: 3D proposal generation avoids 2D-to-3D proposal transformation and commonly uses multi-view or voxelization approaches.BEV retains orientation and x,y information, whereas voxelization enables standard 3D convolutions but loses some spatial resolution.
  • A. 2D Proposal Based Sequential Models: MV3D generates BEV LiDAR proposals, projects them into LiDAR front-view and image planes, and fuses region-wise features through ROI pooling.Its input top-down feature map contains height, density, and intensity channels.

1) Feature-level fusion:

Feature-level fusion combines image and point-cloud representations within learned networks, using object-centric, point-wise, voxel-wise, and multi-view strategies. The reviewed methods address geometric-detail loss, small-object degradation, and computational redundancy in earlier designs.

  • Feature-level fusion: MV3D projects point clouds into BEV, FV, and RGB views, then uses ROI pooling to obtain view-specific feature vectors.The transformation maps 3D points into multiple views before region-based feature extraction.
  • Feature-level fusion: MV3D can lose small-object information through BEV occlusion and convolutional down-sampling, while ROI pooling spoils fine-grained geometry.Redundant proposals also increase computation during bounding-box regression.
  • Feature-level fusion: AVOD improves MV3D by using BEV and image feature maps for proposals and an auto-encoder to restore feature-map resolution.These changes target small objects that may be reduced to one feature-map pixel.
  • Feature-level fusion: ContFuse replaces object-centric ROI fusion with point-wise continuous convolutions that connect multi-scale image and point-cloud features.It retrieves neighboring BEV points, projects them into the image, and weights features by geometric offsets.
  • Feature-level fusion: MVX-Net fuses image and point-cloud data point-wise or voxel-wise before VoxelNet processing and reports state-of-the-art KITTI results.The method also lowered false-positive and false-negative rates relative to VoxelNet.

C. One-step Models

One-step fusion models combine proposal generation and bounding-box regression in a single stage, making them computationally efficient for real-time platforms. The section also situates fusion strategies across road detection and traffic-sign pipelines.

  • C. One-step Models: One-step models merge proposal generation and bounding-box regression, often reducing computation for real-time mobile applications.Their single-stage design removes a separate proposal-to-regression transition.
  • C. One-step Models: Meyer et al. extend LaserNet into a multimodal multitask network for 3D object detection and semantic segmentation using parallel depth-image and front-view CNNs.Image features are associated with points by projection into the image plane before LaserNet prediction.
  • C. One-step Models: Road-detection methods are grouped into BEV-based and front-camera-view-based approaches according to where LiDAR depth and image information are projected.BEV methods retain objects’ original x, y coordinates and orientation, whereas front-view methods project depth into the image plane.
  • C. One-step Models: Signal-level fusion of dense RGB and sparse depth can lose dense texture information because of low grid resolution.This limitation is reported for a U-net-based road-segmentation network.

B. Traffic sign recognition

Traffic-sign recognition combines LiDAR’s strong sign detectability with camera images’ dense texture for classification. Reviewed pipelines mainly use hierarchical result-level fusion, while newer methods colorize point clouds or exploit geometric and prior information.

  • B. Traffic sign recognition: LiDAR detects traffic signs effectively through retro-reflectivity, but sparse texture limits classification; cameras classify patches well but struggle with 3D localization.This complementary behavior motivates camera-LiDAR fusion for traffic-sign recognition.
  • B. Traffic sign recognition: Typical pipelines locate signs in LiDAR scans and classify corresponding image patches using DBMs, SVMs, or DNNs.These methods use result-level fusion and hierarchical object detection.
  • B. Traffic sign recognition: Typical traffic-sign pipelines assume signs are visible in LiDAR scans, which can fail under occlusion and is limited by mobile-LiDAR detection range.The limitation affects both sign localization and the subsequent image-classification association.
  • B. Traffic sign recognition: Deng et al. combine image and point cloud into a colorized point cloud for traffic-sign detection and classification.Detected signs’ 3D geometrical properties are additionally used to reduce false positives.
  • B. Traffic sign recognition: The reviewed segmentation literature spans 2D, 3D, and instance segmentation, with methods fusing image and point features in different representations.Examples include multi-view projection into 3D voxels, point-wise fusion, and permutohedral-lattice processing.

C. Instance Segmentation

Instance-segmentation methods extend semantic segmentation by distinguishing individual instances, and camera-LiDAR systems implement this through voxel, volumetric, or point-clustering pipelines. Joint reconstruction and tracking can further support temporal scene understanding, but BEV-derived semantics remain sensitive to sensor displacement.

  • C. Instance Segmentation: Instance segmentation jointly performs semantic segmentation and object detection while distinguishing individual instances within each class.This additional instance discrimination makes the task more challenging than semantic segmentation alone.
  • C. Instance Segmentation: 3D-SIS performs voxel-wise 3D instance segmentation by projecting down-sampled multi-view image features into 3D voxel space and combining them with RGB-D data.The down-sampling addresses the resolution mismatch between image and voxelized point-cloud features.
  • C. Instance Segmentation: Narita et al. jointly perform scene reconstruction, 3D semantic segmentation, and 3D instance segmentation from RGB and depth frames.Frame-wise panoptic annotations and depth are associated and integrated into a volumetric map across time.
  • C. Instance Segmentation: 3D-BEVIS propagates semantic and instance features from a 2D BEV representation to points with a graph neural network, then clusters points using mean shift.The framework jointly performs 3D semantic and instance segmentation.
  • C. Instance Segmentation: 3D-BEVIS depends on BEV semantic features, so sensor displacement can introduce occlusions that constrain its instance segmentation.The limitation follows from propagating BEV-derived semantics to 3D points.
  • C. Instance Segmentation: Camera-LiDAR multi-object tracking is organized into Detection-Based Tracking and Detection-Free Tracking according to object initialization.The survey evaluates these approaches on the KITTI multi-object tracking benchmark.

B. Detection-Free Tracking (DFT)

Detection-free tracking uses filtering or fused image–point-cloud representations to track objects without a conventional detection-first pipeline. Online calibration remains important because sensor misalignment can substantially reduce fusion reliability.

  • B. Detection-Free Tracking (DFT): DFT manually initializes objects and tracks them using filtering-based methods.
  • B. Detection-Free Tracking (DFT): Complexer-YOLO decouples real-time 3D object detection and tracking by fusing 2D semantics pointwise into the cloud before voxelization.
  • B. Detection-Free Tracking (DFT): Fusion pipelines require accurate camera–LiDAR registration because calibration parameters vary with mechanical vibration and heat fluctuation.
  • B. Detection-Free Tracking (DFT): Mutual-information calibration is vulnerable to texture-rich environments, large decalibrations, and occlusions from sensor displacement.
  • B. Detection-Free Tracking (DFT): LiDAR-enabled visual odometry also struggles with large decalibrations and cannot operate in real time.

A. Performance-related Open Research Questions

Performance-related research questions concern how fused data should be represented, how temporal information and learning schemes should be used, and when deep learning is appropriate. The review highlights point-based representations, temporal context, less-supervised learning, and explicit objectives as promising directions.

  • 1) Data representation: Image, point, and intermediate representations each trade off compatibility, resolution efficiency, scalability, and preservation of 3D structure.Image representations append geometry to image channels, point representations append RGB features to points, and voxelized intermediates suffer from poor scalability.
  • 1) Data representation: Point representations coupled with point convolutions are identified as having strong potential for camera–LiDAR fusion.
  • 2) Temporal context: Ignoring temporal context can produce point-cloud deformation and sensor-time mismatches, motivating temporal modeling with RNNs or LSTMs.
  • 3) Learning scheme: Supervised fusion depends on large annotated datasets, but multimodal annotation is expensive and time-consuming.
  • 3) Learning scheme: Unsupervised and weakly supervised frameworks could use larger unlabeled or coarsely labeled datasets and potentially improve performance.
  • 4) Deep learning use: Traditional methods retain advantages in interpretability, computational efficiency, and real-time capability, while deep models are not back-traceable.
  • 4) Deep learning use: The review recommends deep learning for applications with explicit objectives that can be verified objectively.

B. Reliability-related Open Research Questions

Reliability-related questions span sensor coupling, compute constraints, adverse conditions, adversarial and corner cases, and open-set detection. The review calls for sensor-agnostic and multimodal systems, broader datasets, and explicit evaluation of unknown objects and attacks.

  • 1) Camera–LiDAR coupling: Camera–LiDAR fusion improves perception but creates signal coupling, so failure of one signal path can cripple downstream modules.
  • 1) Camera–LiDAR coupling: A sensor-agnostic framework with multiple fusion modules and asynchronous multimodal inputs is proposed, although the best solution remains open.
  • 2) All-weather and lighting conditions: Current datasets and methods emphasize favorable lighting and weather, which leads to poor real-world performance under complex conditions.
  • 2) All-weather and lighting conditions: Broader lighting and weather datasets and multimodal methods for complex conditions require further investigation.
  • 3) Adversarial attacks and corner cases: Camera-targeted adversarial attacks are dangerous in safety-critical driving, while multimodal verification may help identify modality-specific attacks.
  • 3) Adversarial attacks and corner cases: Corner cases such as unusual obstacles, costumes, and deceptive vehicle imagery are difficult for camera-only or LiDAR-only pipelines.
  • 4) Open-set object detection: Open-set detection lacks sufficient awareness, testing protocols, and metrics despite autonomous vehicles encountering unknown object classes.
  • 5) Speed–accuracy trade-offs: Processing high-resolution images and large point clouds can cause frame drops, high power consumption, and lower reliability on mobile platforms.
Loading 2004.05224v2…