Source-linked AI summary
Multi-modal Sensor Fusion for Auto Driving Perception: A Survey
Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, Yikang Li
TL;DR
Autonomous-driving multimodal perception must address noisy raw data, underutilized information, and sensor misalignment. This paper reviews over 50 LiDAR-and-camera studies, proposes a fusion-stage taxonomy, and analyzes unresolved problems and research opportunities. Its supported conclusion is a taxonomy with two major classes and four minor classes for organizing existing fusion methods.
Problem
Multimodal perception is difficult because raw data are noisy, information is underutilized, and sensors are misaligned; existing reviews mainly use traditional fusion-stage categories.
Method
The paper reviews over 50 LiDAR-and-camera perception papers and classifies their fusion methods using a fusion-stage taxonomy with two major classes and four minor classes.
Results
The survey presents strong-fusion and weak-fusion as major classes, with early-, deep-, late-, and asymmetry-fusion as strong-fusion subclasses.
Takeaways & Limitations
The taxonomy organizes multimodal autonomous-driving perception methods while highlighting remaining problems and potential directions for future research.
Takeaways & Limitations
Different-modality sensors can have mismatched spatial resolutions, causing information loss during projection and possible dominance by one modality.
Abstract
from arXiv · showhide
Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers. However, achieving a rather good performance is not an easy task due to the noisy raw data, underutilized information, and the misalignment of multi-modal sensors. In this paper, we provide a literature review of the existing multi-modal-based methods for perception tasks in autonomous driving. Generally, we make a detailed analysis including over 50 papers leveraging perception sensors including LiDAR and camera trying to solve object detection and semantic segmentation tasks. Different from traditional fusion methodology for categorizing fusion models, we propose an innovative way that divides them into two major classes, four minor classes by a more reasonable taxonomy in the view of the fusion stage. Moreover, we dive deep into the current fusion methods, focusing on the remaining problems and open-up discussions on the potential research opportunities. In conclusion, what we expect to do in this paper is to present a new taxonomy of multi-modal fusion methods for the autonomous driving perception tasks and provoke thoughts of the fusion-based techniques in the future.
1. Introduction
Autonomous-driving perception benefits from combining complementary LiDAR and camera data, but noisy inputs, sensor misalignment, and limited fusion taxonomies remain challenges. This survey reviews over 50 papers and proposes a fusion-stage taxonomy with two major classes and four minor classes.
- Motivation: LiDAR and camera complement each modality’s weaknesses, improving perception despite camera occlusion and LiDAR’s distance-dependent resolution and weather vulnerability.Camera data can be occluded in complex scenes, while LiDAR is vulnerable to fog and heavy rain.
- Research gap: Existing reviews mainly classify fusion methods as early-, deep-, or late-fusion, with limited focus on fusion methodology itself.The paper identifies this traditional categorization as insufficient for a systematic methodology-focused review.
- Contributions: The survey reviews over 50 autonomous-driving perception papers and proposes a taxonomy with two major classes and four minor classes.The taxonomy is organized from the fusion-stage perspective.
- Contributions: The taxonomy separates strong-fusion and weak-fusion, with early-, deep-, late-, and asymmetry-fusion as strong-fusion subclasses defined by branch feature representations.The proposed classes characterize how LiDAR and camera representations are combined.
- Scope: The paper surveys LiDAR and camera data formats, analyzes remaining fusion problems, and discusses potential future research directions.Its organization covers perception tasks, input representations, fusion methodology, unresolved problems, and research opportunities.
2. Tasks, and Open Competitions
The paper focuses mainly on autonomous-driving object detection and semantic segmentation, while situating them among other perception tasks and commonly used benchmarks. It also notes that several multimodal tasks are omitted because their fusion modules are not widely discussed.
- Perception tasks: Autonomous-driving perception includes object detection, semantic segmentation, depth completion and prediction, with this paper focusing mainly on the first two.These tasks cover obstacles, traffic lights, traffic signs, lanes, and free space.
- Object Detection: Object detection localizes and classifies predefined categories using 2D rectangles or 3D cuboids.The passage represents 2D boxes as (x, y, h, w, c) and 3D boxes as (x, y, z, h, w, l, θ, c).
- Semantic Segmentation: Semantic segmentation assigns pixels or 3D points to semantic regions, supporting free-space and lane detection.Free-space methods distinguish drivable from non-drivable ground, while lane methods use multiclass masks.
- Scope: The paper omits object classification and depth completion or prediction because multimodal fusion is not widely discussed for those tasks.It states that other perception tasks can often be treated as variants of object detection or semantic segmentation.
- Open Competitions: KITTI, Waymo, and nuScenes are identified as the three commonly used autonomous-driving datasets among more than ten related datasets.KITTI supports 2D, 3D, and bird’s-eye-view detection; Waymo and nuScenes provide multimodal benchmark data with their own evaluation metrics.
3. Representations for LiDAR and Image
The paper organizes sensor inputs around image data and three LiDAR point-cloud formats, then relates each representation to fusion and downstream model design. LiDAR can be processed directly, voxelized, or projected into camera-plane or bird’s-eye-view maps.
- Input representations: Image methods usually retain the raw image format, whereas LiDAR representation varies substantially and influences downstream model design.The survey groups LiDAR formats into point-based, voxel-based, and 2D-mapping-based representations.
- Point-based Point Cloud Representation: LiDAR raw points use quaternion-like coordinates (x, y, z, r), with reflectance providing additional information from surface texture.Here, r denotes each point’s reflectance.
- Point-based Point Cloud Representation: Point-based methods process LiDAR points directly, while voxel-based methods discretize 3D space into voxels containing centroids and local statistical features.Local features can include density, offsets from voxel centroids, linearity, and curvature.
- Voxel-based Point Cloud Representation: Voxelization reduces unstructured-point redundancy and supports sparse convolutions that improve training speed and accuracy in perception tasks.Cylinder-based voxelization is also identified as a more reasonable discretization strategy in recent work.
- 2D-mapping-based Point Cloud Representation: LiDAR can be projected into camera-plane maps or bird’s-eye-view maps to enable 2D CNN processing and facilitate fusion with images.Camera-plane maps share the image format but may contain corrupted pixels because LiDAR has lower projected resolution.
- 2D-mapping-based Point Cloud Representation: Bird’s-eye-view mapping reduces occlusion effects and avoids ground-plane length and width distortion for detection and localization.The representation exploits LiDAR placement on the vehicle roof and places objects on the ground plane.
4. Fusion Methodology
The survey replaces the traditional early-, deep-, and late-fusion taxonomy with a fusion-stage taxonomy separating strong- and weak-fusion methods. Strong-fusion is divided into early-, deep-, late-, and asymmetry-fusion according to how LiDAR and camera representations are combined.
- Taxonomy: Traditional fusion methods are categorized as early-, deep-, and late-fusion according to whether raw data, features, or prediction results are combined.Early-fusion aligns raw sensor data; deep-fusion mixes cross-modal features; late-fusion combines modality-specific predictions.
- Taxonomy: Recent methods cannot be easily classified by the traditional three-category taxonomy, motivating a new division into strong-fusion and weak-fusion.The paper presents the relationships between these two major classes in Figure 2.
- Strong-fusion: Strong-fusion is divided into early-, deep-, late-, and asymmetry-fusion based on the combination stages of LiDAR and camera representations.Strong-fusion methods are highly dependent on LiDAR point clouds rather than camera data, according to the overview.
- Early-fusion: Early-fusion combines LiDAR data at the data level with camera data at either the data or feature level before the downstream CNN.Examples attach image features to voxelized LiDAR tensors or combine branches before CNN input.
- Deep-fusion: Deep-fusion combines LiDAR features with camera data- or feature-level representations, sometimes cascading raw and high-level semantic information.The cited methods extract modality-specific embeddings and fuse them through downstream modules.
- Late- and asymmetry-fusion: Late-fusion combines modality-specific pipeline outputs, while asymmetry-fusion gives one branch a dominant role and uses another branch for auxiliary information.Late-fusion may refine proposal scores using cross-modal proposals, confidence, distance, and IoU; asymmetry-fusion combines different fusion levels across branches.
5. Opportunities in Multi-Modal Fusion
The survey identifies unresolved fusion problems involving misalignment, information loss, underused multi-source information, self-supervision, domain bias, and sensor-resolution conflicts, then outlines research directions for addressing them.
- Overview of Open Problems: Current fusion models face misalignment, information loss, and overly simple fusion operations that limit further perception improvements.The survey groups these challenges into better alignment and information preservation, and more reasonable fusion operations.
- Misalignment and Information Loss: Direct point-to-pixel projection is inaccurate under sensory noise, while incorporating surrounding information can improve performance.Camera and LiDAR data require reorganization into a shared coordinate system because their intrinsic and extrinsic properties differ substantially.
- Misalignment and Information Loss: Projecting 3D LiDAR point clouds into 2D representations inevitably loses information, motivating high-dimensional representations designed specifically for fusion.The survey highlights bird’s-eye-view projection as an example of dimensionality reduction that can discard substantial information.
- More Reasonable Fusion Operations: Concatenation and element-wise multiplication may not bridge large cross-modal distribution discrepancies, motivating cascading structures and bilinear mappings.These alternatives are presented as ways to fuse features with different characteristics more effectively.
- With More Potential Useful Information: Single-frame, front-view systems underuse semantic, spatial, and contextual information, while temporal sequences can provide more robust results.The survey proposes exploiting continuous frames with temporal, contextual, and spatial model designs.
- Self-Supervision for Representation Learning: Cross-modal data contain mutual supervision, but current methods rarely mine inter-modal relationships for pre-training, fine-tuning, or contrastive learning.The survey identifies self-supervised learning as a promising direction for deeper data understanding in autonomous-driving perception.
- Intrinsic Problems in Perception Sensors: Sensor, weather, season, location, and resolution differences create domain bias and information imbalance that hinder adaptation to new scenarios.The survey calls for representations compatible with sensors having different spatial resolutions.
6. Conclusion
The paper surveys more than 50 studies on multi-modal sensor fusion for autonomous-driving perception, analyzing their data representations, taxonomy, remaining problems, and future directions. It concludes by presenting a fusion-perspective classification and proposing directions intended to inform future research.
- The survey reviews over 50 papers on multi-modal sensor fusion for autonomous-driving perception tasks.
- It classifies the literature using a fusion-perspective taxonomy, surveys LiDAR and camera data formats, and analyzes remaining problems.
- The paper introduces new possible research directions intended to enlighten future work on multi-modal sensor fusion.