PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, Klaus Dietmayer

arXiv:1902.07830v4cs.RO

TL;DR

Autonomous-driving perception lacks general guidance for fusing heterogeneous sensors across modalities and network stages. This review synthesizes sensors, datasets, perception methods, and fusion strategies for object detection and semantic segmentation, then discusses challenges and open questions. It emphasizes data diversity and alignment, efficient labeling, Radar-fusion gaps, robustness, and real-time constraints as priorities for future work.

  • Problem

    Deep multi-modal perception has many proposed methods but no general guideline for what to fuse, when to fuse, or how to fuse, while dataset and labeling issues remain important.

  • Method

    The paper systematically reviews sensors, test vehicles, datasets, object detection and segmentation methods, fusion methodologies, challenges, and references through an interactive platform.

  • Results

    The review identifies data diversity, temporal and spatial alignment, labeling efficiency, Radar-fusion research, robustness, and computational efficiency as central challenges and open questions.

  • Takeaways & Limitations

    Progress in deep multi-modal autonomous-driving perception requires considering modality representation, fusion operations and stage, robustness, and real-time computational and memory constraints.

Abstract

from arXiv · show

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of "what to fuse", "when to fuse", and "how to fuse" remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

I. INTRODUCTION

Autonomous-driving perception must be accurate, robust, and real-time in complex environments, motivating multi-modal sensing and deep-learning fusion. This review organizes datasets, fusion design questions, methodologies, challenges, and open questions for object detection and semantic segmentation.

  • Requirements: Perception systems must remain accurate, robust to adverse or unfamiliar conditions, and real-time, especially at high speed.Robustness includes operation when sensors are degraded or defective.
  • Multi-modal motivation: Camera and LiDAR fusion often outperforms using either modality alone in published KITTI car-detection methods.The comparison covers deep-learning approaches evaluated by average precision and runtime.
  • Open design questions: The review addresses dataset availability and labeling, scenario diversity and quality, and the unresolved questions of what, when, and how to fuse.These questions concern modality representation, fusion operations, and the feature-processing stage used for fusion.
  • Review scope: The paper summarizes newly published multi-modal datasets, fusion methodologies, and remaining challenges for autonomous-driving perception.Its scope covers object detection and semantic segmentation and is intended to narrow a stated survey gap.
  • Organization and resources: The review combines sensor, vehicle, dataset, perception, and fusion background with an interactive platform for navigating references.It highlights data diversity, temporal and spatial alignment, labeling efficiency, Radar fusion, and network robustness as discussion topics.

1) Visual and Thermal Cameras:

Automotive sensors provide complementary information: cameras offer texture, LiDAR offers depth, Radar offers velocity and weather robustness, and ultrasonics support near-range sensing. Their operating constraints differ across lighting, weather, range, resolution, and environmental conditions.

  • Visual and Thermal Cameras: Visual and thermal cameras provide detailed texture, while thermal cameras are more robust to daytime/nighttime changes than visual cameras.Neither camera type directly provides depth information.
  • LiDAR: LiDAR provides accurate 3D depth and is robust to lighting, fog, and rain, but lacks camera-like fine texture and becomes sparse.Its points are also sparse and irregular, complicating downstream processing.
  • Radar: Radar is robust to lighting and weather and estimates radial velocity through the Doppler effect, but its low resolution makes object classification difficult.Radar is commonly used in adaptive cruise control and traffic-jam assistance.
  • Ultrasonic Sensors: Ultrasonic sensors measure near-range distances and are mainly used for low-speed scenarios such as automated parking.Humidity, temperature, and dirt substantially affect their sensing properties.
  • Test Vehicles and Data: Research vehicles and open datasets commonly combine cameras, LiDARs, and GPS/IMUs to collect images, point clouds, and localization information.Examples include the DARPA Urban Challenge vehicle and Waymo testing platforms.
  • Deep Object Detection: Deep object detection recognizes and localizes multiple scene objects using classification probabilities and bounding boxes.Modern image-based detectors primarily use two-stage or one-stage pipelines.

1) Two-stage Object Detection:

Two-stage object detectors first generate region proposals and then classify and refine them, typically achieving higher accuracy at the cost of speed and training complexity.

  • 1) Two-stage Object Detection:: Two-stage detection extracts class-agnostic regions of interest before verifying, classifying, and refining their scores and locations.Faster R-CNN uses a preprocessing feature extractor, an RPN for proposals, and a head that fine-tunes each proposal.
  • 1) Two-stage Object Detection:: Faster R-CNN comprises a high-level image-feature network, a Region Proposal Network, and a proposal-refinement head.
  • 1) Two-stage Object Detection:: Two-stage detectors generally provide better detection accuracy, but require higher inference time and more complex training than one-stage detectors.
  • 1) Two-stage Object Detection:: Semantic segmentation labels scene pixels or combines object detection with per-instance pixel labeling, while panoptic segmentation unifies both levels.
  • 1) Two-stage Object Detection:: Segmentation pipelines commonly use fully convolutional encoder-decoder networks to transform hierarchical features into dense pixel-label predictions.Two-stage variants generate proposals before fine-tuning instance-level masks; one-stage FCNs produce and upsample coarse output maps.
  • 1) Two-stage Object Detection:: Real-time semantic segmentation remains important because many studies emphasize accuracy rather than operations and inference speed.Comparative evaluations use GFLOPs and frames per second to assess runtime.

C. Dataset Size

Reviewed autonomous-driving datasets vary widely in size, labels, object classes, and recording conditions, with substantial growth but persistent imbalance and diversity limitations.

  • C. Dataset Size: Dataset size ranges from 1,569 frames to over 11 million frames, while nuScenes contains nearly 1.4M labeled frames.
  • C. Dataset Size: Multi-modal dataset size grew by two orders of magnitude between 2014 and 2019, yet remains relatively small compared with computer-vision image datasets.
  • D. Labels: Most reviewed datasets provide labels for 2D detection and semantic segmentation, while several also support 3D scene understanding and tracking.
  • D. Labels: Datasets differ substantially in class granularity, ranging from people and cyclists to fine-grained categories such as trucks, tricycles, traffic cones, and trash cans.
  • D. Labels: Object classes are highly imbalanced, with substantially more cars labeled than people or cyclists.

B. Deep Multi-modal Semantic Segmentation

Multi-modal semantic segmentation combines complementary sensors, but effective systems must choose suitable representations and fusion points for sparse, heterogeneous inputs.

  • B. Deep Multi-modal Semantic Segmentation: Fewer studies address multi-modal semantic segmentation than multi-modal object detection, using combinations of RGB, thermal, depth, and LiDAR data.
  • B. Deep Multi-modal Semantic Segmentation: Network design must determine what modalities to fuse, how to represent them, which fusion operations to use, and when to combine features.
  • 1) LiDAR Point Clouds:: LiDAR point clouds encode depth and reflectance through coordinates, distance, density, HHA features, and intensity.
  • 1) LiDAR Point Clouds:: Point clouds can be voxelized, processed directly as points, or projected into spherical, camera-plane, and bird’s-eye-view feature maps.Voxelization preserves 3D shape but creates many empty voxels; direct point processing avoids voxelization; 2D projections enable 2D convolutions.
  • 1) LiDAR Point Clouds:: Spherical maps provide compact dense point representations but may differ in size from camera images, complicating early fusion.
  • 1) LiDAR Point Clouds:: Bird’s-eye-view maps preserve object length and width and provide ground-plane positions, making localization easier and supporting 3D perception.
  • B. Deep Multi-modal Semantic Segmentation: Camera images offer rich texture but lack direct depth, while thermal cameras are more robust to daytime/nighttime changes.

3) Processing LiDAR Points and Camera Images in Deep Multi-modal Perception:

Deep multi-modal perception commonly processes LiDAR and camera signals through 2D feature representations, with fusion methods aligning or combining features across sensor viewpoints. Radar signals can likewise be represented as 2D feature maps and processed by convolutional networks.

  • Processing LiDAR Points and Camera Images: Most reviewed methods fuse LiDAR and camera features extracted by 2D convolutional networks.LiDAR points are projected onto a 2D plane before feature-map processing; only a few methods use PointNet or 3D convolutions for LiDAR features.
  • Processing LiDAR Points and Camera Images: Several object-detection methods cluster and segment 3D LiDAR points to generate 3D region proposals, while still using LiDAR 2D representations for fusion.
  • Processing LiDAR Points and Camera Images: Methods align sensor features by projecting LiDAR points onto the camera plane or RGB images onto the LiDAR BEV plane.Other methods directly fuse LiDAR BEV features with RGB images, suggesting that networks can implicitly learn viewpoint alignment.
  • Processing LiDAR Points and Camera Images: Accurate spatial and temporal calibration is important when combining features from different sensor viewpoints.
  • Radar Signals: Radar data can be represented as 2D feature maps and processed by convolutional neural networks for classification and semantic segmentation.Radar grid maps may accumulate data over several timestamps for static-object classification and semantic segmentation.
  • How to Fuse: Fusion operations combine modality feature maps by element-wise addition, averaging, concatenation, or ensembling.Concatenated features are stacked along depth before convolution, flattened and concatenated for fully connected layers, or used to fuse regions of interest in detection networks.

4) Mixture of Experts:

Deep networks can fuse modalities at early, middle, or late representation stages, while Mixture of Experts explicitly models modality-specific feature weights. The review finds no conclusive evidence that one fusion method is universally better; performance depends on modalities, data, and architecture.

  • Mixture of Experts: Mixture of Experts processes each sensing modality with a domain-specific expert network and explicitly models feature-map weights.The approach addresses differences in modality informativeness, such as RGB imagery providing less information than LiDAR at night.
  • When to Fuse: Deep networks offer early, middle, and late fusion according to the stage at which modality representations are combined.Methods spanning early through late stages are categorized as middle fusion for simplicity.
  • Comparison and Limitations: No conclusive evidence shows that one fusion method is better than the others; performance depends on sensing modalities, data, and network architectures.
  • Early Fusion: Early fusion combines raw or pre-processed sensor data and learns joint features from the beginning of the network.It has low computation and memory requirements but is inflexible when modalities or input channels change.
  • Late Fusion: Late fusion combines decision outputs from domain-specific modality networks.

GMi

Middle fusion combines modality-specific feature representations at intermediate layers, offering flexibility but leaving the best fusion placement architecture-dependent. The review distinguishes one-time and hierarchical fusion variants alongside early and late fusion.

  • GMi: Late fusion is flexible and modular but requires high computation and memory, while discarding potentially useful intermediate features.Adding a modality generally requires training only its domain-specific network.
  • GMi: Middle fusion combines feature representations from different sensing modalities at intermediate network layers.This allows cross-modal learning across different feature representations and depths.
  • GMi: Fusion can begin at a selected intermediate layer and proceed through subsequent network layers.The formulation defines l⋆ as the layer where intermediate features begin to be fused.
  • GMi: Middle-fusion architectures include one-time, hierarchical, and short-cut fusion strategies.The review presents these as alternative ways to combine intermediate features.
  • GMi: The optimal intermediate-layer fusion strategy is difficult to identify for a specific network architecture.The review treats this as an open challenge rather than providing a general selection rule.

4) Fusion in Object Detection Networks:

Object-detection networks offer multiple locations and mechanisms for fusing modalities, including proposals, regional features, and decision-level outputs. The review places these design choices within broader requirements for accurate, robust, and real-time perception.

  • 4) Fusion in Object Detection Networks:: Two-stage detectors can fuse modalities when generating regional proposals or when combining regional multi-modal features.This creates multiple fusion locations within the detection pipeline.
  • 4) Fusion in Object Detection Networks:: Feature concatenation is the most common fusion operation, while averaging, addition, ensembles, and Mixture of Experts are also used.Concatenation is especially common at early and middle stages; ensembles and Mixture of Experts extend toward decision-level fusion.
  • 4) Fusion in Object Detection Networks:: Multi-modal perception must satisfy accuracy, robustness, and real-time requirements because its outputs feed maneuver prediction and decision-making modules.Reliable perception is described as a prerequisite for safe operation in complex, uncontrolled environments.
  • 4) Fusion in Object Detection Networks:: MV3D, AVOD, Frustum PointNet, and Ensemble Proposals exemplify fusion architectures for two-stage object detection.The figure lists these four representative architectures.
  • 4) Fusion in Object Detection Networks:: Open challenges include limited scenario diversity, labeling errors, sensor misalignment, and the lack of clear architecture-design guidance.The review organizes these issues as challenges in data preparation and network architecture design.

1) Data Diversity:

Multi-modal perception is constrained by limited and costly training data, imperfect labels and alignment, and incomplete treatment of uncertainty and sensor redundancy. The review surveys augmentation, collaborative labeling, calibration, and robustness-oriented directions.

  • 1) Data Diversity:: Real-world multi-modal datasets are usually smaller than image datasets because collection is costly, time-consuming, and hardware-constrained.Large datasets with diverse conditions, labels, and sensors are desirable for training deep networks.
  • 1) Data Diversity:: Simulation-based augmentation can improve training data by adding blank areas, illumination changes, occlusions, and random noise.The review cites a KITTI study attributing its largest performance gain to data augmentation rather than network-architecture advances.
  • 1) Data Diversity:: Collaborative labeling combines weak human annotations with pre-trained network refinement to reduce the burden of labeling LiDAR point clouds.The approach is illustrated for 3D detection and semantic segmentation data.
  • 1) Data Diversity:: Label quality affects perception performance, with randomly distributed errors producing greater robustness than biased labeling errors.The comparison is illustrated using mAP normalized to performance on undisturbed labels.
  • 1) Data Diversity:: Spatial and temporal sensor misalignment can severely degrade training and deployment performance, especially when alignment is learned implicitly.Deep networks have been proposed to classify or regress calibration errors and iteratively refine extrinsic calibration.
  • 1) Data Diversity:: Open questions include comparing LiDAR representations, supporting redundant sensors, propagating uncertainty, and using generative models for robustness.The review mentions voxels, BEV maps, spherical maps, depth maps, sensor defects, motion planning, VAEs, and GANs as relevant directions.

3) When to Fuse:

The review finds no conclusive evidence that one fusion timing is superior and argues that architecture selection, efficiency, evaluation, and temporal consistency remain unresolved. These gaps limit direct comparison and deployment assessment.

  • 3) When to Fuse:: Reviewed fusion timing choices are mainly based on intuition and empirical results, with no conclusive evidence favoring one scheme.The review suggests neural-network structure search as a possible route toward automatic architecture selection.
  • 3) When to Fuse:: Visual analytics could help diagnose multimodal network behavior and improve architecture design, but this application remains unstudied.Existing visualization methods discussed in the review concern CNNs for image classification.
  • 3) When to Fuse:: Real-time perception requires efficient modality representations and fusion operations compatible with acceleration techniques such as pruning and quantization.The review frames efficiency at the what-to-fuse and how-to-fuse levels.
  • 3) When to Fuse:: Inference speed is difficult to compare because reviewed methods use different hardware and programming conditions without a standardized benchmark.Reported timings are author-provided, leaving automotive-hardware performance open to evaluation.
  • 3) When to Fuse:: Standard detection and segmentation metrics do not specifically evaluate open-set conditions or degraded and defective sensors.The review points to calibration, log predictive probabilities, and detection error as uncertainty-related evaluation tools.
  • 3) When to Fuse:: Most reviewed methods process single frames, so their predictions are not dependent on previous frames and can be temporally inconsistent.Only a few works incorporate temporal cues, motivating future time-series methods.

VII. CONCLUSION AND DISCUSSION

The survey consolidates multimodal datasets and fusion methodologies for autonomous-driving object detection and segmentation, while identifying unresolved evidence and research gaps.

  • The survey summarizes multimodal datasets and fusion methodologies for deep object detection and semantic segmentation.It organizes fusion around what, how, and when to fuse.
  • An interactive online tool lets readers navigate topics and methods for each reference and is intended to be updated frequently.
  • Most reviewed datasets and methods use RGB cameras, thermal cameras, and LiDARs, while Radar fusion has only recently been investigated.Examples of newer Radar datasets include nuScenes, Oxford Radar RobotCar, and Astyx HiRes2019.
  • The survey finds no conclusive evidence that one fusion method is better than others.Fusion methods vary in sensor representation, fusion operation, and fusion stage.
  • Research remains limited for multimodal perception in open-set conditions and with sensor failures.The authors identify these as challenging topics requiring more attention.

(benchmark) Recording area Size Categories / Remarks Link

The benchmark material covers multimodal datasets and summarized detection and segmentation methods, including their sensing modalities, task categories, fusion choices, and selected KITTI performance and runtime tables.

  • Datasets: The dataset overview includes entries such as Singapore data with camera and Radar recordings and nuScenes data with six visual cameras and 3D annotations.The supplied dataset fragments also identify object classes including cars, trucks, buses, persons, animals, traffic cones, and temporary traffic barriers.
  • Task coverage: The benchmark section includes separate summaries for multimodal object detection and semantic segmentation methods.
  • Object detection: The object-detection summaries include 3D car, pedestrian, and cyclist detection using LiDAR BEV maps, RGB images, or both.Listed methods include R-CNN-based systems and representations processed with PointNet or 2D CNNs.
  • Fusion design: The method tables distinguish fusion stages such as early, middle, and late fusion, along with feature concatenation and continuous fusion.Some entries use region proposals or detection outputs as the fusion interface, while others include auxiliary depth-estimation tasks.
  • Evaluation: Separate tables report performance and runtime for 3D object detection and urban road segmentation on the KITTI test set.
Loading 1902.07830v4…