PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Object Detection for Autonomous Driving: A Comprehensive Survey

Jiageng Mao, Shaoshuai Shi, Xiaogang Wang, Hongsheng Li

arXiv:2206.09474v2cs.CVcs.AIcs.RO

TL;DR

3D object detection is a core perception task for autonomous driving, but challenges remain across sensing modalities, localization, and deployment. This paper comprehensively surveys methods, applications, performance trends, and future directions; it reports substantial progress, including stereo-based methods outperforming monocular approaches on KITTI.

  • Problem

    3D object detection must estimate the locations, sizes, and categories of critical surrounding objects, while monocular methods remain limited by insufficient depth information and poor 3D localization.

  • Method

    The paper systematically reviews LiDAR-based, camera-based, and multi-modal 3D object detection methods, along with applications, temporal data, label-efficient learning, trends, and future directions.

  • Results

    64.66% moderate AP3D for a state-of-the-art stereo method exceeds 16.34% for a state-of-the-art monocular method on KITTI, while LiDAR-based grid detectors reached 82.09% moderate AP.

  • Takeaways & Limitations

    The survey organizes progress across sensor types and identifies improved 3D representations, backbone networks, and detection heads as major drivers of performance gains.

  • Takeaways & Limitations

    Existing detectors are constrained by closed datasets covering limited scenarios and basic classes, limiting performance in critical scenarios and recognition of unknown categories.

Abstract

from arXiv · show

Autonomous driving, in recent years, has been receiving increasing attention for its potential to relieve drivers' burdens and improve the safety of driving. In modern autonomous driving pipelines, the perception system is an indispensable component, aiming to accurately estimate the status of surrounding environments and provide reliable observations for prediction and planning. 3D object detection, which intelligently predicts the locations, sizes, and categories of the critical 3D objects near an autonomous vehicle, is an important part of a perception system. This paper reviews the advances in 3D object detection for autonomous driving. First, we introduce the background of 3D object detection and discuss the challenges in this task. Second, we conduct a comprehensive survey of the progress in 3D object detection from the aspects of models and sensory inputs, including LiDAR-based, camera-based, and multi-modal detection approaches. We also provide an in-depth analysis of the potentials and challenges in each category of methods. Additionally, we systematically investigate the applications of 3D object detection in driving systems. Finally, we conduct a performance analysis of the 3D object detection approaches, and we further summarize the research trends over the years and prospect the future directions of this area.

1 Introduction

3D object detection is a core perception task for autonomous driving, predicting critical objects’ 3D locations, sizes, and classes. This survey addresses fragmented prior research through a comprehensive, systematic analysis across sensory inputs, methods, applications, performance, and trends.

  • Motivation: 3D object detection predicts the locations, sizes, and classes of critical objects in 3D space for autonomous-driving perception.It provides observations used by subsequent driving-system components such as tracking, prediction, and planning.
  • Research gap: Prior methods addressed particular sensory types or data representations, leaving a need for systematic comparison across categories.The survey frames comprehensive analysis of methods’ strengths and weaknesses as valuable to the research community.
  • Survey scope: The paper reviews detection methods across LiDAR, cameras, multi-modal inputs, temporal sequences, and label-efficient learning.It also covers applications of 3D object detection in driving systems.
  • Analysis: The survey structurally and hierarchically organizes approaches and analyzes the potentials and challenges of different method categories.Its taxonomy is presented as hierarchically structured.
  • Outlook: The paper conducts performance and speed analysis, identifies research trends over the years, and discusses future directions.These analyses complement the method review and application coverage.

2 Background

3D object detection predicts 3D bounding boxes from one or more sensory inputs, with object representation and sensor characteristics shaping the task. This background section introduces representations, sensors, datasets, and evaluation metrics while highlighting driving-specific distribution and safety challenges.

  • Problem definition: 3D object detection predicts 3D object bounding boxes in driving scenarios from sensory inputs.Its general formulation maps one or more sensor inputs through a detection model to a set of scene objects.
  • Object representation: A 3D object is commonly represented as a cuboid containing center coordinates, dimensions, heading angle, and object class.Some methods additionally represent ground-plane velocities vx and vy.
  • Sensory inputs: Driving perception commonly uses cameras, LiDAR, and radar, whose cost, semantics, geometric fidelity, range, weather robustness, and velocity measurements differ.Cameras support semantic understanding, while LiDAR directly provides accurate 3D information and radar supplies velocity measurements.
  • Driving-specific challenges: Driving scenes present distinctive point-cloud distributions in which many points concentrate near the LiDAR sensor and distant objects may receive few surface points.This distribution differs from indoor scenes, where scanned surfaces and object coverage are more uniform.
  • Datasets: Driving datasets pair multi-modal sensory data with 3D cuboid annotations, with later datasets improving scale and other collection or annotation properties.KITTI established a standard vehicle-mounted camera-and-LiDAR collection and annotation paradigm.
  • Evaluation metrics: AP-based metrics match predicted and ground-truth cuboids using 3D or bird’s-eye-view IoU, whereas practical metrics relate detection quality to downstream planning or ego-vehicle distance.The survey notes that AP can overlook safety differences between nearby and distant object misdetections.

3 LiDAR-based 3D Object Detection

LiDAR-based 3D object detection is reviewed through point, grid, point-voxel, and range-based representations, with emphasis on balancing effective 3D feature extraction and real-time efficiency. The survey organizes these methods, analyzes their strengths and limitations, and highlights representation-specific trade-offs.

  • Organization: LiDAR-based detection methods are categorized by point clouds or range images and by point-based, grid-based, point-voxel, and range-based representations.The review also examines anchor-based and anchor-free learning objectives and auxiliary tasks.
  • Point-based methods: Point-based detectors progressively sample raw points, learn point features, and predict 3D bounding boxes from downsampled points and features.Their main components are point-cloud sampling and feature learning, including set abstraction, graph operators, attention, and Transformers.
  • Point-based methods: Farthest point sampling improves spatial coverage over random sampling but is sequential and typically too time-consuming for real-time detection.Random sampling is more parallel and efficient, yet nonuniform LiDAR density can cause over-sampling of dense regions and under-sampling of sparse regions.
  • Grid-based methods: Grid-based detectors rasterize point clouds into voxels, pillars, or BEV maps, then use 2D convolutional or 3D sparse networks to detect objects.Sparse convolutions operate only on non-empty voxels and can provide realtime inference speed.
  • Grid-based methods: Grid representations trade spatial detail against memory: smaller cells preserve finer 3D information but increase memory consumption quadratically for 2D grids.Voxels retain more structured 3D information than BEV maps and pillars, while quantization inevitably loses continuous-coordinate information.
  • Point-voxel methods: Point-voxel detectors combine fine-grained point geometry with computationally efficient voxel features through hybrid single-stage or two-stage architectures.Backbone designs use point-to-voxel and voxel-to-point transforms, point-voxel convolutions, auxiliary point networks, and multi-scale fusion.

Analysis: potentials and challenges of the point-voxel based

The survey analyzes range-based, anchor-based, and anchor-free 3D detection methods, emphasizing their representations, learning strategies, efficiency, flexibility, and localization challenges.

  • Range-based 3D object detection: Range images encode 3D distance densely in 2D, enabling conventional or specialized convolutions for efficient feature extraction.Range-based methods vary their models, operators, and views, including range view, bird’s-eye view, and point view.
  • Range-based 3D object detection: Range-view detection remains vulnerable to occlusion and scale variation caused by spherical projection.Using bird’s-eye or point views is one strategy for addressing these issues.
  • Anchor-based 3D object detection: Anchor-based methods use predefined cuboids and IoU-based assignments, combining classification and regression losses to predict 3D boxes.The framework can incorporate binary cross entropy, focal, SmoothL1, and corner losses for classification, localization, dimensions, and orientation.
  • Anchor-based 3D object detection: Around 70k anchors are used on KITTI in one example, while large cells can assign low-IoU anchors to small pedestrians and cyclists.These requirements and assignments complicate training for small objects.
  • Anchor-free 3D object detection: Anchor-free methods remove complicated anchor designs and assign positives through grids, points, range pixels, voxels, pillars, or set-to-set matching.The Hungarian algorithm provides one-to-one assignments in set-to-set approaches.
  • Anchor-free 3D object detection: Center-based anchor-free methods have shown strong potential for small-object detection and outperformed anchor-based methods on widely used benchmarks.Anchor-free assignments also support multiple representations without explicit shape priors.

4 Camera-based 3D Object Detection

Camera-based 3D detection includes monocular, stereo, and multi-view approaches. The survey highlights depth estimation, geometric transformation, and view fusion as central challenges affecting localization and performance.

  • Monocular 3D object detection: Monocular 3D detection is ill-posed because a single image lacks sufficient depth information, leaving localization weaker than LiDAR-based detection.Methods address this through image-based depth inference, geometric constraints, and shape priors, but the problem remains open.
  • Monocular 3D object detection: Image-only methods directly regress 3D box parameters from images using modified 2D detection frameworks and are generally end-to-end, simple, and efficient.Their critical weakness is depth prediction; replacing predicted depth with ground truth yielded more than 20% car AP gain on KITTI.
  • Monocular 3D object detection: Depth-assisted methods use pretrained depth estimators, fuse depth images with monocular images, or transform depth into pseudo-LiDAR point clouds.PatchNet reported that pseudo-LiDAR’s benefit comes from coordinate transformation, since coordinate maps can achieve comparable results.
  • Stereo-based 3D object detection: Stereo methods exploit pixel-wise disparity constraints and stereo matching to improve depth estimation and object localization over monocular detection.Their stronger localization comes with additional time and memory costs from an auxiliary stereo matching network.
  • Multi-view 3D object detection: Multi-view methods commonly project camera features into a unified BEV space or use 3D object queries with Transformer cross-view attention.Accurate camera-to-BEV transformation, reliable query generation, and effective attention design remain major challenges.

5 Multi-Modal 3D Object Detection

Multi-modal 3D object detection combines complementary sensory inputs, especially cameras and LiDAR, through early, intermediate, or late fusion, while radar and HD maps offer additional information and challenges.

  • Overview: Multi-modal approaches fuse LiDAR-camera data, radar signals, or high-definition maps, with LiDAR-camera methods categorized as early-, intermediate-, or late-fusion.The taxonomy is organized by sensor types and, for LiDAR-camera fusion, by the stage at which information is combined.
  • LiDAR-camera fusion: Cameras provide rich color-based semantic features, whereas LiDAR provides 3D localization and structural information, motivating their fusion for accurate detection.
  • Early-fusion based 3D object detection: Early fusion sequentially extracts image knowledge, transfers it to point clouds, and feeds the enhanced point clouds into a LiDAR-based detector.Region-level fusion uses 2D boxes extruded into 3D viewing frustums to narrow point-cloud search regions; point-level fusion incorporates image knowledge at points.
  • Intermediate-fusion based 3D object detection: Intermediate fusion combines image and LiDAR features within detector backbones, proposal generation, or region-of-interest refinement stages.Reported fusion operators include continuous convolutions, hybrid voxel feature encoding, gated attention, BEV pooling, learnable alignments, and Transformers.
  • Late-fusion based 3D object detection: Late fusion combines the 2D and 3D boxes produced independently by camera and LiDAR detectors, improving efficiency but limiting access to rich cross-modal deep features.Parallel modality-specific detection avoids complicated input-point-cloud or intermediate-feature interactions, while restricting semantic integration.
  • Radar and map fusion: Radar offers lower cost, weather robustness, longer range, and velocity measurements, but its sparse and noisy signals make effective processing difficult.Radar is fused with LiDAR or cameras using mechanisms including voxel-based, attention-based, graph-based, and range-azimuth-doppler representations.
  • Radar and map fusion: HD maps provide semantic road information that can serve as a strong prior, yet effectively incorporating map information into detection remains an open challenge.Methods use bird’s-eye-view representations, separate LiDAR and map backbones, feature-map fusion, or map-based prediction filtering.

6 Transformer-based 3D Object Detection

Transformer-based 3D detectors extend conventional convolutional pipelines with flexible representation interactions and larger receptive fields, progressing from specialized modules to DETR-like and Vision Transformer designs.

  • Motivation and challenges: Transformers have shown strong potential in 3D detection because query-key-value interactions are flexible and self-attention provides a larger receptive field than convolutions.Fully connected self-attention also creates a computational challenge because its complexity is quadratic.
  • Transformer architectures: One line of work inserts Transformer modules into conventional 3D detection pipelines, including point backbones, voxel backbones, and detection heads.Examples include point-feature self-attention, sparse and submanifold voxel attention, proposal-to-point attention, and channel-wise attention.
  • Transformer architectures: DETR-like 3D detectors use object queries that interact with features to predict 3D boxes, with reference points aggregating multi-view image features.DETR3D applies cross-attention between object queries and image features before decoding 3D bounding boxes.
  • Transformer architectures: Vision Transformer designs divide inputs into patches and apply self-attention within and across patches, including sparse regional attention for voxel groups.SST additionally shifts regions to generate new patch groupings.
  • Applications: Transformer architectures have been applied across point-based, voxel-based, and point-voxel 3D object detectors.

7 Temporal 3D Object Detection

Temporal 3D object detection uses sequential LiDAR, streaming packets, or videos to improve scene understanding by aggregating temporal information, while incurring latency, memory, and completeness challenges.

  • Overview: Temporal methods process LiDAR sequences, streaming inputs, or videos to exploit relationships across time.The section organizes approaches according to these three input types.
  • Detection from LiDAR sequences: Multi-frame point-cloud methods aggregate temporal features and merge object points across frames to obtain more accurate detections and more complete 3D shapes.
  • Detection from LiDAR sequences: Processing multiple frames inevitably increases time and memory costs, creating severe deployment concerns on embedded devices.
  • Detection from streaming inputs: A rotating LiDAR typically needs 50-100 ms to produce a complete 360° sweep, so the resulting point cloud may not represent the scene at the exact capture time.This temporal mismatch challenges applications requiring minimal reaction times.
  • Detection from streaming inputs: Individual LiDAR packets provide incomplete point clouds, so streaming detectors add context such as spatial memory banks and temporal modules.
  • Detection from videos: Video-based 3D detection can exploit temporal relationships in sequential images, but comparatively few works investigate this setting.

8 Label-Efficient 3D Object Detection

Label-efficient 3D detection addresses poor generalizability and expensive annotation through domain adaptation, weakly supervised, semi-supervised, and self-supervised approaches.

  • Overview: Label-efficient techniques target poor generalizability and limited annotations that arise when detectors are trained fully supervised on specific domains.
  • Domain adaptation: Domain gaps arise from differing sensors, placements, locations, and weather, and detectors trained in one domain often perform poorly in others.Existing methods commonly address particular transfers, leaving generally applicable domain adaptation as a research direction.
  • Weakly supervised detection: Weakly supervised detection reduces annotation effort by using less expensive signals, such as 2D annotations, to train 3D detectors.Manual 3D bounding-box annotation is described as laborious and expensive.
  • Semi-supervised detection: A vehicle can collect more than 100k point-cloud frames daily, while a skilled annotator can label only 100-1k frames per day, producing large unlabeled datasets.
  • Semi-supervised detection: Semi-supervised methods jointly use limited labeled data and abundant unlabeled data, mainly through pseudo-labeling or teacher-student learning.Pseudo-labeling retrains detectors with predictions on unlabeled data, while teacher-student methods adapt teacher-based learning.
  • Self-supervised detection: Self-supervised pre-training first learns from large unlabeled datasets and then fine-tunes on labeled data, but it remains insufficiently explored for autonomous-driving 3D detection.

9 3D Object Detection in Driving Systems

The survey examines 3D object detection within driving systems, emphasizing end-to-end task integration, simulation for rare scenarios, robustness, and collaborative multi-agent perception.

  • Overview: The section surveys applications and challenges spanning end-to-end driving, simulation, robustness, and multi-agent collaboration.These topics broaden 3D detection beyond isolated single-vehicle perception.
  • 9.1 End-to-end driving systems: End-to-end systems jointly train 3D detection with tracking, trajectory prediction, motion planning, or localization to better connect perception with downstream driving tasks.FaF jointly reasons about detection, tracking, and trajectory prediction using a single 3D convolutional network.
  • 9.2 Simulation: Simulation addresses long-tail data by generating rare but critical scenarios, while realistic visual and LiDAR synthesis remains an open challenge.Reviewed approaches simulate images, point clouds, traffic flows, and vehicle safety scenarios using graphics engines, surfels, real-world data, and learned simulators.
  • 9.3 Robustness: Learning-based 3D detectors are vulnerable to adversarial perturbations or objects that can induce misdetections, motivating practical attack and defense algorithms.The survey also notes adversarial messages in multi-agent communication systems as an attack avenue.
  • 9.4 Collaborative detection: Single-vehicle detection suffers from occlusion and sparse far-away objects, motivating collaborative detection that fuses information from vehicles or infrastructure.Collaborative systems may fuse raw sensory inputs or compressed feature maps, trading communication bandwidth against detection performance.

10 Analysis and Outlooks

The paper systematically compares 3D object detectors across modalities, datasets, performance, and inference speed, then uses the analysis to identify research trends and future directions.

  • 10 Analysis and Outlooks: The survey analyzes LiDAR-based, camera-based, and multi-modal detectors on multiple datasets using both detection performance and inference speed.It organizes the analysis around Section 10.1 and proposes future research directions in Section 10.2.

10.1 Research trends

The survey finds substantial progress across LiDAR, camera, and multi-modal detection, while highlighting persistent trade-offs among accuracy, speed, cost, and depth quality.

  • 10.1.1 Trends of dataset selection: Evaluation shifted from KITTI-focused 2D AP2D before 2018 toward AP3D or APBEV and larger datasets such as nuScenes and Waymo.LiDAR performance on KITTI converged, while monocular camera detection retained substantial room for improvement.
  • 10.1.3 Trends of the LiDAR-based methods: Point-based moderate AP increased from 53.46% to 79.57% on KITTI through improved point samplers and increasingly powerful point-cloud operators.Better sampling improves scene coverage under sparsity, while graph operators and Transformers strengthen feature representations.
  • 10.1.3 Trends of the LiDAR-based methods: Grid-based moderate AP increased from 50.81% to 82.09% on KITTI as backbones evolved from 2D BEV processing to sparse 3D convolutions and Transformers.Detection heads also progressed from anchor-based to center-based designs and further localization refinements.
  • 10.1.4 Trends of the camera-based methods: Stereo-based detection reached 64.66% moderate AP3D on KITTI, far exceeding the 16.34% achieved by the state-of-the-art monocular method.The survey attributes the gap mainly to more accurate stereo-derived depth and disparity.
  • 10.1.4 Trends of the camera-based methods: Stereo moderate AP increased from 4.37% to 64.66% on KITTI, with pseudo-LiDAR and 3D volume representations improving the use of disparity information.Accurate transformation from disparity maps into 3D representations is described as crucial for detection performance.
  • 10.1.5 Trends of the multi-modal methods: Multi-modal detection generally improves over single-modal baselines but adds inference time; one detector gained 8.8% mAP while latency rose from 70 ms to 542 ms.The survey notes that early fusion can make the latency problem more severe.
  • 10.1.5 Trends of the multi-modal methods: Intermediate and late fusion improved moderate AP from 62.35% to 80.67% on KITTI and mAP from 52.7% to 69.2% on nuScenes.Recent methods increasingly favor BEV-space fusion, while Transformer-based fusion operators show prominent benchmark performance.
  • 10.1.6 Systematic comparisons: 80.28% moderate AP3D at 30 FPS shows that LiDAR-based detection offers a strong speed–accuracy balance on KITTI.The survey identifies LiDAR as the best overall solution when speed and accuracy dominate, while noting camera-based methods are cheaper.

10.2 Future outlooks

The outlook emphasizes open-world recognition, interpretability, hardware-aware efficiency, and system-level task integration as directions for deploying robust 3D detection.

  • 10.2.1 Open-set 3D object detection: Closed datasets cover limited scenarios and basic classes, leaving detectors weak on rare events and unknown obstacles; open-set 3D detection is therefore promising.The survey specifically highlights traffic accidents and unknown categories as underrepresented or absent.
  • 10.2.2 Interpretability: Deep learning detectors lack interpretability about object identification, occlusion and noise effects, and the context required for detection.The survey attributes these unanswered questions to the black-box property of deep neural networks.
  • 10.2.3 Efficient hardware design: Sparse and irregular point-cloud and voxel operators are poorly suited to existing GPUs, motivating hardware architectures optimized for 3D operators and new sensors.The survey cites solid-state LiDAR, Doppler LiDAR, and 4D radar as sources of future detector-design opportunities.
  • 10.2.4 Task integration: Optimizing average precision alone may not be optimal because 3D detection is correlated with other perception tasks and downstream prediction and planning.The survey motivates end-to-end treatment of detection alongside these related tasks.

11 Conclusion

The paper reviews and analyzes 3D object detection for autonomous driving across problem formulation, sensing approaches, temporal and label-efficient learning, applications, trends, and future directions.

  • The review covers the problem definition, datasets, and evaluation metrics for 3D object detection.
  • It surveys LiDAR-based, camera-based, and multi-modal 3D object detection approaches.
  • It investigates 3D object detection using temporal data and label-efficient learning.
  • It examines applications of 3D object detection in autonomous driving systems.
  • It summarizes recent research trends and prospects future research directions.
Loading 2206.09474v2…