PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Object Detection from Images for Autonomous Driving: A Survey

Xinzhu Ma, Wanli Ouyang, Andrea Simonelli, Elisa Ricci

arXiv:2202.02980v6cs.CV

TL;DR

Image-based 3D detection is a challenging autonomous-driving problem whose literature lacked a recent comprehensive survey. This paper synthesizes pipelines and components, proposes two taxonomies, and reviews challenges and future directions, including limitations in generalization and multimodal coverage.

  • Problem

    Image-based 3D detection has advanced rapidly, but prior surveys did not provide a comprehensive, structured account of its methods and challenges.

  • Method

    The paper surveys detection pipelines, components, auxiliary-data strategies, datasets, taxonomies, and open research challenges across image-based 3D detection.

  • Results

    The survey organizes existing approaches with two taxonomies and synthesizes methods, technical choices, datasets, evaluation issues, and research directions.

  • Takeaways & Limitations

    The review supports systematic analysis of image-based 3D detection and fairer comparisons in future work.

  • Takeaways & Limitations

    Image-based detectors show substantial performance drops on unseen datasets, objects, or challenging weather, and commonly depend on fixed camera intrinsics.

Abstract

from arXiv · show

3D object detection from images, one of the fundamental and challenging problems in autonomous driving, has received increasing attention from both industry and academia in recent years. Benefiting from the rapid development of deep learning technologies, image-based 3D detection has achieved remarkable progress. Particularly, more than 200 works have studied this problem from 2015 to 2021, encompassing a broad spectrum of theories, algorithms, and applications. However, to date no recent survey exists to collect and organize this knowledge. In this paper, we fill this gap in the literature and provide the first comprehensive survey of this novel and continuously growing research field, summarizing the most commonly used pipelines for image-based 3D detection and deeply analyzing each of their components. Additionally, we also propose two new taxonomies to organize the state-of-the-art methods into different categories, with the intent of providing a more systematic review of existing methods and facilitating fair comparisons with future works. In retrospect of what has been achieved so far, we also analyze the current challenges in the field and discuss future directions for image-based 3D detection research.

1 INTRODUCTION

Image-based 3D detection is important for autonomous driving because cameras are cheaper and easier to deploy than LiDAR, while prior surveys emphasized LiDAR. This survey systematically reviews image-based methods, introduces two taxonomies, and identifies challenges and future directions.

  • Cameras offer a cheaper, easier-to-deploy alternative to LiDAR, whose cost, bulk, and limited long-range coverage restrict applications.
  • Previous 3D detection surveys focused mainly on LiDAR, leaving image-based models without a clear, detailed review.
  • The survey reviews image-based 3D detection frameworks, component-level design choices, datasets, and evaluation considerations.
  • Two taxonomies organize existing methods by framework and input data to support systematic analysis and fairer future comparisons.
  • The paper reviews 80+ image-based detectors and 200+ related works, then summarizes open issues and future research directions.

2 TASK

Image-based 3D detection predicts object categories and 3D bounding boxes from RGB images and camera parameters. Boxes encode location, dimensions, and orientation, while benchmarks may require additional outputs.

  • The task classifies and localizes objects from RGB images and camera parameters in 3D world space.
  • Each 3D bounding box represents location [x, y, z], dimensions [h, w, l], and orientation [θ, ϕ, ψ].
  • Location and orientation are also termed translation and rotation in some works.
  • Some benchmarks additionally require 2D bounding boxes, velocity, or attributes beyond category and 3D-box predictions.

3 DATASETS AND EVALUATION

Public datasets provide the benchmarks needed for image-based 3D detection, with KITTI 3D, nuScenes, and Waymo Open especially influential. Evaluation remains concentrated on KITTI 3D, motivating broader testing on large-scale datasets.

  • KITTI 3D, nuScenes, and Waymo Open are the most commonly used datasets and have strongly promoted 3D detection development.
  • Most papers evaluate mainly on KITTI 3D, often focusing on the Car category, while relatively few report nuScenes or Waymo Open results.
  • KITTI 3D reports more than 40K released images, with about 15K used for its 3D detection benchmark.
  • KITTI 3D contains daylight, good-weather scenes from Karlsruhe and evaluates Car, Pedestrian, and Cyclist categories across three difficulty levels.

4 FRAMEWORKS

The survey organizes image-based 3D detectors by how they represent and lift information from 2D inputs into 3D results. It then reviews result-, feature-, and data-lifting pipelines, including region-based and single-shot designs and their principal methodological trade-offs.

  • 4.1 Taxonomy: The taxonomy first separates detectors based on 2D features from those based on 3D features, then categorizes them by result, feature, or data lifting.Result lifting predicts 2D detections before recovering 3D results; feature lifting constructs 3D features; data lifting transforms input data into 3D representations.
  • 4.1 Taxonomy: The proposed taxonomy emphasizes feature representations and the alignment between 2D inputs and 3D outputs, unlike the conventional single-shot versus region-based division.Pseudo-LiDAR methods illustrate why the conventional division is insufficient: they can use either region-based or single-shot LiDAR detectors.
  • 4.2 Methods Based on 2D Features: Result-lifting methods estimate 2D locations and object properties, then recover 3D locations using predicted depth and camera parameters.These methods require object-center depths rather than dense depth maps and are presented using region-based and single-shot 2D-detection frameworks.
  • 4.2 Methods Based on 2D Features: Region-based methods generate proposals before extracting regional features, whereas single-shot methods directly predict 3D-box properties at feature positions.Single-shot detectors generally provide faster inference and easier hardware deployment, which is important for autonomous driving.
  • 4.3 Methods Based on 3D Features: Feature-lifting methods construct 3D features from image features and predict 3D boxes in 3D space, while data-lifting methods first convert image data into 3D representations.Stereo feature lifting can build a plane-sweep volume and transform it into world space before producing a BEV map; pseudo-LiDAR data lifting enables LiDAR-based detectors to operate on image-derived signals.
  • 4.3 Methods Based on 3D Features: Pseudo-LiDAR pipelines depend strongly on depth-map quality and commonly separate depth estimation from 3D detection, although differentiable representation conversion can enable end-to-end training.A small disparity error can create a large depth error, and the Change of Representation module allows gradients to flow from detection back to depth estimation.

5 COMPARISON OF COMPONENTS

This section compares modular components of image-based 3D detectors, covering feature extraction, auxiliary feature learning, feature alignment and pooling, result prediction, multi-scale designs, and multi-camera or self-ensembling strategies.

  • 5.1 Feature Extraction: Standard backbones commonly use 2D CNNs for image features, while feature- and data-lifting methods also use 3D CNNs and point-wise CNNs.DLA and ResNet are common 2D backbones, while sparse 3D convolution is a popular 3D-feature backbone.
  • 5.1 Feature Extraction: Attention modules re-weight features along selected dimensions, while stereo attention can scale channels using left-right feature correlation.The stereo scheme uses cosine similarity between corresponding left- and right-image channels, making its scaling factors more interpretable.
  • 5.1 Feature Learning: Depth augmentation, feature mimicking, ego-pose alignment, and visible-surface pooling provide complementary strategies for improving image-based 3D features.These methods respectively inject depth cues, transfer LiDAR-derived representations, rectify roll/pitch misalignment, or pool warped features from projected 3D-box surfaces.
  • 5.2 Result Prediction: Multi-scale prediction addresses varied object sizes through layer-specific predictions, feature aggregation, or hybrid designs rather than relying only on the last CNN layer.The baseline uses the last CNN layer, whose receptive field and semantic level may not suit all object scales.
  • 5.2 Result Prediction: Multi-camera systems either process views separately and merge them with global NMS or lift features into a shared BEV representation.Self-ensembling methods alternatively divide objects by depth or difficulty and use parallel prediction heads.

6 LEVERAGING AUXILIARY DATA

Auxiliary data supplies depth, spatial, temporal, or pretrained information that RGB images lack, and the survey organizes methods by these inputs while emphasizing fair comparisons under matched settings.

  • 6. LEVERAGING AUXILIARY DATA: Image-based 3D detection is challenging because RGB inputs lack depth, motivating auxiliary data and complementary feature extraction.The survey considers LiDAR signals, temporal sequences, stereo pairs, and pretrained models as auxiliary sources.
  • 6.1 LiDAR Signals: LiDAR can provide depth annotations, distill spatial cues into image models, or support data lifting through GAN-based generation.These uses occur during training or representation construction even when LiDAR is unavailable at deployment.
  • 6.2 Temporal Sequences: Temporal methods estimate boxes from separate frames, combine adjacent-frame features, estimate camera motion, and update results with a 3D Kalman filter.The initial predictions remain single-frame estimates, while filtering mainly improves temporal consistency and can provide velocity without annotations.
  • 6.3 Stereo Pairs: Stereo images support image-based 3D detection by generating more accurate depth maps or enriching features through fusion, attention, and cost-volume representations.The survey reports that high-quality stereo depth maps, especially for foreground regions, can significantly improve final performance.
  • 6.4 Pretraining: Pretrained depth estimators and external backbone pretraining can improve detector accuracy, but KITTI Depth or Stereo overlap with KITTI 3D validation creates possible data leakage.Reported pretraining sources include DDAD15M for monocular models and Scene Flow for stereo models.
  • 6.5 Input Data as Taxonomy: 39 of 63 methods in Table 2 use at least one auxiliary data type or pretrained weights, and stereo data at inference is reported as especially valuable.The survey therefore argues that comparisons should use the same underlying input-data setting.

7 FUTURE DIRECTIONS

The survey identifies unresolved challenges in image-based 3D detection, including depth ambiguity, costly 3D annotations, limited multimodal and temporal modeling, weak generalization, and underused pre-training. It proposes future work spanning depth–detection integration, richer sensing, sequences, reduced supervision, pre-training, and robustness across conditions.

  • Depth Estimation: Depth estimation remains central because image-based 3D detection depends heavily on precise object distances.The survey highlights joint depth estimation and 3D detection as a promising direction, with reported performance gains when both tasks are trained together.
  • Multi-Modality: Integrating images, LiDAR, RADAR, and thermal data could improve robustness across long-range, adverse-weather, and adverse-illumination conditions.RADAR offers longer sensing distance and greater stability in rain and fog, while thermal cameras can address difficult illumination; the survey notes that multimodal research remains limited.
  • Temporal Sequences: Temporal sequences are underused despite their potential to add spatio-temporal constraints and reduce reliance on full supervision.The survey notes that most methods use single frames, while sequences have enabled self-supervised depth estimation and may support 3D object learning with weaker labels.
  • Beyond Fully Supervised Learning: Reducing dependence on fully supervised learning is important because 3D detection datasets are expensive to create, time-consuming to annotate, and error-prone for long-range objects.Differentiable rendering and RGB-only supervision are identified as promising approaches for relaxing 3D bounding-box annotation requirements.
  • Pre-Training: Pre-training remains underexplored beyond ImageNet weights, although large-scale unlabeled driving data may support better detectors, especially in unsupervised settings.The survey points to techniques such as BERT and MoCo as relevant precedents from NLP and 2D vision.
  • Generalization: Generalization is a security-relevant challenge because image-based detectors can lose performance on unseen datasets, objects, and difficult weather conditions.Camera-intrinsic dependence and dataset-specific object priors are identified as factors contributing to this performance drop.

8 CONCLUSIONS

The survey systematically organizes image-based 3D detection research from 2015 to 2021 by taxonomy, component-level comparison, auxiliary-data analysis, and discussion of open challenges.

  • The survey reviews numerous papers on image-based 3D detection published from 2015 to 2021.
  • It first classifies existing methods according to their high-level structure.
  • It compares detection algorithms through components including feature extraction, loss formulation, and post-processing.
  • The survey discusses auxiliary data, fair-comparison protocols, open challenges, and potential research directions.

A APPENDIX

The appendix covers evaluation metrics, non-maximum suppression algorithms, and relevant issues in image-based 3D detection.

  • The appendix introduces 3D object detection evaluation metrics.
  • It summarizes non-maximum suppression algorithms.
  • It discusses relevant problems in image-based 3D detection.

A.1 Evaluation Metrics

Average Precision is the main evaluation metric used in 3D object detection, with dataset-specific variants adopted by common benchmarks.

  • Average Precision (AP) constitutes the main evaluation metric used in 3D object detection.
  • Benchmark datasets modify the original AP definition to produce dataset-specific evaluation metrics.
  • The survey reviews AP variants used by the KITTI 3D, nuScenes, and Waymo Open benchmarks.

A.1.1 Review of the AP Metric

The AP metric assigns predictions to ground truths using IoU, classifies matches by threshold, computes precision and recall, and integrates interpolated precision across recall positions.

  • Predictions are assigned to ground truths using Intersection over Union between ground-truth and estimated 3D bounding boxes.
  • IoU compares matched predictions with a threshold to determine True Positives or False Positives.
  • Recall and precision are computed from ranked detection results using True Positives, False Negatives, and False Positives.
  • Interpolated precision reduces the impact of wiggles in the precision-recall curve when computing AP.
  • At each recall position, AP uses the maximum precision observed at that position or any greater recall value.

A.1.2 Dataset Specific Metrics

KITTI 3D and Waymo Open adapt AP-based evaluation around 3D or BEV IoU, while nuScenes separates localization, scale, orientation, velocity, and attribute errors before combining them with mAP into NDS.

  • KITTI 3D and Waymo Open: KITTI 3D computes AP using 3D IoU and replaces R11 with the denser R40 recall sampling.The modified sampling removes recall position 0.
  • BEV detection: BEV AP evaluates ground-plane localization using the same calculation process as 3D AP, but computes IoU on the ground plane.BEV detection is treated as an alternative to full 3D detection and is also included in Waymo Open.
  • KITTI 3D and Waymo Open: Waymo Open replaces R11 with R21 and additionally considers accurate heading prediction important.Its AP evaluation uses the modified recall sampling set.
  • nuScenes metrics: nuScenes supplements localization with ATE, ASE, AOE, AVE, and AAE, averages each metric across categories as mTP, and combines mAP with mTP into NDS.ATE measures ground-plane center distance, ASE measures 1 − IoU after alignment, AOE measures yaw difference, AVE measures 2D velocity error, and AAE is 1 − attribute accuracy.

A.2 NMS

Traditional NMS removes redundant detections by iteratively retaining the highest-scoring box and suppressing boxes with high overlap.

  • Traditional NMS: Traditional NMS takes bounding boxes, their scores, and an overlap threshold as inputs, then returns final detections with scores.The algorithm denotes the initial boxes by B, scores by S, and the threshold by Ω.
  • Traditional NMS: The procedure targets duplicate coverage so that a single object is represented by one estimated bounding box.Its output is the final set of detections after recursive suppression.

13 end

The survey compares metrics, efficiency, representations, and sensing modalities, emphasizing that evaluation choices affect apparent performance and that practical deployment requires both accuracy and speed.

  • Metrics and applications: AP-based metrics can be difficult for image-based methods because 3D or BEV IoU falls rapidly with position error, especially for far-away objects.nuScenes instead uses center distance and evaluates object aspects separately, under which image-based methods can surpass LiDAR-based methods in some cases.
  • Metrics and applications: Existing AP metrics penalize false positives more than missed true positives even though different detection errors can pose different practical hazards.The survey therefore argues that ideal metrics should weight object distance and error types differently.
  • Accuracy and speed: Inference speed is as important as prediction accuracy, but many studies emphasize accuracy while pseudo-LiDAR methods add depth-estimation overhead.The survey recommends evaluating both accuracy and speed because standardized comparison protocols remain immature.
  • Image-based and LiDAR-based methods: PointPillars and MonoDIS show a large performance gap on KITTI 3D but similar accuracy on nuScenes, illustrating dataset-dependent comparisons between LiDAR and image-based methods.The survey selects these concurrent representative methods for detailed comparison, while more recent methods are listed on the project website.
  • Image-based and LiDAR-based methods: PointPillars is stronger for large objects, localization, and velocity, whereas MonoDIS performs better for some thin objects, size estimation, and object attributes.These differences reflect complementary strengths across object categories and predicted properties.
  • Pseudo-LiDAR representation: Pseudo-LiDAR avoids expensive LiDAR equipment and provides a bridge to LiDAR detection techniques, but its depth-generation stage adds cost and can cause sub-optimal stage-wise training.The representation also offers higher density than real LiDAR because it derives from high-resolution RGB images.
Loading 2202.02980v6…