PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving

Di Feng, Ali Harakeh, Steven Waslander, Klaus Dietmayer

arXiv:2011.10671v2cs.CVcs.RO

TL;DR

Probabilistic object detection matters for autonomous driving because challenging conditions can increase perception errors, while uncertainty can characterize semantic and spatial prediction reliability. This paper reviews uncertainty-estimation methods, surveys probabilistic detectors and metrics, and benchmarks practical approaches across three driving datasets. The study finds that mAP alone is insufficient for uncertainty quality, while Bayesian Fusion improves spatial uncertainty measures at a runtime cost and sampling-based epistemic methods often add limited benefit relative to their computation.

  • Problem

    Probabilistic object detection lacks consistent comparisons because methods use different architectures, datasets, and evaluation metrics, complicating model selection for autonomous driving.

  • Method

    The paper reviews uncertainty estimation and probabilistic detection methods, then benchmarks RetinaNet-based approaches across KITTI, BDD100k, and Lyft.

  • Results

    mAP alone is insufficient for uncertainty evaluation; Bayesian Fusion gains 3% PDQ and reduces regression NLL by 2.24 points versus Loss Attenuation.

  • Takeaways & Limitations

    Uncertainty quality should be assessed with uncertainty-specific metrics alongside detection accuracy, rather than mAP alone.

Abstract

from arXiv · show

Capturing uncertainty in object detection is indispensable for safe autonomous driving. In recent years, deep learning has become the de-facto approach for object detection, and many probabilistic object detectors have been proposed. However, there is no summary on uncertainty estimation in deep object detection, and existing methods are not only built with different network architectures and uncertainty estimation methods, but also evaluated on different datasets with a wide range of evaluation metrics. As a result, a comparison among methods remains challenging, as does the selection of a model that best suits a particular application. This paper aims to alleviate this problem by providing a review and comparative study on existing probabilistic object detection methods for autonomous driving applications. First, we provide an overview of generic uncertainty estimation in deep learning, and then systematically survey existing methods and evaluation metrics for probabilistic object detection. Next, we present a strict comparative study for probabilistic object detection based on an image detector and three public autonomous driving datasets. Finally, we present a discussion of the remaining challenges and future works. Code has been made available at https://github.com/asharakeh/pod_compare.git

I. Introduction

Probabilistic object detection is framed as essential for safe autonomous driving because difficult conditions increase perceptual errors and uncertainty can support introspection and reaction. The paper reviews uncertainty estimation and compares probabilistic detectors across autonomous-driving settings.

  • I. Introduction: Snow, darkness, and unfamiliar objects can compromise sensors or image quality, increasing perception errors and failure rates.Reliable uncertainty can reflect perception inaccuracy or sensor noise in these environments.
  • I. Introduction: Object detection jointly classifies and localizes traffic participants from RGB, LiDAR, or Radar data.Probabilistic detection estimates both semantic uncertainty for classes and spatial uncertainty for bounding boxes.
  • I. Introduction: The paper systematically reviews uncertainty estimation approaches applied to deep object detectors and conducts a comparative autonomous-driving study.It surveys generic and object-detection-specific methods and evaluation metrics before benchmarking practical approaches.
  • I. Introduction: Bayesian neural networks estimate predictive distributions by integrating the observation likelihood over a posterior distribution of network weights.The paper notes that exact posterior calculation is intractable for high-dimensional, multimodal weight spaces, motivating approximate techniques.
  • I. Introduction: Epistemic uncertainty concerns model capability in fitting training data, whereas aleatoric uncertainty reflects sensor limitations in changing environments.The paper identifies both uncertainty types as important for safe autonomous-driving perception.

D. Practical Methods for Uncertainty Estimation

The paper summarizes four practical approaches for estimating predictive uncertainty: MC-Dropout, deep ensembles, direct modeling, and error propagation. These methods differ in whether they model epistemic or aleatoric uncertainty and in their computational requirements.

  • D. Practical Methods for Uncertainty Estimation: MC-Dropout and deep ensembles target epistemic uncertainty, while direct modeling targets aleatoric uncertainty.Error propagation can target either type depending on its focus.
  • D. Practical Methods for Uncertainty Estimation: MC-Dropout approximates Bayesian inference by averaging predictions from multiple stochastic forward passes.It commonly requires 10 to 50 test-time runs, making real-time critical use difficult because of computational cost.
  • D. Practical Methods for Uncertainty Estimation: Deep ensembles approximate predictive probability with a uniformly weighted mixture of independently trained networks.Their computation and memory costs scale linearly with the number of networks during training and inference.
  • D. Practical Methods for Uncertainty Estimation: Direct modeling assumes output probability distributions and predicts their parameters, such as Gaussian means and variances for regression.Training uses maximum likelihood, with negative log likelihood as the optimization objective.
  • D. Practical Methods for Uncertainty Estimation: Error propagation estimates activation-layer variances and propagates them through the network to its outputs.It can model either epistemic or aleatoric uncertainty depending on the source of uncertainty being propagated.

E. Uncertainty Evaluation

Uncertainty evaluation requires assessing predictive distributions rather than relying only on detection accuracy. The paper presents calibration, sharpness, proper scoring rules, and related metrics for evaluating probabilistic object detection.

  • E. Uncertainty Evaluation: Calibration measures statistical consistency between predicted probabilities and observed outcomes, while sharpness measures predictive concentration around the realized target.A calibrated 0.8 probability should correspond to an event occurring approximately 80% of the time.
  • E. Uncertainty Evaluation: Calibration plots place predictive probability on the horizontal axis and empirical probability on the vertical axis.A well-calibrated model follows the diagonal; curves above or below it indicate under-confidence or over-confidence, respectively.
  • E. Uncertainty Evaluation: Expected Calibration Error ranges from 0 to 1, with smaller values indicating better uncertainty calibration.The paper also identifies ACE and Maximum Calibration Error as common calibration metrics.
  • E. Uncertainty Evaluation: Proper scoring rules are minimized when the predicted distribution matches the distribution that generated the observed target.They can account for both calibration and sharpness without requiring access to the theoretical data-generating distribution.
  • E. Uncertainty Evaluation: Brier Score evaluates classification probability quality from squared differences between predicted class scores and ground-truth labels.Scores range from 0 to 1, with lower values indicating better uncertainty estimation.

A. Introduction

Probabilistic object detectors extend conventional detection pipelines to estimate both classification and bounding-box uncertainty. Existing methods vary in architecture, uncertainty modeling, post-processing, and evaluation, motivating more consistent assessment.

  • Probabilistic object detectors augment deterministic detectors to estimate category and bounding-box probability distributions.Their pipelines commonly include a base network, detection head, and post-processing stage.
  • Deep Ensembles and MC-Dropout are used to estimate epistemic uncertainty, while output layers commonly model aleatoric uncertainty.MC-Dropout can estimate uncertainty through repeated stochastic inferences, and ensembles combine predictions from multiple models.
  • Methods differ in how they represent bounding-box uncertainty, including univariate Gaussian, multivariate Gaussian, and Gaussian-mixture distributions.Some approaches model full covariance matrices or multimodal correlations.
  • Aleatoric uncertainty can also influence post-processing through spatial clustering, uncertainty-aware NMS, or criteria combining regression and classification uncertainty.Output-redundancy methods replace standard NMS with clustering and estimate distributions from redundant detections.
  • Many prior studies evaluate probabilistic detectors primarily with mAP, while proper scoring rules such as NLL assess predicted distribution quality more directly.The review notes that mAP, MUE, and PDQ are not proper scoring rules for predicted distributions.

D. Sensor Modalities

Probabilistic object detection research spans multiple sensor modalities, but sensor coverage is uneven and sensor properties affect aleatoric uncertainty behavior.

  • LiDAR and RGB cameras dominate probabilistic object detection research, whereas radar uncertainty modeling has been proposed by only one cited study.LiDAR studies often address 3D detection, while RGB-camera studies commonly address image-based detection.

E. Applications and Use Cases of Predictive Uncertainty

Studies connect predictive uncertainty components to different sources of variation in autonomous-driving perception.

  • Epistemic uncertainty is associated with detections unlike training samples, while aleatoric uncertainty reflects sensor-observation noise such as distance and occlusion.These behaviors have been studied across object detection and related perception tasks.

IV. Comparative Study

The comparative study addresses inconsistent prior benchmarks by evaluating common probabilistic-detector modifications under a controlled setup. It uses one base network family, shared training data, three datasets, and multiple metrics for 2D image detection.

  • The study targets the lack of consistent benchmarks caused by differing dataset and evaluation-metric combinations across prior methods.This inconsistency makes it difficult to determine which method works best for autonomous driving.
  • All networks are trained on BDD100K and evaluated on BDD100K, KITTI, and Lyft for seven object categories.The experiments focus on 2D image-based detection.
  • The paper compares common modifications that extend deterministic detectors to predict probability distributions using the same base network, datasets, and multiple evaluation metrics.The controlled setup is intended to support a fair comparison.
  • The study also compares aleatoric-uncertainty behavior in image-based and LiDAR-based object detectors.This comparison is presented as part of the experimental analysis.

A. Our Probabilistic Object Detectors

The study re-implements probabilistic object detectors on a common RetinaNet-based image detector and evaluates them across three autonomous-driving datasets, with uncertainty modeled throughout the detection pipeline.

  • Detector implementation: A RetinaNet-based image detector provides the common architecture for fairly re-implementing the uncertainty mechanisms.Bounding boxes are modeled as multivariate Gaussians with diagonal covariance, while classification uses softmax categorical distributions.
  • Detector implementation: The implementations model uncertainty across the base network, detection head, and post-processing stages.Table II summarizes the uncertainty-modeling choices for these three stages across the implemented detectors.
  • Uncertainty mechanisms: Loss Attenuation + Dropout and BayesOD + Dropout use stochastic MC-Dropout during inference, with BayesOD + Dropout fusing uncertainty through Gaussian-mixture data association.Loss Attenuation + Dropout applies standard NMS after multiple stochastic runs, whereas BayesOD + Dropout fuses aleatoric and epistemic estimates.
  • Uncertainty mechanisms: Deep ensembles are constructed from independent Loss Attenuation models, with fusion performed either before or after NMS.Pre-NMS Ensembles fuse detections before NMS; Post-NMS Ensembles associate and fuse detections afterward.
  • Evaluation setting: The comparative evaluation uses the BDD, KITTI, and Lyft autonomous-driving datasets with RGB camera images.The cross-dataset comparison evaluates only Car and Person because these classes share definitions across all three datasets.

C. Training and Inference

The comparative experiments evaluate probabilistic RetinaNet detectors using detection, uncertainty, calibration, runtime, and dataset-shift metrics. Results show that uncertainty quality and computational cost can differ substantially despite similar mAP.

  • General Results and Evaluation Metrics: The evaluation combines mAP, PDQ, classification and regression NLL, calibration errors, minimum uncertainty errors, and runtime.NLL is measured on true-positive detections with IOU > 0.7; MCE evaluates classification calibration and ECE evaluates bounding-box regression calibration.
  • General Results and Evaluation Metrics: All probabilistic extensions remain within 2% mAP of deterministic RetinaNet, whose BDD validation baseline reaches 28.62% mAP.The baseline conforms to the 28%−30% range reported for the original BDD dataset.
  • General Results and Evaluation Metrics: Under dataset shift, mAP drops 2−3% on KITTI and over 30% on Lyft relative to BDD, indicating stronger shift from BDD to Lyft.The dataset-shift evaluation considers the Car and Person categories.
  • Comparison of each method and discussions: mAP does not track regression NLL: Black Box has regression NLL two orders of magnitude larger than other detectors while maintaining comparable mAP.PDQ is more directly related to regression NLL, although a 384.85 NLL reduction from Output Redundancy to Black Box corresponds to only a 0.18 PDQ increase.
  • Comparison of each method and discussions: Aleatoric methods impose smaller runtime losses than MC-Dropout and Deep Ensembles, while ensembles provide only slight mAP gains and no substantial uncertainty-quality improvement.The comparative conclusions are specific to a single-stage detector trained on autonomous-driving datasets; broader 3D and LiDAR comparisons remain future work.
  • Comparison of each method and discussions: BayesOD achieves 3% higher PDQ and 2.24-point lower regression NLL than Loss Attenuation, but reducing NMS-based inference from 15.21 to 9.75 FPS increases inference time by approximately 37 ms.BayesOD also has the lowest regression calibration error and a substantially lower regression minimum uncertainty error than 0.48.

E. Aleatoric Uncertainty in Camera and LiDAR Perception

The paper compares aleatoric uncertainty in front-view RGB image perception and LiDAR perception represented in bird’s-eye view, using KITTI car detections from RetinaNet and ProbPIXOR.

  • RGB images and LiDAR point clouds have distinct sensing properties and observation-noise characteristics that contribute to aleatoric uncertainty.
  • The comparison uses front-view RGB detections from RetinaNet and bird’s-eye-view LiDAR detections from ProbPIXOR on KITTI car objects.
  • Aleatoric regression and classification uncertainties are measured with normalized Total Variance and Shannon Entropy, respectively.Total Variance is normalized by bounding-box diagonal length to account for size differences across detection distances.
  • The section frames modality-specific uncertainty analysis as a basis for comparing how image and LiDAR perception reflect distance and occlusion.

A. Efficient Epistemic Uncertainty Estimation

The paper identifies sampling-based epistemic uncertainty estimation as computationally demanding and highlights broader limitations in uncertainty decomposition, evaluation, and dataset supervision.

  • A. Efficient Epistemic Uncertainty Estimation: MC-Dropout and Deep Ensembles usually require sampling for explicit epistemic uncertainty estimation, creating substantial time and memory costs.MC-Dropout with 10 inferences achieves only 2−3 FPS in the reported benchmark, challenging online autonomous driving.
  • B. Aleatoric Uncertainty Decomposition: Current probabilistic object detectors generally model aleatoric uncertainty as a whole rather than decomposing it into distinct sources.The paper identifies weather, sensor quality, and annotation ambiguity as examples of aleatoric sources.
  • C. Uncertainty Propagation to Downstream Modules: Uncertainty estimates could be propagated to downstream autonomy modules, including object-based SLAM, behavior prediction, and driver assistance.For object-based SLAM, predicted bounding-box means and covariance matrices can serve as measurement updates.
  • D. Better Evaluation of Probabilistic Object Detection: mAP does not assess bounding-box uncertainty quality, so proper scoring rules and calibration errors are recommended for probabilistic detector evaluation.PDQ can summarize performance but is not a proper scoring rule and should not independently rank detectors.
  • E. Ground Truth Uncertainty for Datasets: Open object-detection datasets lack ground-truth information that directly supervises predictive uncertainty, while generating annotation-disagreement labels is expensive and time-consuming.
  • Conclusion: The paper presents a survey, comparative evaluation on a one-stage 2D detector and three autonomous-driving datasets, and open-source benchmarking code.

Appendix

The appendix expands the paper’s uncertainty taxonomy, classification-uncertainty formulation, evaluation metrics, and examples of uncertainty-estimation applications in autonomous driving.

  • Appendix: The appendix details epistemic and aleatoric uncertainty and introduces an alternative categorization based on perception-system development and operation.
  • Another Uncertainty Categorization Method: The alternative categorization defines seven perceptual uncertainty sources spanning problem definition, data collection and annotation, model training, and operation.
  • Estimating Classification Uncertainty with Softmax Logits: Classification uncertainty can be modeled by assigning independently Gaussian distributions to softmax logits, whose means and variances are predicted by the network.Samples are transformed through softmax to compute a standard classification loss such as cross-entropy or focal loss.
  • Applications: Examples of uncertainty-estimation applications include camera pose estimation, semantic segmentation, image annotation, optical flow, trajectory prediction, depth completion, visual tracking, and end-to-end driving.
  • Additional Scoring Metrics: Shannon Entropy measures predictive classification uncertainty, reaching its minimum for certain predictions and its maximum for a uniform distribution.
  • Additional Scoring Metrics: Mutual Information captures prediction variation caused by posterior weight variation and therefore reflects model uncertainty.Its score ranges from 0 to 1, with larger values indicating higher uncertainty.
  • Additional Scoring Metrics: The appendix also discusses error-based uncertainty curves and Total Variance for regression, with Total Variance ignoring correlations among regression variables.

D. Benchmarking Uncertainty Estimation Methods

Prior studies mainly evaluate uncertainty estimation in open-world settings involving domain shift or previously unseen objects, a setting of practical interest for autonomous driving.

  • Open-world evaluations test data distributions that differ from training data or include objects absent from the perception module’s training experience.
  • The paper identifies open-world uncertainty evaluation as practically relevant to autonomous-driving applications.

E. Detailed Introduction of Generic Object Detection

Deep object detection pipelines jointly recognize and localize traffic participants, while autonomous-driving datasets differ in sensors, viewpoints, resolutions, scene appearance, and annotation procedures. These dataset shifts motivate comparative evaluation across BDD, KITTI, and Lyft.

  • Object detection pipelines: Object detectors jointly classify objects and localize them with tightly fitting bounding boxes.One-stage, two-stage, and sequence-to-sequence pipelines differ in how they generate and refine detections.
  • Object detection pipelines: One-stage detectors map inputs directly to boxes and classification scores, whereas two-stage detectors first extract region proposals and then refine them.Common detectors may also require prior anchors and Non-Maximum Suppression for duplicate removal.
  • Datasets and dataset shift: The study evaluates probabilistic object detectors on three RGB autonomous-driving datasets to assess uncertainty under dataset shift.The datasets are BDD, KITTI, and Lyft.
  • Datasets and dataset shift: BDD contains 80,000 frames at 1280 × 720, with diverse North American scenes, weather conditions, and equal daytime and nighttime coverage.Its training/validation split is 70,000/10,000 frames.
  • Datasets and dataset shift: KITTI contains 7,481 frames at 1242 × 375, while Lyft contains 158,757 frames at 1224 × 1024 from Palo Alto.KITTI has clear daytime imagery; Lyft provides 3D annotations from which 2D boxes are projected.
  • Datasets and dataset shift: KITTI and Lyft are naturally shifted from BDD, with KITTI showing a lower shift magnitude than Lyft.The shift is amplified for Lyft because its 2D boxes are projected from 3D labels, unlike BDD’s directly labeled 2D boxes.
Loading 2011.10671v2…