PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic and Geometric Depth: Detecting Objects in Perspective

Tai Wang, Xinge Zhu, Jiangmiao Pang, Dahua Lin

arXiv:2107.14160v3cs.CV

TL;DR

Monocular 3D detection is economical but difficult because depth is absent from the image and inaccurate instance depth limits other predictions. PGD combines uncertainty-aware depth estimation with graph-based geometric propagation, achieving leading benchmark performance while retaining real-time efficiency.

  • Problem

    Monocular 3D detection is a low-cost alternative to sensor-rich systems, but its missing direct depth information makes the task ill-posed and challenging.

  • Method

    PGD represents instance depth probabilistically to capture uncertainty and propagates depth through geometric relations among predicted objects.

  • Results

    PGD achieves significant improvements on KITTI and nuScenes, ranks first among monocular vision-only methods, and maintains real-time efficiency.

  • Takeaways & Limitations

    The results support treating instance depth estimation as the central focus for improving monocular 3D detection.

  • Takeaways & Limitations

    The geometric-depth scheme assumes ground-related geometry, and the authors identify relaxing this assumption and exploiting temporal geometry as future directions.

Abstract

from arXiv · show

3D object detection is an important capability needed in various practical applications such as driver assistance systems. Monocular 3D detection, as a representative general setting among image-based approaches, provides a more economical solution than conventional settings relying on LiDARs but still yields unsatisfactory results. This paper first presents a systematic study on this problem. We observe that the current monocular 3D detection can be simplified as an instance depth estimation problem: The inaccurate instance depth blocks all the other 3D attribute predictions from improving the overall detection performance. Moreover, recent methods directly estimate the depth based on isolated instances or pixels while ignoring the geometric relations across different objects. To this end, we construct geometric relation graphs across predicted objects and use the graph to facilitate depth estimation. As the preliminary depth estimation of each instance is usually inaccurate in this ill-posed setting, we incorporate a probabilistic representation to capture the uncertainty. It provides an important indicator to identify confident predictions and further guide the depth propagation. Despite the simplicity of the basic idea, our method, PGD, obtains significant improvements on KITTI and nuScenes benchmarks, achieving 1st place out of all monocular vision-only methods while still maintaining real-time efficiency. Code and models will be released at https://github.com/open-mmlab/mmdetection3d.

1 Introduction

Monocular 3D detection offers a lower-cost alternative to sensor-rich systems but remains difficult because depth is not directly visible in monocular images. The paper identifies instance depth estimation as the dominant bottleneck and proposes probabilistic and geometric context to address it.

  • Motivation: Monocular 3D detection reduces sensing cost compared with LiDAR- and multi-sensor systems but is inherently ill-posed because monocular images do not directly reveal depth.The paper motivates this setting for robotic systems such as autonomous vehicles.
  • Key observation: Oracle analyses show that inaccurate instance depth blocks other localization predictions from improving final detection performance.The study motivates reducing the problem to instance depth estimation.
  • Approach: PGD predicts probabilistic instance depths to model uncertainty and constructs a geometric relation graph to enhance estimation through contextual connections.The framework uses uncertainty to guide depth propagation across predicted objects.
  • Results: PGD achieves significant improvements on KITTI and nuScenes and ranks first among monocular vision-only methods while maintaining real-time efficiency.The paper reports this outcome across different benchmark settings and metrics.

2 Related Work

Prior monocular 3D detection methods use auxiliary subnetworks, transformed 3D representations, or end-to-end designs, but commonly lack customized treatment of instance depth. The paper positions depth estimation as a specific unresolved challenge rather than merely another regression target.

  • 2D Object Detection: Anchor-based and anchor-free 2D detectors differ in their initial guesses and prior settings, and FCOS3D is selected as a baseline for handling overlapped objects and scale variance.FCOS3D is the 3D-adapted version of FCOS.
  • Monocular Depth Estimation: Monocular depth estimation predicts dense global depth from RGB images and shares the ill-posed nature of monocular 3D detection.The related work distinguishes supervised and self-supervised approaches based on stereo pairs or monocular videos.
  • Monocular 3D Object Detection: Earlier monocular 3D detection methods rely on auxiliary depth or detection subnetworks, transformed 3D representations, or end-to-end detection designs.Some transformed-representation methods depend on dense depth labels and can introduce sensor-domain gaps or efficiency concerns.
  • Remaining gap: Existing methods rarely customize instance depth estimation and instead treat it as ordinary regression for isolated points or instances.The paper identifies this design choice as a barrier to addressing the depth problem directly.

3 Preliminary and Motivating Study

The paper performs oracle-based error analysis on a basic monocular 3D detector to identify which prediction targets limit performance. Across KITTI and nuScenes, inaccurate depth emerges as the dominant bottleneck, motivating the subsequent depth-focused framework.

  • Preliminary setup: The study uses FCOS3D as a simple baseline because monocular 3D detection must regress seven degrees of freedom instead of conventional 2D boxes and labels.The analysis focuses on the more difficult localization targets in the 3D formulation.
  • Oracle analysis: Oracle analysis replaces predictions with ground truths step by step on KITTI and nuScenes to identify the performance bottleneck.The analysis is conducted under different evaluation metrics.
  • Main finding: Inaccurate depth blocks other subtask predictions from improving overall detection performance on both datasets under different metrics.The finding is especially relevant to 3D localization.
  • Framework motivation: Figure 3 summarizes a depth-focused module that estimates probabilistic depth, derives geometric depth through propagation, and integrates both into the final prediction.The module is added to the detector head using features from the regression branch.

4 Our Approach

PGD combines uncertainty-aware local depth estimates with perspective-geometry propagation across detected objects. It then fuses local and graph-based estimates using a location-aware weight map, while pruning and gating edges to improve reliability and efficiency.

  • Motivation: Direct isolated-instance depth regression is difficult because appearance and size vary widely, motivating an uncertainty-aware propagation graph.The graph uses confidence information to guide how depth is propagated between instances.
  • 4.1 Uncertainty Modeling with Probabilistic Representation: A probabilistic depth head quantizes the depth interval and decodes its output as the expectation of a softmax distribution.The resulting distribution also supplies a depth confidence score used during inference ranking.
  • 4.1 Uncertainty Modeling with Probabilistic Representation: The local estimate combines direct regression and probabilistic depth through a sigmoid-weighted interpolation, producing DL for each isolated instance.The probabilistic depth and its confidence score form the basis for the propagation graph.
  • 4.2 Depth Propagation from Perspective Geometry: Perspective geometry estimates one object’s depth from another under the same-ground constraint, including the relation v1d1 = v2d2.Objects closer to the vanishing line are inferred to be farther away, while height differences provide a more general relation.
  • 4.2 Depth Propagation from Perspective Geometry: A 0.1m height error can produce about 1.5m depth error when v2 = 50 pixels, motivating edge pruning, gating, and location-aware weighting.The paper describes this error as acceptable for objects 30m away but still requires mechanisms to avoid possible large errors.
  • 4.2 Depth Propagation from Perspective Geometry: The propagation graph retains influential edges using depth confidence, classification similarity, and 2D distance, then gates message contributions by edge importance.The top-k edge rule limits computation and helps avoid large geometric errors.
  • 4.3 Depth Fusion: A location-aware weight map fuses local DL and graph-based DG into the final depth D, replacing the baseline’s direct regression output.The geometric depth map is parameter-free and gradients through its computation are stopped.

5 Experiments

Experiments evaluate PGD on KITTI and nuScenes, showing state-of-the-art performance, real-time efficiency, and gains from probabilistic and geometric depth components.

  • PGD is evaluated on KITTI and nuScenes using quantitative performance, efficiency, and ablation analyses.The experiments include implementation details, benchmark comparisons, and component studies.
  • PGD achieves state-of-the-art results on both benchmarks with different settings and metrics while maintaining outstanding speed.
  • 36Hz: the KITTI framework achieves substantially better car-detection performance than compared methods despite their stronger backbones or augmentation.The comparison covers methods including M3D-RPN and RTM3D.
  • mAP 37.0% and NDS 43.2%: the single-model RGB-only result is best among compared nuScenes test-set methods.The method exceeds the previous best by more than 3% in mAP and surpasses PointPillars and CenterFusion on mAP.
  • About 9% improvement on NDS: PGD outperforms CenterNet on the nuScenes validation set while training and inferring more efficiently.PGD trains in one day versus three days for CenterNet, with comparable performance before the reported NDS improvement.
  • Probabilistic and geometric depth significantly boost baseline performance, especially for mAP and translation error, when their supporting design choices are enabled.Ablations identify probabilistic depth, depth-score use, edge gating, and gradient cut-off as effective choices.

6 Conclusion

The paper identifies instance depth estimation as the key monocular 3D detection challenge and proposes PGD to improve it using uncertainty and geometric relationships. Results on KITTI and nuScenes demonstrate the approach's efficacy, while future work targets broader geometric settings and temporal information.

  • Instance depth estimation is identified as the dominating bottleneck, particularly for monocular 3D localization.
  • PGD connects uncertain instance estimations through perspective-based geometric relationships to predict depths more accurately.
  • The approach demonstrates efficacy on both KITTI and the large-scale nuScenes dataset.
  • Future work will relax the ground assumption through 2D height regression or ground-normal estimation and extend validation to other 2D detectors.
  • Temporal geometry from consecutive frames remains a proposed direction for addressing instance-depth estimation difficulty.

Appendix

The appendix includes an overview figure of the framework.

  • Figure 5 presents an overview of the proposed framework.

1 Implementation Details

The implementation augments FCOS3D with 2D–3D geometric consistency, dataset-specific losses, and uncertainty-aware depth weighting. Inference combines classification, centerness, and depth confidence before bird’s-eye-view rotated NMS.

  • Geometric constraints: 2D localization regularizes 3D outputs through a consistency loss between predicted 2D boxes and projected 3D boxes.The loss uses GIoU between B2D and Bproj.
  • KITTI-specific losses: Auxiliary key-point offsets strengthen local geometric consistency for KITTI, with eight 3D-box corners contributing 16 additional regression dimensions weighted by 0.2.These offsets are normalized like four-side 2D-box offsets.
  • Uncertainty-aware depth: Depth uncertainty is modeled by predicting log variance s, so exp(−s) adaptively weights depth loss and provides a confidence score.The formulation uses target and predicted depths with a smooth L1 term.
  • Depth-loss variants: Intermediate depth predictions receive alternative weighted L1 supervisions, but these designs stabilize training without improving the final results.The supplied passage reports this as an empirical comparison of loss designs.
  • Inference: At inference, overall confidence multiplies class score, centerness, and depth confidence before rotated bird’s-eye-view NMS.The procedure outputs boxes, class scores, optional attribute scores, and centerness predictions.

2 Explanation of Oracle Analyses

Oracle analyses replace predicted attributes with ground truth values to identify performance bottlenecks across KITTI and nuScenes metrics. Accurate depth is consistently pivotal, while localization and classification matter differently across datasets.

  • Metric interpretation: NuScenes mAP uses center-distance thresholds of 0.5, 1, 2, and 4 meters, so it includes some predictions with location errors between 2 and 4 meters.This criterion differs from KITTI’s 3D-IoU matching.
  • Basic observations: Oracle depth produces the only considerable single-oracle improvement, indicating that current depth estimation is the main performance bottleneck.This pattern appears across the reported metrics and datasets.
  • Basic observations: Accurate depth enables other oracle improvements; without it, even accurate remaining predictions remain close to the baseline.The analysis reports this relationship using sequential ground-truth replacements.
  • Cross-dataset comparison: The oracle trend is consistent across KITTI and nuScenes, but KITTI emphasizes localization whereas nuScenes additionally emphasizes classification.The datasets use different metrics and category settings.
  • Comparison with LiDAR methods: On nuScenes, monocular predictions with location oracles still underperform the best LiDAR-based methods, a result the authors describe as dataset-specific.The paper conjectures that ten-category classification difficulty or point-cloud-based annotation may contribute.

3 Supplementary Experimental Results

The supplementary experiments validate PGD’s geometric depth propagation, analyze its probabilistic and geometric components, and characterize where the approach succeeds or remains vulnerable. Results show improved depth accuracy, while distant objects, inconsistent bottoms, and dataset-dependent scoring effects remain important considerations.

  • 3.1 Basic Validation of Depth Propagation: Using only two accurate depths, propagation estimates the remaining six car depths but produces larger errors for distant objects.The true depths are {5.23, 11.80, 16.50, 22.05, 23.64, 28.53, 29.07, 42.85}, while propagated estimates are {5.23, 11.74, 16.78, 22.92, 21.13, 26.59, 25.78, 36.51}.
  • 3.1 Basic Validation of Depth Propagation: The ground constraint usually holds, but actual topography and annotation noise can create inconsistent object bottoms, especially for distant objects.The proposed safeguards include edge pruning or gating and location-aware integration of geometric depth.
  • 3.2 Quantitative Analysis: The depth score improves precision under low recall and strict matching but can reduce performance at high recall, especially for large objects and tolerant thresholds.On nuScenes, it adds about 0.3% NDS by reducing mATE rather than increasing mAP, indicating dataset-dependent effects.
  • 3.2 Quantitative Analysis: The probabilistic and geometric depth components significantly improve ablation performance, particularly for mAP and mATE, while geometric depth contributes substantially beyond local estimation.Direct regression accounts for about 25.6% of local depth fusion, suggesting an auxiliary residual-regression role.
  • 3.3 Qualitative Analysis: Qualitative nuScenes results show notably improved bird’s-eye-view depth accuracy over FCOS3D, consistent with the reported reduction in mATE.The comparison uses six camera views and a top-view representation, with the clearest improvements appearing for objects marked by red circles.
Loading 2107.14160v3…