Delving into Localization Errors for Monocular 3D Object Detection

Xinzhu Ma; Yinmin Zhang; Dan Xu; Dongzhan Zhou; Shuai Yi; Haojie Li; Wanli Ouyang

Source-linked AI summary

Delving into Localization Errors for Monocular 3D Object Detection

Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, Wanli Ouyang

arXiv:2103.16237v1Mar 30, 2021cs.CV

TL;DR

Monocular 3D detection remains less accurate than LiDAR- or stereo-based detection, motivating quantitative diagnosis of its limiting factors. The paper identifies localization error as the bottleneck and proposes strategies targeting annotations, training samples, and optimization losses, with experiments on KITTI showing their effectiveness.

Problem
Monocular-image 3D detection is substantially less accurate than LiDAR- or stereo-based detection, and the factors restricting its performance require quantitative identification.
Method
The paper conducts diagnostic experiments and proposes three strategies: supervise projected 3D centers, discard difficult distant training samples, and use an IoU-oriented loss for 3D size estimation.
Results
Localization error is identified as the bottleneck of monocular 3D detection, while the proposed strategies are effective in extensive KITTI experiments.
Takeaways & Limitations
Improving localization and addressing its effects through annotations, sample selection, and loss optimization can boost monocular 3D detection.
Takeaways & Limitations
Qualitative results attribute remaining failures mainly to imprecise depth or center estimation, while dimension and orientation estimation remain accurate.

Abstract

from arXiv · show

Estimating 3D bounding boxes from monocular images is an essential component in autonomous driving, while accurate 3D object detection from this kind of data is very challenging. In this work, by intensive diagnosis experiments, we quantify the impact introduced by each sub-task and found the `localization error' is the vital factor in restricting monocular 3D detection. Besides, we also investigate the underlying reasons behind localization errors, analyze the issues they might bring, and propose three strategies. First, we revisit the misalignment between the center of the 2D bounding box and the projected center of the 3D object, which is a vital factor leading to low localization accuracy. Second, we observe that accurately localizing distant objects with existing technologies is almost impossible, while those samples will mislead the learned network. To this end, we propose to remove such samples from the training set for improving the overall performance of the detector. Lastly, we also propose a novel 3D IoU oriented loss for the size estimation of the object, which is not affected by `localization error'. We conduct extensive experiments on the KITTI dataset, where the proposed method achieves real-time detection and outperforms previous methods by a large margin. The code will be made available at: https://github.com/xinzhuma/monodle.

1. Introduction

The paper diagnoses monocular 3D detection and identifies localization error as its main bottleneck. It proposes strategies addressing localization, distant training samples, and 3D size optimization, achieving improved KITTI bird’s-view and 3D detection.

Monocular 3D detection remains substantially less accurate than LiDAR- or stereo-based detection, motivating quantitative diagnosis and targeted solutions.
Ground-truth location produces a performance leap comparable to state-of-the-art LiDAR methods, identifying localization error as the key restricting factor.
The method addresses localization by revisiting 2D-box and projected-3D-center misalignment while retaining 2D detection as an auxiliary task for shared feature learning.
Detection accuracy decreases with distance, and distant samples can impair overall performance; the proposed schemes remove these samples or reduce their training-loss weights.
Even with ground-truth location, dimension estimation leaves 27.4% room for improvement, motivating an IoU-oriented loss that weights sides by their contribution to 3D IoU.
The proposed method improves bird’s-view and 3D object detection by at least 1.6 points AP40 on KITTI compared with existing best-performing monocular approaches.

2. Related Work

Related work includes standard RGB-based monocular 3D detectors, geometric and orthographic feature transformations, and joint 2D/3D detection approaches.

Standard monocular 3D detection methods use RGB images, annotations, and camera calibrations provided by KITTI.
Prior work improves representation through geometric constraints, orthographic feature transformation, or disentangled joint training of 2D and 3D detection.
The paper’s related discussion connects projected 3D-center supervision with localization accuracy and treats 2D detection as necessary for shared 3D-detection features.

3. Approach

The approach diagnoses monocular 3D detection errors by replacing predictions with ground truth, then targets localization, difficult distant samples, and size optimization with three complementary designs.

Baseline Model: The baseline uses an anchor-free CenterNet-style detector with a DLA-34 backbone and lightweight heads for joint 2D and 3D detection.Seven heads predict classification, 2D center and size, plus 3D center, depth, size, and heading angle.
Error Analysis: Replacing the projected 3D center prediction with ground truth raises AP40 from 11.12% to 18.97% on KITTI validation under moderate settings.This diagnostic identifies center localization as a major source of monocular 3D detection error.
Error Analysis: A 4–8 pixel image-plane localization error can prevent correct detection at 60 meters, while the theoretical maximum acceptable error is 0.62m at IoU 0.7.Depth estimation errors accumulate with localization errors, making distant monocular 3D boxes especially difficult to estimate.
Revisiting Center Detection: Using the projected 3D center rather than the 2D box center supervises coarse localization while preserving 2D detection to learn geometric shared features.The 2D branch estimates the offset between the real 2D center and the projected 3D center.
Training Samples: Training samples beyond 60 meters can be discarded through hard coding, or downweighted with a reverse sigmoid-like scheme based on sample depth.Both schemes are reported to be effective and to produce similar results.
IoU Oriented Optimization: The proposed IoU-oriented loss adjusts 3D size-estimation weights using each size side’s partial derivative with respect to IoU.This isolates size optimization from localization-related errors that could otherwise overwhelm the size-estimation objective.

4. Experimental Results

Experiments on KITTI evaluate benchmark performance, component ablations, runtime, and qualitative behavior. The proposed method improves detection results, runs in real time, and shows remaining errors tied mainly to depth or center estimation.

Main results: The proposed method improves KITTI test-set Car 3D detection by 2.47/2.27/1.64 AP40 under easy/moderate/hard settings.These gains are reported over previous methods under fair conditions.
Main results: The method improves moderate KITTI test-set BEV detection and AOS by 4.06/4.12 AP40, reaching 18.89/90.23.The values correspond to BEV detection/AOS under the moderate setting.
Validation results: On KITTI validation, the method outperforms all previous methods in 3D detection and beats MonoPair under the strict 0.7 IoU BEV condition.MonoPair is slightly better under the looser 0.5 IoU condition; the proposed method also shows better validation-test consistency.
Latency analysis: The model runs at 25 FPS on a single GTX 1080Ti, and is 4× faster than M3D-RPN.It also saves 16 ms per image compared with MonoPair, while requiring no postprocessing.
Runtime considerations: Pretrained depth estimators can add substantial computational overhead; DORN takes about 400 ms to process a standard KITTI image.This overhead contrasts with the proposed model's reported real-time runtime.
Other categories: Pedestrian and cyclist detection remains more difficult and may fluctuate because the two categories have relatively few training samples.The proposed method performs better than [2] and comparably to [10] on these categories.
Ablation studies: The projected-center design improves 3D detection accuracy by 1.5, the IoU loss by 0.3, and distant-sample discarding by 0.7.These gains are measured in the accumulation study on the KITTI validation set.
Qualitative results: Qualitative outputs are accurate for objects at reasonable distances, but false positives remain, chiefly from imprecise depth or center estimation.Dimension and orientation estimates for the cited false-positive cases remain accurate.

5. Conclusion

The paper identifies localization error as the bottleneck in monocular 3D detection and proposes three strategies targeting annotations, training samples, and optimization losses. KITTI experiments support the effectiveness of these strategies.

Conclusion: Localization error is identified as the bottleneck restricting monocular 3D detection.The paper links this diagnosis to systematic analysis of the task's problems.
Conclusion: The method directly detects the projected 3D center, discards distant training samples, and uses an IoU-oriented loss for 3D size estimation.The strategies operate on annotations, training samples, and optimization losses, respectively.
Conclusion: Extensive KITTI experiments show the effectiveness of the proposed strategies.

Supplementary Material

The supplementary material adds implementation details, uncertainty-based depth-estimation experiments, theoretical analysis, and qualitative results. It reports that uncertainty modeling improves depth and overall monocular 3D detection, with Laplace modeling outperforming Gaussian modeling for this task.

Supplementary scope: The supplement provides implementation details for depth estimation and ablations for the proposed IoU-oriented loss.It also includes additional discussion and visual results.
Depth estimation: The depth branch simultaneously predicts depth d and standard deviation σ, modeling heteroscedastic aleatoric uncertainty.The supplement considers σ or variance σ2 as the uncertainty output.
Depth estimation: The depth-estimation loss is formulated with Laplace or Gaussian likelihoods using the predicted uncertainty and ground-truth depth.The network is parameterized by w and takes input data x.
Supplementary experiments: Uncertainty-based estimation improves depth-map accuracy and overall monocular 3D detection performance.
Supplementary experiments: Laplace-distribution uncertainty modeling is more suitable for this task than Gaussian-distribution modeling.The main-paper models use the Laplace setting.

C.1. Proof of Proposition

The proposition analyzes how predicted 3D dimensions contribute to 3D IoU when all other prediction components are correct. It derives this contribution across different relationships between predicted and ground-truth dimensions.

The proposition assumes that all predicted items except the 3D sizes h, w, and l are completely correct.
The proof formulates 3D IoU using predicted dimensions and ground-truth dimensions h∗, w∗, and l∗.
The proof considers cases defined by whether each predicted dimension is greater than or less than its ground-truth counterpart.
The derivation evaluates partial derivatives of 3D IoU with respect to h, w, and l to obtain each predicted side’s contribution ratio.
When predicted and ground-truth dimensions are approximately equal, the derivation yields the contribution formulation used in Equation 5 of the main paper.

C.2. Experiments

The experiments examine the proposed IoU-oriented loss against 3D GIoU loss on KITTI validation data. They evaluate Car AP40 under the moderate setting.

The ablation study compares the proposed loss function with 3D GIoU loss on the KITTI validation set.
The proposed IoU-oriented optimization improves accuracy, while 3D-GIoU with different weights is evaluated as a reference.
The metric is AP40 for the Car category under the moderate setting.

D. Performance for the Close Objects

Performance degrades for very close objects because they are harder to localize and are less adequately represented in the training data. Larger center misalignment and truncation further compound this difficulty.

Very close objects perform poorly partly because their 2D-box centers have larger misalignment with projected 3D centers.
Closer objects are usually more truncated, increasing the difficulty of detecting them.
The close-range training distribution is limited, with 5,979 cars in [5m, 15m] and 6,707 cars in [10m, 20m].After excluding samples tagged unknown, the effective counts are 4,522 and 6,149, respectively.

E.1. Learned features

The learned-feature comparison evaluates coarse center detection supervised by either the 2D center or the projected 3D center. Projected-3D-center supervision produces more accurate coarse centers and improves localization accuracy.

The qualitative results show that projected 3D center supervision makes the coarse center more accurate.
The improved coarse center from projected-3D-center supervision thereby improves localization accuracy.
Figure 7 compares coarse-center features supervised by the 2D center with features supervised by the projected 3D center.

E.2. Comparison of qualitative results

Qualitative comparisons on KITTI show the proposed method outperforming M3D-RPN in image-plane and 3D-world visualizations, while inaccurate depth produces representative false positives.

The proposed method performs better than M3D-RPN for dense objects in image-plane visualizations on the KITTI validation set.Red circles highlight the main differences in the paired images.
In 3D-world visualizations, the proposed model outputs better results than M3D-RPN, especially for orientation estimation.
Inaccurate depth creates a representative failure case: projected 3D boxes fit the object's image appearance but form a false positive in 3D world space.The object outline can be perceived through point clouds despite the inaccurate depth.