Source-linked AI summary
Objects are Different: Flexible Monocular 3D Object Detection
Yunpeng Zhang, Jiwen Lu, Jie Zhou
TL;DR
Monocular 3D object detection avoids the implementation costs of LiDAR and stereo cameras, but unified processing struggles with heavily truncated objects. The paper decouples truncated-object prediction and uses uncertainty-guided depth ensembles, significantly outperforming existing competitors on KITTI, including methods using extra information.
Problem
Monocular 3D object detection seeks precise 3D localization from images without LiDAR or stereo depth, while unified processing performs poorly on heavily truncated objects.
Method
The framework decouples truncated-object prediction and combines directly regressed depth with geometry-based keypoint depths using uncertainty-guided weights.
Results
The method significantly outperforms all existing competitors on the KITTI benchmark, with especially strong improvements for moderate and hard samples.
Takeaways & Limitations
Flexible processing improves detection of ordinary and extremely truncated objects in street scenes, including cases important for autonomous-driving safety.
Abstract
from arXiv · showhide
The precise localization of 3D objects from a single image without depth information is a highly challenging problem. Most existing methods adopt the same approach for all objects regardless of their diverse distributions, leading to limited performance for truncated objects. In this paper, we propose a flexible framework for monocular 3D object detection which explicitly decouples the truncated objects and adaptively combines multiple approaches for object depth estimation. Specifically, we decouple the edge of the feature map for predicting long-tail truncated objects so that the optimization of normal objects is not influenced. Furthermore, we formulate the object depth estimation as an uncertainty-guided ensemble of directly regressed object depth and solved depths from different groups of keypoints. Experiments demonstrate that our method outperforms the state-of-the-art method by relatively 27\% for the moderate level and 30\% for the hard level in the test set of KITTI benchmark while maintaining real-time efficiency. Code will be available at \url{https://github.com/zhangyp15/MonoFlex}.
1. Introduction
Monocular 3D object detection avoids the cost of LiDAR and stereo cameras but must localize objects from a single image. The paper addresses failures caused by uniformly processing normal and heavily truncated objects by decoupling their predictions and combining multiple depth estimators.
- Monocular 3D object detection uses only image information, avoiding the implementation costs associated with LiDAR or stereo depth estimation.
- Unified processing can miss heavily truncated objects and increase the learning burden that affects predictions for general objects.
- Inside objects are represented by projected 3D centers, while outside objects use selected edge points and separate feature-map regions.
- The edge fusion module further decouples feature learning and prediction for objects outside the image.
- The framework combines direct depth regression with geometric depths from groups of keypoints using uncertainty-guided weighting.
2. Related Work
Prior monocular 3D detectors use geometric, proposal-based, direct-regression, and region-proposal strategies, while related work also studies uncertainty and ensemble learning. The proposed framework extends this landscape with multiple prediction branches and uncertainty-guided depth combination.
- Monocular 3D object Detection: Some monocular 3D detection methods rely on extra information such as pretrained depth modules, annotated keypoints, or CAD models.
- Monocular 3D object Detection: Prior monocular methods include geometric constraints, proposal fitting, direct depth prediction, and depth-aware convolutional region proposals.
- Uncertainty Estimation: Uncertainty-estimation research distinguishes epistemic parameter uncertainty from aleatoric observation noise and applies these concepts to object detection.
- Ensemble Learning: Ensemble learning combines multiple learners through strategies such as bagging, boosting, stacking, and gating networks.
- The proposed framework uses multiple prediction branches and combines regressed depth with keypoint- and dimension-based depths through uncertainty guidance.
3. Approach
MonoFlex decomposes monocular 3D detection into visual-property regression and flexible localization and depth estimation. It decouples inside and truncated objects, then combines direct and keypoint-based depth estimates using uncertainty.
- Framework Overview: 3D location is represented by the projected 3D center xc and object depth z, while dimensions and orientation are inferred from appearance.The framework decomposes the overall problem into four independent subtasks.
- Visual Properties Regression: The network regresses 2D boxes, dimensions, orientations, keypoints, and depth through multiple branches on a shared backbone.The final depth combines regressed depth with depths computed from estimated keypoints and dimensions.
- Decoupled Representations: Objects are split by whether their projected 3D centers lie inside or outside the image, using separate representative-point formulations.Inside objects use xc; outside objects use an image-edge intersection xI between xb and xc.
- Decoupled Representations: The edge fusion module further decouples feature learning for outside objects by extracting and processing the feature-map boundaries.Boundary features are concatenated and processed with one-dimensional convolutions for spatial-variant predictions.
- Visual Properties Regression: Ten keypoints comprise projected 3D-box vertices and top and bottom centers, supporting orientation and geometric depth estimation.Keypoint offsets are regressed from the representative point, and vertical supporting lines provide independent depth estimates.
- Adaptive Depth Ensemble: Depth is estimated by an adaptive ensemble of direct regression and independent geometric solutions from three keypoint groups, weighted by uncertainty.The direct estimator uses an inverse-sigmoid transformation, while keypoint depths use pixel height, object height, and camera geometry.
4. Experiments
The method is evaluated on the KITTI 3D Object Detection benchmark using predefined training and validation splits, with results reported across three difficulty levels.
- KITTI contains 7,481 training images and 7,518 testing images, with the training images split into 3,712 train and 3,769 validation images.
- Results are reported for easy, moderate, and hard difficulty levels defined by bounding-box height, occlusion, and truncation.Models jointly detect Car, Pedestrian, and Cyclist.
4.1. Implementation Details
The implementation uses a modified DLA-34 backbone with standardized input size and lightweight prediction heads, trained with AdamW under a scheduled learning rate.
- The model uses a modified DLA-34 backbone and pads input images to 384 × 1280.
- Each prediction head contains convolution, BatchNorm, and ReLU layers followed by a 1 × 1 convolution, while edge fusion uses a 1D convolution without ReLU.
- Training uses AdamW with learning rate 3e-4, weight decay 1e-5, 34k iterations, batch size 7, and learning-rate reductions at 22k and 30k iterations.Random horizontal flipping is the only data augmentation.
4.2. Evaluation Metrics
Evaluation uses average precision for 3D bounding boxes, with different reporting protocols for validation and test sets and class-specific IoU thresholds.
- 3D detection performance is measured using average precision for 3D bounding boxes, AP3D.
- Validation results include AP3D|R11 and AP3D|R40, while test results report AP3D|R40 from the test server.
- The AP3D IoU threshold is 0.7 for Car and 0.5 for Pedestrian and Cyclist.
4.3. Quantitative Results
On KITTI, the proposed monocular framework outperforms prior methods for Car, improves difficult cases, remains competitive across other classes, and supports real-time inference.
- 45%, 42% and 42% higher AP3D|R40 validation results are achieved than MonoPair on the easy, moderate, and hard levels, respectively.The comparison is for Car on the KITTI validation set.
- The method surpasses all existing methods on the KITTI Car test set, including methods using extra information.The significant improvement on hard samples is reported as evidence of improved detection of heavily truncated objects.
- For Pedestrian and Cyclist on the test set, the method outperforms M3D-RPN and Movi3D while achieving comparable performance with MonoPair.
- The framework is much faster than most existing methods, achieving a state-of-the-art trade-off between performance and latency.
4.4. Ablation Study
The ablations show that decoupled representations improve truncated-object handling, while uncertainty-guided depth combination strengthens both direct and keypoint-based estimation.
- Decoupled Representations: Discarding outside objects improves performance over the baseline, supporting separate treatment of truncated objects.
- Decoupled Representations: Projected 3D centers represent inside objects better than 2D centers because offsets from 2D centers are irregular and difficult to learn.
- Decoupled Representations: Decoupled offset losses and edge fusion are crucial for improving moderate and hard samples containing heavily truncated objects.
- Object Depth Estimation: Uncertainty estimation improves both direct regression and keypoint-based depth estimation by downweighting difficult outliers.
- Object Depth Estimation: The adaptive depth ensemble outperforms each individual depth estimator by combining direct and keypoint-based predictions with uncertainty-guided weights.
4.5. Depth Combination
The depth-combination analysis compares individual estimators with soft and hard ensembles, finding that soft uncertainty-guided fusion is generally strongest but hard fusion can be sensitive to uncertainty errors.
- Depth Combination: Soft ensemble depth outperforms every individual estimator, especially for moderate Car and all Pedestrian difficulty levels.
- Depth Combination: Hard ensemble depth is inferior, possibly because it is sensitive to mismatch between actual depth error and estimated uncertainty.
- Depth Combination: Soft ensemble performance is close to the oracle upper bound for Pedestrian, while Car oracle results indicate further combination potential.
4.6. Qualitative Results
Qualitative KITTI results show strong detection of ordinary objects and successful detection of some extremely truncated objects, supporting the framework’s flexible handling.
- Qualitative Results: The framework produces superior performance for ordinary objects across varied street scenes.
- Qualitative Results: The framework successfully detects some extremely truncated objects highlighted in the qualitative results.
5. Conclusion
The paper proposes a flexible monocular 3D detection framework that decouples truncated objects and combines depth estimators using uncertainty. Experiments on KITTI show improved performance over existing competitors.
- Conclusion: The framework decouples truncated objects with an edge heatmap and edge fusion module to handle different object distributions.
- Conclusion: Object depth estimation uses an uncertainty-guided ensemble of multiple approaches for more robust and accurate predictions.
- Conclusion: Experiments on the KITTI benchmark show that the method significantly outperforms all existing competitors.