Source-linked AI summary
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships
Yongjian Chen, Lei Tai, Kai Sun, Mingyang Li
TL;DR
Monocular 3D detection lacks direct depth information and is especially difficult for partially occluded objects, which conventional object-independent targets do not adequately address. MonoPair models adjacent-object relationships with uncertainty-aware location and distance predictions, jointly optimized through spatial constraints, and achieves state-of-the-art KITTI performance, especially for hard samples.
Problem
Monocular 3D detection is difficult for partially occluded objects because visible information is limited and most detectors treat objects independently.
Method
MonoPair predicts uncertainty-aware object locations and pairwise 3D distances, then jointly optimizes adjacent-object spatial constraints through a post-optimization module.
Results
MonoPair achieves state-of-the-art performance on the KITTI 3D detection benchmark, with particularly large improvements for hard, heavily occluded samples.
Takeaways & Limitations
Pairwise geometric relationships provide useful context for monocular 3D detection, including occluded-object detection, without additional information.
Takeaways & Limitations
The method does not consider the 3D distance component invariant across viewing angles and instead uses local-coordinate distance as the meaningful target.
Abstract
from arXiv · showhide
Monocular 3D object detection is an essential component in autonomous driving while challenging to solve, especially for those occluded samples which are only partially visible. Most detectors consider each 3D object as an independent training target, inevitably resulting in a lack of useful information for occluded samples. To this end, we propose a novel method to improve the monocular 3D object detection by considering the relationship of paired samples. This allows us to encode spatial constraints for partially-occluded objects from their adjacent neighbors. Specifically, the proposed detector computes uncertainty-aware predictions for object locations and 3D distances for the adjacent object pairs, which are subsequently jointly optimized by nonlinear least squares. Finally, the one-stage uncertainty-aware prediction structure and the post-optimization module are dedicatedly integrated for ensuring the run-time efficiency. Experiments demonstrate that our method yields the best performance on KITTI 3D detection benchmark, by outperforming state-of-the-art competitors by wide margins, especially for the hard samples.
1. Introduction
Monocular 3D detection is difficult for heavily occluded objects because their visible information is limited. MonoPair addresses this gap by modeling relationships between paired objects with uncertainty-aware spatial constraints and reports strong KITTI performance, especially on hard samples.
- Monocular 3D detection is challenging because heavily occluded objects provide limited useful visual information.
- MonoPair captures spatial relationships between paired objects instead of treating each object as an independent target.
- The detector jointly predicts object locations and object-to-object spatial constraints using uncertainty-aware outputs.
- The pairwise spatial constraint encodes geometric information for occluded objects and improves their detection accuracy.
- MonoPair achieves the best performance on the KITTI 3D detection benchmark, outperforming state-of-the-art competitors by wide margins, particularly on hard samples.
2. Related Work
Prior monocular 3D detectors infer depth from RGB images using geometric priors, transformed features, or specialized losses, while related work also studies object relationships and uncertainty. MonoPair differs by using uncertainty to weight global post-optimization alongside pairwise spatial constraints.
- Monocular 3D Object Detection: Monocular 3D detection methods address limited 3D information using ground-plane assumptions, prior object shapes, projection geometry, transformed features, or geometric reasoning.
- Visual Relationship Detection: Visual relationship methods model dependencies between objects for image understanding, especially image captioning and scene generation.
- Uncertainty Estimation in Object Detection: Uncertainty estimation has been used to improve training quality or provide additional references in detection and localization.
- MonoPair Architecture: The architecture divides eleven prediction branches into 2D detection, 3D detection, and pair constraint prediction.
- Uncertainty Estimation in Object Detection: MonoPair uses uncertainty to weight the post-optimization cost function, integrating detection estimates and predicted uncertainties in a global context.
3. Approach
The approach combines one-stage monocular 3D detection with pairwise geometric constraints and uncertainty-aware nonlinear optimization. Object locations and inter-object distances are predicted from feature-map keypoints, then jointly refined for paired objects.
- 3.2. 2D Detection: The one-stage architecture uses a backbone and task-specific dense prediction branches for monocular image input.The backbone outputs a feature map, followed by eleven output branches divided among detection tasks.
- 3.2. 2D Detection: The detector predicts 2D keypoints, bounding-box size, center offsets, 3D center offsets, depth, dimensions, and local orientation.Depth is predicted through inverse depth, dimensions in meters, and orientation with eight scalars trained using MultiBin loss.
- 3.4. Pairwise Spatial Constraint: Adjacent object pairs are selected using a range circle whose diameter is the distance between their 2D bounding-box centers; pairs containing other centers are discarded.The resulting effective pairs define the pairwise regression targets during training and inference.
- 3.4. Pairwise Spatial Constraint: Each pairwise constraint uses a feature-map keypoint at the midpoint of paired 2D centers and regresses their 3D absolute distance in local coordinates.The camera-coordinate distance is not used because it is invariant across viewing angles, whereas local-coordinate distance better reflects appearance changes.
- 3.5. Uncertainty: Aleatoric uncertainty branches for depth, 3D center offset, and pairwise distance weight the errors, giving lower-uncertainty predictions greater influence.Objects not connected to another vertex are not updated during post-optimization.
- 3.6. Spatial Constraint Optimization: The post-optimization module represents objects as graph vertices and pair constraints as edges, then solves a nonlinear least-squares problem over projected centers and depths.The error vector combines object-location and pairwise-constraint terms, while the weight matrix has dimension 3N_G + 3M.
4. Implementation
The implementation evaluates MonoPair on KITTI3D using defined data splits, AP40 metrics, and three progressively enhanced experimental setups.
- KITTI3D is split into 3712 training samples and 3769 validation samples, with Easy, Moderate, and Hard labels based on truncation, occlusion, and bounding-box height.
- Training uses an end-to-end 70-epoch Adam schedule with batch size 32 across four GPUs and augmentation for 2D, 3D, and pairwise predictions.The learning rate starts at 1.25e-4 and is multiplied by 0.1 at epochs 45 and 60.
- Evaluation reports AP40 for bird-eye-view and 3D bounding boxes at IoU thresholds of 0.5 and 0.7.AP40 averages precision over 40 recall positions, replacing the earlier AP11 metric.
- The ablation compares a CenterNet-derived Baseline, uncertainty-augmented +σz + σuv, and final MonoPair with eleven branches and pairwise constraint optimization.
- Results are tabulated for car validation and test sets, plus pedestrian and cyclist test samples at the 0.7 IoU threshold.
5. Experimental Results
MonoPair improves monocular 3D detection on KITTI, particularly for heavily occluded and hard samples, while retaining real-time inference. Ablations indicate that uncertainty modeling and carefully selected pairwise constraints are important to its performance.
- Quantitative Results: MonoPair outperforms competing monocular 3D detectors, especially on hard samples with heavier occlusions.The gains are attributed to uncertainty and pairwise spatial constraints.
- Quantitative Results: MonoPair achieves the highest score for Moderate samples and ranks first among monocular detectors without additional information.It outperforms M3D-RPN by a large margin except for Easy APbv and AP3D results.
- Quantitative Results: Pedestrian and cyclist results show significant improvement over M3D-RPN despite relatively few training samples.The pairwise constraint uses object relationships more deeply than target-independent detectors.
- Efficiency: The one-stage anchor-free detector runs in real time at 57 ms per image and is more than two times faster than region-proposal-based methods.This comparison is reported on an Nvidia GTX 1080 Ti.
- Ablation Study: Uncertainty prediction for both depth and projected offset improves over the baseline, with depth uncertainty producing the larger gain.The result supports uncertainty modeling because depth prediction is more challenging.
- Ablation Study: Alternative weight-matrix strategies did not improve detection, and coarse post-optimization can make the baseline worse.The findings indicate that selecting optimization weights is nontrivial.
6. Conclusions
The paper concludes that uncertainty-aware training and spatial relationships between object pairs enable state-of-the-art monocular 3D detection on KITTI without additional information. It identifies cross-category relationships and improved pair matching as future directions.
- Conclusions: MonoPair combines aleatoric uncertainty with object-pair spatial relationships in a monocular 3D detection post-optimization method.The method achieves state-of-the-art KITTI performance without additional information.
- Conclusions: Exploring spatial constraints between object pairs reveals potential for geometric relationships in 3D object detection.The paper notes that such relationships were rarely considered previously.
- Conclusions: Future work includes finding spatial relationships across object categories and developing new pair-matching strategies.