Source-linked AI summary
RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving
Peixuan Li, Huaici Zhao, Pengfei Liu, Feidao Cao
TL;DR
Monocular 3D detection is important for autonomous driving but is constrained by the cost of LiDAR and the sensitivity of 2D-box-based methods to detector errors. RTM3D predicts nine projected 3D-box keypoints and recovers object properties through geometric constraints in a one-stage framework. The authors report real-time image-only detection with state-of-the-art KITTI performance.
Problem
Monocular 3D detection must recover 3D information from images without costly LiDAR, while 2D-box methods offer only four constraints and are sensitive to small detector errors.
Method
RTM3D uses a one-stage multi-scale network to predict nine projected keypoints and combines them with perspective geometry and an energy function to recover 3D properties.
Results
The authors report the first real-time image-only monocular 3D detector and state-of-the-art performance on KITTI.
Takeaways & Limitations
Keypoint-based geometric recovery provides a stable 3D bounding box without stand-alone networks or additional annotation while maintaining real-time speed.
Abstract
from arXiv · showhide
In this work, we propose an efficient and accurate monocular 3D detection framework in single shot. Most successful 3D detectors take the projection constraint from the 3D bounding box to the 2D box as an important component. Four edges of a 2D box provide only four constraints and the performance deteriorates dramatically with the small error of the 2D detector. Different from these approaches, our method predicts the nine perspective keypoints of a 3D bounding box in image space, and then utilize the geometric relationship of 3D and 2D perspectives to recover the dimension, location, and orientation in 3D space. In this method, the properties of the object can be predicted stably even when the estimation of keypoints is very noisy, which enables us to obtain fast detection speed with a small architecture. Training our method only uses the 3D properties of the object without the need for external networks or supervision data. Our method is the first real-time system for monocular image 3D detection while achieves state-of-the-art performance on the KITTI benchmark. Code will be released at https://github.com/Banconxuan/RTM3D.
1. Introduction
The paper targets monocular 3D detection for autonomous driving, avoiding costly LiDAR and the speed limitations of methods dependent on 2D detectors. It predicts nine projected 3D-box keypoints and combines them with geometric constraints in a one-stage, multi-scale framework.
- Motivation: Monocular cameras offer a cheaper, more convenient alternative to LiDAR for autonomous-driving 3D detection.3D object detection supports scene perception and motion prediction, while LiDAR systems are expensive and difficult to embed in current vehicle shapes.
- Motivation: Four 2D-box edges provide only four constraints, and small 2D-detector errors sharply reduce 3D detection performance.The possible vertex-to-edge correspondences can require 4,096 calculations, motivating alternatives to 2D-box-based recovery.
- Approach: The method predicts nine projected keypoints—eight vertices and the 3D-box center—to provide 18 geometric constraints for recovery.The keypoints are grouped and ordered using relationships between the vertices and center, avoiding reliance on a separate 2D detector.
- Approach: The framework jointly optimizes prior and 3D-object information through an overall energy function.Its geometric formulation combines these constraints to generate 3D-object properties efficiently and accurately.
- Results: On KITTI, the authors report the first real-time image-only 3D detector with better accuracy at the same running time than competing methods.The claimed system uses images alone and is designed for real-time operation.
2. Related Work
Related work contrasts LiDAR-based and image-based 3D detection, emphasizing the extra data or networks often used to compensate for missing monocular depth. The paper instead predicts projected keypoints and minimizes reprojection error with geometric constraints.
- Image-based detection: Image-based 3D detection is attractive because cameras are cheaper and more convenient, but monocular input makes reliable 3D information difficult to obtain.Prior approaches supplement images with instance segmentation, stereo, wire-frame models, CAD priors, depth, or other data and networks.
- Comparison: Table 1 compares image-based approaches by real-time status and requirements for additional data.The comparison concerns whether methods need supplementary supervision or processing beyond image input.
- Prior methods: Earlier monocular methods commonly combine geometric constraints with 2D detectors to describe 3D object information.One example estimates dimensions and orientation from features extracted from a 2D box, then recovers location through perspective constraints.
- Proposed direction: This paper predicts nine 2D keypoints projected by the 3D-box vertices and center, then minimizes reprojection error to obtain an optimal result.The approach replaces direct dependence on 2D box edges with keypoint-based geometric recovery.
3. Proposed Method
The proposed one-stage network predicts nine projected 3D-box keypoints and optional priors from RGB images, then estimates 3D boxes by minimizing reprojection and prior errors. Its multi-scale design targets fast, accurate keypoint detection, while geometric optimization recovers box dimensions, orientation, and location.
- Architecture: The RGB-only architecture uses a backbone, keypoint feature pyramid, and detection head in a one-stage design for fast inference.The detection head has three fundamental components and six optional components that can improve accuracy with little computational cost.
- Architecture: The base module predicts a main-center heatmap, vertex heatmap, and vertex coordinates to estimate the 3D bounding box.Optional priors can further improve 3D detection performance.
- Keypoint feature pyramid: The keypoint feature pyramid resizes scale-specific features and combines them through softmax-derived weights into a scale-space score map.This multi-scale aggregation is intended to provide accurate projected points for objects at different scales.
- Prediction targets: The network regresses object dimensions, orientation, depth, center offsets, and vertex offsets alongside keypoint heatmaps.Orientation uses a two-bin multi-bin formulation, while depth is predicted as an additional prior.
- 3D bounding-box estimation: The 3D box is estimated by minimizing reprojection error together with dimension and orientation prior errors in a nonlinear least-squares problem.Keypoint heatmap confidence supplies the projection-error covariance, and detected priors initialize Gauss–Newton or Levenberg–Marquardt optimization.
4. Experimental
Experiments on KITTI evaluate RTM3D’s accuracy, speed, qualitative robustness, and component effects. The method achieves strong image-only 3D detection while avoiding dependence on external networks or data.
- Evaluation Setup: The KITTI evaluation reports AP3D, APBEV, and AOS across easy, moderate, and hard difficulty settings.The difficulty levels reflect object occlusion, truncation, and image-space height.
- Comparison with Other Methods: ResNet-18 is more than 100 times faster than Mono3D while exceeding it by over 10% on both APBEV and AP3D across datasets.The comparison uses image-based methods evaluated on KITTI.
- Comparison with Other Methods: DLA-34 is about three times faster than M3D-RPN while improving most metrics, and it achieves the best accuracy among the reported RTM3D backbones.The authors note that comparisons are not fully equivalent because many competing methods use extra networks or data.
- Qualitative Results: Qualitative projections show that the method handles crowded and truncated objects, while BEV results indicate accurate localization across different scenes.Figure 4 visualizes keypoints, projected 3D boxes, and BEV images, with ground truth in green and predictions in blue.
- Ablation Study: Combining dimension, orientation, distance, and keypoint-offset predictions gives the best accuracy and faster running speed in the optional-component study.The network outputs initialize geometric optimization, reducing the gradient-descent search space.
- Ablation Study: Keypoint FPN improves AP3D and APBEV across all evaluated sets without significantly changing time consumption.The comparison uses models with and without KFPN on the KITTI validation data.
- 2D Detection and Orientation: RTM3D’s AP3D exceeds Deep3DBox by about 20% on moderate sets despite Deep3DBox using stronger 2D detectors.The authors attribute better 2D accuracy of projected 3D results to inferring occlusive object regions.
5. Conclusion
The conclusion presents RTM3D as a fast monocular 3D detector that reformulates detection around keypoints and geometric constraints. Its network predicts 3D-box keypoints and object priors from images, while geometry produces stable 3D boxes.
- Conclusion: RTM3D reformulates monocular 3D detection as keypoint detection and recovers 3D bounding boxes using geometric constraints.The framework is designed for autonomous-driving scenarios.
- Conclusion: The customized point-detection network predicts 3D-box keypoints and other object priors using only images.These predictions supply the geometry module with information for 3D-box recovery.
- Conclusion: The geometry module converts the predicted priors into easy-to-optimize loss functions and produces stable, accurate 3D bounding boxes.