Source-linked AI summary
Deep Fitting Degree Scoring Network for Monocular 3D Object Detection
Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, Jie Zhou
TL;DR
Monocular 3D detection is challenging because depth is unavailable and tight-constraint localization depends on 2D detection accuracy. The paper regresses dimensions and orientation, then uses FQNet to score projected 3D proposals by estimated 3D IoU; on KITTI, it reports strong performance among monocular methods, including first place on full 3D detection, while training is limited to Cars.
Problem
Monocular 3D detection is ill-conditioned without depth, and tight-constraint localization does not use appearance cues and depends strongly on 2D detection accuracy.
Method
The method regresses dimension and orientation with anchor-based regression, densely samples 3D candidates, projects them into the image, and uses FQNet to estimate their 3D IoU.
Results
The method ranks first among pure monocular methods for full 3D object detection and outperforms stereo-based 3DOP at a 3D IoU threshold of 0.7.
Takeaways & Limitations
Projected-box fitting provides a high-precision monocular localization strategy supported by quantitative and qualitative results on KITTI.
Takeaways & Limitations
Experiments were conducted only on Cars because there was insufficient data to train the model for Pedestrians and Cyclists.
Abstract
from arXiv · showhide
In this paper, we propose to learn a deep fitting degree scoring network for monocular 3D object detection, which aims to score fitting degree between proposals and object conclusively. Different from most existing monocular frameworks which use tight constraint to get 3D location, our approach achieves high-precision localization through measuring the visual fitting degree between the projected 3D proposals and the object. We first regress the dimension and orientation of the object using an anchor-based method so that a suitable 3D proposal can be constructed. We propose FQNet, which can infer the 3D IoU between the 3D proposals and the object solely based on 2D cues. Therefore, during the detection process, we sample a large number of candidates in the 3D space and project these 3D bounding boxes on 2D image individually. The best candidate can be picked out by simply exploring the spatial overlap between proposals and the object, in the form of the output 3D IoU score of FQNet. Experiments on the KITTI dataset demonstrate the effectiveness of our framework.
1. Introduction
Monocular 3D object detection remains difficult because depth is unavailable in a single image, making location estimation ill-conditioned. The proposed method addresses this by scoring how well projected 3D proposals fit the object rather than relying only on tight 2D-box constraints.
- The task requires estimating object dimension, orientation, and location, but monocular appearance provides weaker cues for location than for the other properties.Objects with identical pose can appear substantially similar at different distances.
- Monocular 3D detection is important for applications lacking affordable or available depth and point-cloud sensors.The setting is relevant to web images, mobile applications, gastroscopy, and scenarios where other sensors may fail.
- Existing monocular approaches remain relatively underexplored and often address only subproblems such as orientation estimation.The passage contrasts monocular methods with stereo-, RGBD-, and point-cloud-based detection.
- Tight constraint estimates location by placing a 3D proposal compactly inside the 2D bounding box.This approach does not use image-appearance cues and depends strongly on 2D detection accuracy.
- The proposed pipeline regresses dimension and orientation, then uses FQNet to infer proposal–object 3D IoU from projected 2D information.Dense candidate evaluation is based on the spatial relation between projected 3D proposals and the object.
2. Related Work
Prior monocular 3D detection work includes handcrafted and deep-learning approaches, while box-refinement methods improve localization accuracy. This paper relates to refinement work but retains 3D bounding boxes and learns projected-box patterns from 2D images.
- Monocular 3D object detection is difficult because mapping between 2D images and 3D geometry is ambiguous.
- Handcrafted approaches: Early handcrafted approaches designed features such as contours, deformable 3D cuboids, viewpoint information, and part-level 3D geometry.
- Deep learning based approaches: Deep-learning approaches use end-to-end training and labeled data for proposal generation, orientation regression, tight-constraint translation, or full 3D shape and pose prediction.
- Box Refinement Techniques: Box-refinement techniques focus on improving localization accuracy through bounding-box regression, boundary probabilities, corner localization, or stronger geometric representations.
- Box Refinement Techniques: Unlike methods using detailed geometric models, this work retains 3D bounding boxes and learns patterns of their projections on 2D images.
3. Approach
The approach estimates object dimensions and local orientation, constructs 3D proposals, and evaluates densely sampled locations through projected-box fitting. FQNet predicts proposal–object 3D IoU from 2D projection patterns to select the best location candidate.
- Regression Module: The regression module uses cropped 2D detections to estimate each object's dimension and orientation before constructing a 3D cuboid.Its fully connected architecture separates dimension and orientation regression.
- Dimension Estimation: Anchor cuboids represent clustered object dimensions, with the highest-confidence anchor refined by predicted offsets.The dimension output contains confidence and three-dimensional offsets for each anchor cuboid.
- Orientation Estimation: Anchor-based regression reduces continuous prediction difficulty by selecting an anchor first and then regressing a smaller offset.The orientation branch similarly clusters angles and predicts confidence and offsets, focusing on local orientation in camera coordinates.
- Location Estimation: The method obtains a tight-constraint seed candidate, then generates dense samples by perturbing its 3D location within a small range.The generated samples retain the estimated dimensions and orientation while varying location offsets.
- FQNet: FQNet evaluates each sample by learning projected-box alignment patterns and regressing the 3D IoU between the proposal and object.Projected vertices are computed through camera projection and drawn on the 2D image so the network can learn spatial relations from 2D cues.
4. Experiments
Experiments on KITTI evaluate the method across orientation, dimension, location, robustness, and qualitative detection performance. The approach is reported to be robust to 2D detection quality, accurate in dimension estimation, and competitive with monocular and stereo methods.
- Experimental Setup: The method is evaluated on KITTI, primarily for the Car category because insufficient data prevented training on Pedestrian and Cyclist.The dataset includes 7481 training images and 7518 testing images, with annotations for orientation, 2D location, dimension, 3D location, and global orientation.
- Effectiveness: The approach is more robust than the tight-constraint baseline to changes in 2D detection Average Precision.The comparison varies the 2D detection results through jittering and measures resulting 3D detection performance.
- Effectiveness: FQNet’s average 3D IoU estimation error is lowest for proposals with 3D IoU around 0.4 to 0.5, at about 0.05.This evaluation supports FQNet’s ability to assess candidate proposals.
- Experimental Setup: The experiments compare the method with six state-of-the-art 3D object detection methods on two validation splits.The comparisons include 3DOP, Mono3D, 3DVP, SubCNN, Deep3DBox, and 3D-RCNN.
- Orientation and Dimension Evaluation: The method achieves state-of-the-art Average Orientation Similarity on both train/val settings, with especially significant gains on Easy and Moderate train/val 1 cases.Orientation is evaluated using KITTI’s official AOS metric.
- Orientation and Dimension Evaluation: The method obtains the lowest reported dimension estimation error, averaging about 0.15 meters against 3DOP, Mono3D, and Deep3DBox.The result is presented as evidence for the effectiveness of the anchor-based regression module.
- Comparison with State-of-the-Arts: The method outperforms Mono3D and Deep3DBox by about 3% in bird’s-eye-view AP and ranks first among pure monocular methods in full 3D AP.It also outperforms stereo-based 3DOP when the 3D IoU threshold is 0.7.
- Qualitative Results: Qualitative visualizations show that the approach fits objects well and produces high-precision 3D perception from a single monocular image.Detection results are visualized in both the 2D image and 3D space.
5. Conclusions
The paper proposes a unified monocular 3D object-detection pipeline that combines anchor-based dimension and orientation regression with dense 3D sampling and FQNet-based 3D IoU estimation. Quantitative and qualitative results are reported to outperform state-of-the-art monocular methods.
- The pipeline regresses object dimensions and orientations with an anchor-based method.
- It densely samples 3D space, projects candidates into the image, and uses FQNet to estimate 3D IoU for candidate filtering.
- Quantitative and qualitative results demonstrate performance superior to state-of-the-art monocular 3D object-detection methods.
- Extending the method to monocular 3D object tracking is identified as future work.
Supplementary Material
The supplementary material identifies the paper’s authors, provides contact information, and records the arXiv version date.
- The paper lists Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, and Jie Zhou as authors.
- The cited manuscript version is arXiv:1904.12681v2, dated 8 June 2019.
- The supplementary material provides Huawei contact addresses for the authors.
A. Global Orientation
Global orientation is computed from the regressed local orientation and the object’s ray direction, with the latter approximated from the object’s image-center displacement. The proportionality coefficient is learned from training data.
- Global orientation is required to construct the rotation matrix, while local orientation is regressed from the object’s appearance.
- The object ray direction is approximated as proportional to the distance between the object center and the image center.
- The object center is estimated from the 2D bounding-box boundaries, although the center of the eight projected corners is more precise.
- The proportionality coefficient k is regressed by minimizing an objective over N training samples.
- The fitted coefficient is k⋆ = 0.0012408.
B. Tight Constraint
The tight-constraint procedure obtains a seed 3D location by fitting a projected 3D box inside the detected 2D box. It evaluates possible vertex-to-boundary assignments and selects the feasible setting with the smallest residual.
- The method fixes the 3D box model and solves only for its location by fitting its projection tightly within the 2D bounding box.
- The location equations are rewritten as an overdetermined linear system Ax = b.
- Each of the four 2D-box boundaries is assigned a projected vertex, producing 4^8 = 4096 possible settings.
- For every setting, least squares estimates the location from four equations and three unknowns.
- Settings whose projections exceed the 2D box are discarded, and the remaining setting with the least residual is selected.
C. Tradeoffs of the System
Figure 4 compares accuracy and latency across model configurations on the KITTI validation set. The experiment also notes prior knowledge can reduce the candidate count to 64.
- Figure 4 compares accuracy against latency for different model configurations on the KITTI validation set.
- Prior knowledge that cars normally remain upright can reduce the candidate count to 64.
- Most cars in KITTI are directed toward the street, providing a dataset-specific directional assumption.
D. Contributions of Each Module
The system combines a regression module with FQNet, and the experiments show that regression quality strongly affects FQNet’s localization gains. Replacing regression outputs with ground-truth dimensions and orientations reaches 54.47% 3D AP with tight constraint.
- The proposed method combines a regression module (RM) with FQNet.
- FQNet improves 2% with Mean+Tight seeds but 8% with RM+Tight seeds.
- 54.47% 3D AP is achieved by GT+Tight when RM is replaced with ground-truth dimension and orientation.
E. More Implementation Details
Implementation uses MS-CNN detections filtered at a 0.1 score threshold, runs at about 0.33 frames per second, and produces visualizations from single monocular images without temporal information.
- MS-CNN supplies 2D bounding boxes, and detections scoring below 0.1 are discarded.
- The complete pipeline runs at about 0.33 frames per second on the reported CPU and GPU setup.
- The visualizations use single monocular images without temporal information from KITTI tracking sequences.