Source-linked AI summary
Learning Depth-Guided Convolutions for Monocular 3D Object Detection
Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, Ping Luo
TL;DR
Monocular 3D detection lacks accurate depth, while conventional 2D convolutions miss local structure and scale, and pseudo-LiDAR depends on estimated-depth precision. The paper proposes D4LCN, which learns depth-guided dynamic filters and receptive fields for local image features. D4LCN ranks first on KITTI and achieves a 9.1% relative improvement over the second-best competitor under the moderate setting.
Problem
Monocular 3D detection lacks accurate depth; 2D convolutions miss local object structure and scale, while pseudo-LiDAR performance depends heavily on estimated-depth precision.
Method
D4LCN generates sample-wise, position-wise, depthwise local convolution kernels and adaptive dilation rates from depth maps within a single-stage 3D detection framework.
Results
9.1% relative improvement over the second-best monocular 3D car detector on KITTI under the moderate setting, with D4LCN ranking first.
Takeaways & Limitations
D4LCN narrows the gap between 2D convolutional features and point-cloud-based 3D operations while retaining high-level RGB semantic information.
Takeaways & Limitations
The calibrated formulation assumes per-image camera intrinsics are available during both training and testing.
Abstract
from arXiv · showhide
3D object detection from a single image without LiDAR is a challenging task due to the lack of accurate depth information. Conventional 2D convolutions are unsuitable for this task because they fail to capture local object and its scale information, which are vital for 3D object detection. To better represent 3D structure, prior arts typically transform depth maps estimated from 2D images into a pseudo-LiDAR representation, and then apply existing 3D point-cloud based object detectors. However, their results depend heavily on the accuracy of the estimated depth maps, resulting in suboptimal performance. In this work, instead of using pseudo-LiDAR representation, we improve the fundamental 2D fully convolutions by proposing a new local convolutional network (LCN), termed Depth-guided Dynamic-Depthwise-Dilated LCN (D$^4$LCN), where the filters and their receptive fields can be automatically learned from image-based depth maps, making different pixels of different images have different filters. D$^4$LCN overcomes the limitation of conventional 2D convolutions and narrows the gap between image representation and 3D point cloud representation. Extensive experiments show that D$^4$LCN outperforms existing works by large margins. For example, the relative improvement of D$^4$LCN against the state-of-the-art on KITTI is 9.1\% in the moderate setting. The code is available at https://github.com/dingmyu/D4LCN.
1. Introduction
Monocular 3D detection seeks a cheaper alternative to LiDAR, but image-based convolutions miss local 3D structure and scale while pseudo-LiDAR depends heavily on estimated-depth accuracy. D4LCN addresses these issues with depth-guided, dynamic local convolutions that retain RGB semantics.
- Monocular cameras offer a cheaper alternative to LiDAR for 3D object detection, but the problem remains largely unsolved.
- Traditional 2D convolutions struggle with objects at different scales because perspective projection changes apparent object size with distance.
- Camera-plane neighborhoods lose metric depth, causing 2D filters to treat object regions and background areas similarly.
- Pseudo-LiDAR methods depend heavily on estimated-depth precision, and coarse depth can produce wrong coordinates and inaccurate 3D predictions.
- D4LCN generates sample-wise, position-wise, depthwise kernels from depth maps, with different dilation rates providing adaptive receptive fields.
- D4LCN combines depth-guided local filtering with RGB semantics to narrow the gap between 2D convolutional and 3D point-cloud representations.
2. Related Work
Prior monocular 3D detection methods use geometric constraints, pseudo-LiDAR representations, or dynamic filtering, but each approach has limitations in recovering or efficiently processing 3D structure. The paper motivates depth-guided feature learning as a bridge between image and 3D representations.
- Image-based Monocular 3D Detection: Image-based methods impose scene-geometry constraints, such as camera projection, orthographic transforms, and uncertainty modeling, to train 2D-to-3D mappings.
- Image-based Monocular 3D Detection: Geometric constraints do not reliably restore accurate object 3D information from a single monocular image.
- The paper uses depth information to guide 2D-to-3D feature representation because depth bridges the gap between image features and 3D structure.
- Point Cloud-based Monocular 3D Detection: Pseudo-LiDAR methods convert estimated depth maps into point clouds so existing LiDAR-based detectors can process monocular inputs.
- LiDAR-based 3D Detection: LiDAR-based detection benefits from point-based and voxel-based feature learning, while multi-sensor methods combine geometric point clouds with RGB features.
- Dynamic Networks: Existing dynamic filtering methods use spatially or sample-specific kernels, but fixed spatial divisions, computational cost, and scale handling remain concerns.
3. Methodology
D4LCN is a single-stage monocular 3D detector that uses estimated depth to generate dynamic local filters and adaptive receptive fields for RGB feature extraction. Its framework combines a two-branch backbone, depth-guided filtering, and a 2D-3D detection head with prior-based anchors.
- Framework Overview: The framework has a backbone, depth-guided filtering module, and 2D-3D detection head in a single-stage detector.The detection head uses Non-Maximum Suppression for prediction.
- Backbone: A two-branch backbone separately extracts RGB features and generates convolutional kernels from estimated depth before merging outputs at each block.The feature extraction branch uses ResNet-50, while the filter-generation branch uses the first three ResNet-50 blocks to reduce computation.
- Depth-Guided Filtering: D4LCN assigns sample-specific, position-specific kernels to local pixels and channels instead of applying global kernels across all images.This design targets scale variation and depth-lost spatial neighborhoods in conventional 2D convolutions.
- Depth-Guided Filtering: Depth-wise local convolution uses shifted feature maps and element-wise products to implement local filtering efficiently without explicitly materializing every location-specific kernel.The shift grid contains k·k offsets, such as nine directions when k = 3.
- Depth-Guided Filtering: Shift-pooling promotes information flow between depth-wise channels by shifting feature maps and averaging them with the original feature map.Compared with group convolution, it adds no parameters and keeps local-kernel weights at k × k × c_n.
- Detection Head: Adaptive dilation assigns different receptive-field sizes to filters, allowing the module to address diverse object scales while retaining RGB information.The detector uses calibrated per-image camera intrinsics and prior-based 2D-3D anchor boxes for prediction.
4. Experiments
Experiments on KITTI evaluate D4LCN against monocular 3D detection methods, depth-map qualities, convolutional alternatives, and ablated components. D4LCN ranks first overall, with gains linked to depth-guided filtering and adaptive dilation.
- Comparative Results: D4LCN ranks first among the compared monocular methods on KITTI and achieves a 9.1% relative improvement over the second-best competitor for moderate car 3D detection.The comparison reports 10.74 vs. 11.72 under the moderate setting and gives D4LCN 7 firsts and 2 seconds across 9 items.
- Comparative Results: D4LCN is trained end-to-end with a standard ImageNet-pretrained model while still achieving state-of-the-art 3D detection results.Many competitors use COCO/KITTI-pretrained detectors or multi-stage training, whereas D4LCN does not.
- Ablation Study: 3D detection performance improves as more model components are added, supporting contributions from Corner Loss, depth-guided filtering, shift-pooling, and adaptive dilation.The ablation compares 3DNet, +CL, +DLCN, +SP, and full D4LCN.
- Ablation Study: Adaptive dilated convolution contributes 2.69 and 1.76 points for AP|R11 and AP|R40, respectively, by assigning different receptive fields across feature-map channels.The best tested number of filters was nf = 3.
- Multi-Class 3D Detection: D4LCN performs best among the compared convolutional modules and achieves satisfactory pedestrian and cyclist detection, while pseudo-LiDAR methods fail on those categories.The convolution comparison includes Dynamic Convolution, Dynamic Local Filtering, and Deformable Convolution; Figure 5 illustrates scale-specific receptive fields.
5. Conclusion
D4LCN uses depth-conditioned kernels and receptive fields that vary across pixels, channels, and image samples for monocular 3D detection. The model addresses limitations of 2D convolutions, incorporates RGB semantics, and ranked first on KITTI at submission.
- D4LCN dynamically generates depth-conditioned convolutional kernels and receptive fields for different pixels, channels, and images.This design compensates for scale-sensitive and meaningless local structures in conventional 2D convolutions.
- The method combines depth guidance with RGB features to better capture 3D information without relying on point-cloud conversion.Its stated goal is to narrow the gap between 2D convolutional and point-cloud-based 3D operators.
- D4LCN ranked first for monocular 3D object detection on KITTI at the time of submission.
A. Definition of 3D Corners
The section defines each ground-truth box through its eight corners and specifies the rotation conventions used for those corners. Although the egocentric rotation matrix is introduced, allocentric pose is used for regression.
- Each ground-truth 3D box is represented using eight corners in a defined order.
- The corner index m ranges from 1 to 8, while ry denotes the egocentric rotation matrix.
- Allocentric pose is used for rotation regression rather than the egocentric pose.
B. Comparisons between Two Rotation Definitions
The section contrasts egocentric and allocentric pose definitions under viewpoint and translation changes. Allocentric pose preserves the object’s view, while the two poses remain convertible through the viewing angle.
- Egocentric pose changes viewpoint toward the camera when the object is translated.
- Allocentric pose maintains the same view independently of the object’s location.
- Allocentric and egocentric poses can be converted using the viewing angle θ.
- Cars with the same egocentric pose can be observed from different sides, whereas allocentric pose keeps the same view across corresponding observations.
C. Ablative Results for Convolutional Methods
The ablation decomposes depth-guided filtering from standard convolution through progressively more local, depth-wise, pooled, and adaptive variants. Results attribute the main improvement to adaptive dilation, which gives channels different receptive fields.
- The ablation compares traditional convolution, depth-guided convolution, local convolution, depth-wise local convolution, shift-pooled depth-wise convolution, and full D4LCN.
- Using depth maps to guide convolution at each pixel brings a considerable improvement.
- Depth-wise convolution with shift-pooling uses fewer parameters and outperforms standard convolution.
- The main improvement comes from adaptive dilated convolution, which assigns different receptive fields to feature-map channels.
D. Comparisons of Labeling Information and Training Strategies
The comparison examines labeling information and training strategies, finding that the model outperforms existing methods using only monocular depth and supports end-to-end training.
- Table 6 compares methods by labeling information and training strategy, including pretraining, multi-stage training, end-to-end training, and depth supervision.The table defines Depth as the depth map extracted from a monocular image.
- Using only the depth map extracted from a monocular image, the model outperforms all existing methods.
- The model can be trained in an end-to-end manner.
E. Distributions of Different Dilation
The model adjusts dilation rates across blocks, using larger receptive fields early and distributing dilation rates across scales in the final block.
- The first block increases its receptive field with large dilation rates before the second block uses smaller receptive fields.
- The third block uses three dilation rates evenly to handle objects at different scales.