Source-linked AI summary
Categorical Depth Distribution Network for Monocular 3D Object Detection
Cody Reading, Ali Harakeh, Julia Chae, Steven L. Waslander
TL;DR
Monocular 3D detection needs reliable depth despite lacking direct range measurements, while existing approaches are limited by inaccurate or uncertainty-free depth and feature smearing. CaDDN predicts categorical depth distributions to project image features into 3D and BEV space, jointly training depth estimation and detection. It ranks first among published monocular methods on KITTI and reports monocular results on Waymo for the first time.
Problem
Monocular 3D detection must infer object depth from image cues without direct range measurements, while depth inaccuracies and feature smearing limit existing methods.
Method
CaDDN predicts pixel-wise categorical depth distributions, projects image features into frustum and voxel representations, collapses them to BEV, and jointly trains depth estimation with 3D detection.
Results
CaDDN ranks first among published monocular methods on KITTI Car and Pedestrian categories, with margins of 1.69% and 1.46% AP|R40 respectively, and reports first monocular Waymo results.
Takeaways & Limitations
Sharp categorical distributions and joint depth-detection training produce BEV representations that retain depth confidence and support strong monocular 3D detection.
Takeaways & Limitations
Effective frustum-to-voxel transformation requires similar spatial resolutions for the frustum and voxel grids; otherwise high-resolution voxels oversample low-resolution frustum features.
Abstract
from arXiv · showhide
Monocular 3D object detection is a key problem for autonomous vehicles, as it provides a solution with simple configuration compared to typical multi-sensor systems. The main challenge in monocular 3D detection lies in accurately predicting object depth, which must be inferred from object and scene cues due to the lack of direct range measurement. Many methods attempt to directly estimate depth to assist in 3D detection, but show limited performance as a result of depth inaccuracy. Our proposed solution, Categorical Depth Distribution Network (CaDDN), uses a predicted categorical depth distribution for each pixel to project rich contextual feature information to the appropriate depth interval in 3D space. We then use the computationally efficient bird's-eye-view projection and single-stage detector to produce the final output bounding boxes. We design CaDDN as a fully differentiable end-to-end approach for joint depth estimation and object detection. We validate our approach on the KITTI 3D object detection benchmark, where we rank 1st among published monocular methods. We also provide the first monocular 3D detection results on the newly released Waymo Open Dataset. We provide a code release for CaDDN which is made available.
1. Introduction
Monocular 3D perception offers a simpler, lower-cost alternative to LiDAR and stereo but performs worse because image projection removes depth information. CaDDN addresses depth uncertainty and feature smearing with supervised categorical depth distributions and end-to-end detection, achieving leading benchmark results.
- LiDAR and stereo provide precise 3D measurements and strong benchmark performance, whereas monocular perception seeks a lower-cost, easier-to-deploy single-camera solution.
- Monocular methods lag because projecting scenes onto the image plane loses depth information, making accurate 3D localization difficult.
- Explicit depth methods can become over-confident, especially at long range, while separately trained depth estimation cannot adapt depth predictions to 3D detection needs.
- Implicit image-to-BEV methods suffer feature smearing because similar image features may project to multiple scene locations.
- CaDDN predicts pixel-wise categorical depth distributions, using depth confidence to produce sharper, more accurate BEV features while retaining uncertainty when confidence is low.
- CaDDN ranks first among published monocular methods on KITTI Car and Pedestrian categories, with margins of 1.69% and 1.46% AP|R40 respectively.
2. Related Work
Prior monocular 3D detection methods either estimate depth separately or transform image features directly into BEV representations, each leaving important localization issues. CaDDN jointly learns depth and detection while using depth estimates to generate localized BEV features.
- Monocular depth estimation commonly predicts one depth value per image pixel using architectures adapted from semantic segmentation.
- Depth-based 3D detection methods use precomputed depth maps or convert them into Pseudo-LiDAR point clouds, but separate training can produce sub-optimal detection-oriented depth maps.
- Grid-based methods avoid raw depth estimation by predicting BEV grids; OFT populates voxel grids by projecting voxels into the image plane and sampling image features.
- CaDDN jointly trains depth estimation and 3D detection end-to-end, leveraging depth estimates to generate accurate, localized BEV features.
3. Methodology
CaDDN projects image features into 3D using pixel-wise categorical depth distributions, transforms them into voxels and BEV features, and detects objects with an efficient BEV network.
- Frustum Feature Network: CaDDN constructs a frustum feature grid by combining image features with predicted categorical depth distributions over discretized depth bins.Each feature pixel is weighted by its depth-bin probabilities to populate the frustum depth axis.
- Depth Distribution Network: The depth distribution network predicts per-pixel probabilities for predefined depth bins using a DeepLabV3-inspired downsample-upsample architecture.The probabilities represent the network’s confidence that each pixel belongs to a specified depth bin.
- Frustum to Voxel Transformation: Frustum features are transformed into a voxel grid with camera calibration and differentiable sampling, using trilinear interpolation to populate voxel features.Voxel sampling points are generated at voxel centers, projected into the frustum grid, and used to sample frustum features.
- Voxel Collapse to BEV: The voxel grid is collapsed along height into a BEV feature grid, reducing computational overhead while retaining similar detection performance to 3D voxel grids.The resulting BEV representation is processed by a PointPillars-based backbone and detection head.
- Depth Discretization: CaDDN adopts linearly increasing depth discretization, which uses linearly increasing bin sizes and is selected for balanced depth estimation across all depths.Uniform, spacing-increasing, and linear-increasing discretization are presented as alternative schemes.
- Training Losses: The depth distribution network is supervised with focal loss, using αfg = 3.25 for foreground pixels and αbg = 0.25 for background pixels.The weighting addresses the imbalance between fewer object pixels and more background pixels in autonomous-driving images.
4. Experimental Results
CaDDN is evaluated on KITTI and Waymo, including benchmark comparisons and ablations of depth distributions, supervision, uncertainty, and loss weighting.
- Evaluation Setup: CaDDN is evaluated on KITTI and Waymo, with KITTI using AP|R40 and Waymo using mAP and mAPH across difficulty and distance settings.KITTI evaluation separates Easy, Moderate, and Hard settings and object classes; Waymo evaluation uses LEVEL 1 and LEVEL 2 settings and sensor distance ranges.
- KITTI Results: CaDDN ranks first among published monocular methods on KITTI Car and Pedestrian categories, with margins of 1.69% and 1.46% AP|R40 respectively.The comparison includes published single-frame and multi-frame monocular methods.
- Waymo Results: +4.69%/+4.65% and +4.15%/+4.12% AP/APH margins over M3D-RPN are achieved on Waymo LEVEL 1 and LEVEL 2 at IoU 0.7.These results are reported on the Waymo validation set for the Vehicle class.
- Ablation Studies: +1.50%, +0.77%, and +0.46% AP|R40 result from adding depth distributions, while depth-distribution supervision adds +10.40%, +7.60%, and +6.54% across KITTI difficulties.The gains are reported for the Car class on Easy, Moderate, and Hard difficulties.
- Ablation Studies: Entropy generally increases with depth, indicating that estimated categorical depth distributions represent greater uncertainty where depth estimation is more challenging.Figure 6 plots mean entropy and 95% confidence intervals by ground-truth depth bin.
5. Conclusion
CaDDN estimates pixel-wise categorical depth distributions and combines them with image features to generate confidence-preserving BEV representations for 3D detection.
- Conclusion: CaDDN estimates accurate categorical depth distributions for each pixel and combines them with image features to generate bird’s-eye-view representations retaining depth confidence.The representations support downstream 3D object detection.
- Conclusion: Sharp distributions centered on correct depth values and joint depth estimation and object detection are reported as vital for 3D detection performance.The approach led to a first-place ranking among published KITTI monocular methods at submission.
A.1. KITTI Dataset Results
CaDDN achieves strong KITTI BEV detection results against previous monocular methods across Car, Pedestrian, and Cyclist categories.
- Car: +2.91%, +1.59%, and +2.22% AP|R40 over previous single-frame methods are reported for Car on Easy, Moderate, and Hard difficulties.The results are from the KITTI test set.
- Pedestrian: +3.73%, +2.37%, and +1.88% AP|R40 over MonoPair are reported for Pedestrian on Easy, Moderate, and Hard difficulties.MonoPair is identified as the previous state-of-the-art method for this comparison.
- Cyclist: CaDDN achieves first or second place on Cyclist, with margins of -1.37%, -1.33%, and +0.18% AP|R40 relative to MonoPSR.The margins correspond to Easy, Moderate, and Hard difficulties.
A.2. Ablation Studies
Ablations show that depth discretization and feature resolution affect CaDDN performance, with LID selected and smaller image features reducing detection quality.
- Depth Discretization: LID provides the highest performance among the evaluated depth discretization methods and is therefore adopted in CaDDN.The alternatives are uniform discretization and spacing-increasing discretization.
- Feature Resolution: Smaller-resolution image features reduce performance in the feature-resolution ablations.The experiments modify the image feature extraction layer in the Image Backbone.
- Feature Resolution: Smaller spatial resolutions cause oversampling during frustum-to-voxel transformation, producing many voxel features with similar features.This similarity is associated with reduced detection performance in the reported ablations.
B. Additional Details
The implementation adds an out-of-range depth bin for training while excluding it from frustum features, and initializes selected networks from pretrained DeepLabV3 weights.
- Depth Distributions: An additional depth bin represents depths outside the range [dmin, dmax].This bin is included in the depth distribution loss Ldepth.
- Depth Distributions: The out-of-range bin is removed when generating frustum features G.
- Training Details: The Image Backbone and Depth Distribution Network are initialized with DeepLabV3 pretrained on MS-COCO, while other components are randomly initialized.