Source-linked AI summary
Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving
Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger
TL;DR
Stereo-based pseudo-LiDAR offers a cheaper alternative to expensive LiDAR but retains systematic depth errors, especially for faraway objects. The paper introduces direct depth learning with a depth cost volume and uses graph-based propagation of sparse LiDAR measurements to correct dense stereo estimates. On KITTI, stereo-only results improve over prior state of the art, while 4-beam LiDAR adds further gains and approaches 64-beam performance at lower cost.
Problem
Stereo-based pseudo-LiDAR remains less accurate than LiDAR-based detection because its depth estimates are systematically biased, particularly for faraway objects.
Method
The paper combines a stereo depth network optimized directly for depth with graph-based propagation that spreads sparse, exact LiDAR measurements through the dense depth map.
Results
Stereo-only PSEUDO-LIDAR++ outperforms prior image-based state of the art by 10%, while adding 4-beam LiDAR yields another 27% relative improvement and approaches 64-beam LiDAR performance.
Takeaways & Limitations
Cheap sparse LiDAR can correct stereo’s systematic depth bias, while stereo supplies dense object structure that sparse LiDAR alone cannot capture precisely.
Takeaways & Limitations
Stereo depth remains limited by pixel-level disparity quantization, while higher-resolution prediction has cubically increasing computational cost; sparse LiDAR alone is too sparse to capture object shapes.
Abstract
from arXiv · showhide
Detecting objects such as cars and pedestrians in 3D plays an indispensable role in autonomous driving. Existing approaches largely rely on expensive LiDAR sensors for accurate depth information. While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still a notable performance gap. In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation. Concretely, we adapt the stereo network architecture and loss function to be more aligned with accurate depth estimation of faraway objects --- currently the primary weakness of pseudo-LiDAR. Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufficient information for 3D detection, to de-bias our depth estimation. We propose a depth-propagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map. We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection --- outperforming the previous state-of-the-art detection accuracy for faraway objects by 40%. Our code is available at https://github.com/mileyan/Pseudo_Lidar_V2.
1 INTRODUCTION
Pseudo-LiDAR reduces the cost of stereo-based 3D detection, but systematic depth errors—especially for faraway objects—remain a safety-relevant gap. The paper combines improved stereo depth estimation with sparse LiDAR correction to address this trade-off.
- Accurate depth is essential for safe 3D detection and localization, but 64-beam LiDAR can cost around $75,000.
- Stereo-based pseudo-LiDAR remains substantially weaker than LiDAR-based systems, particularly for faraway objects that must be detected early.
- Stereo systems reliably detect objects but systematically estimate an entire object’s depth as too far or too close.
- Disparity-based estimation overemphasizes nearby objects because reciprocal conversion turns equal disparity errors into much larger far-depth errors.A one-pixel disparity error corresponds to 0.1m at 5m but 5.8m at 50m.
- The proposed solution combines stereo cameras for dense structure with cheap sparse LiDAR measurements to correct systematic depth bias.Four-beam LiDAR is described as two orders of magnitude cheaper than 32- or 64-beam sensors.
- 10% improvement over prior state of the art comes from stereo alone, while adding 4-beam LiDAR provides another 27% relative improvement.The combined approach can potentially save 95% in cost compared with 64-beam LiDAR.
2 BACKGROUND
Pseudo-LiDAR estimates per-pixel depth from stereo images, back-projects it into a 3D point cloud, and then applies LiDAR-style 3D detection. Its effectiveness therefore depends strongly on stereo depth quality.
- 3D object detection: Most 3D object detection methods operate on LiDAR point clouds, using direct, voxelized, or bird’s-eye-view representations.
- Pseudo-LiDAR: Pseudo-LiDAR predicts each image pixel’s depth and back-projects the resulting depth map into a 3D point cloud.
- Pseudo-LiDAR: The resulting point cloud is treated as LiDAR signal, allowing LiDAR-based 3D detectors to be applied directly.
- Stereo disparity estimation: Stereo disparity estimation takes rectified left-right images and produces a disparity map from corresponding horizontal pixel shifts.
- Stereo disparity estimation: Disparity pipelines construct cost volumes that encode feature differences across candidate disparities before predicting each pixel’s disparity.
- Stereo disparity estimation: PSMNet extracts deep features from both images, builds a disparity cost volume, and learns the pipeline end-to-end to minimize disparity error.
3 STEREO DEPTH NETWORK (SDN)
The stereo depth network replaces disparity-centered processing with direct depth optimization and a depth-indexed cost volume. This aligns convolutional processing and loss weighting with the continuous depth errors relevant to 3D localization.
- Motivation: For a fixed disparity error, depth error grows quadratically with distance, causing disparity-trained networks to underperform on faraway objects.On KITTI, one pixel of disparity error produces 0.1m error at 5m and 5.8m at 50m.
- Depth Loss: The paper proposes learning stereo networks to directly optimize a depth loss rather than a disparity loss.
- Depth Loss: Depth loss reduces the disproportionate emphasis on tiny nearby-object errors, although this change alone is insufficient.
- Depth Cost Volume: Shared 3D convolution kernels in disparity volumes are problematic because reciprocal depth-disparity geometry makes their effect nonhomogeneous.
- Depth Cost Volume: SDN constructs a depth cost volume whose entries encode the likelihood of each pixel taking a candidate depth, then convolves over the depth grid.
- Depth Cost Volume: SDN uses a 1m depth grid over [1m, 80m] and applies convolution operations exclusively within the depth volume.
4 DEPTH CORRECTION
Graph-based depth correction combines dense stereo geometry with sparse, accurate LiDAR measurements to reduce stereo depth bias while preserving object shapes. It constructs a weighted KNN graph over pseudo-LiDAR points and propagates landmark depth corrections across the graph.
- Motivation: 4-beam LiDAR is too sparse to capture object shapes alone but provides exact depths for landmark pixels when projected into the image plane.These measurements are paired with stereo images to correct dense stereo depth estimates.
- Overview: GDC combines dense stereo depth, which preserves object shapes, with sparse accurate LiDAR measurements.The corrected map should retain exact landmark depths while preserving local shapes from neighboring back-projected points.
- Graph construction: The method builds a directed KNN graph over the pseudo-LiDAR cloud and assigns weights that reconstruct each point’s depth from its neighbors.LiDAR points are matched to corresponding stereo points, while the graph structure is computed using predicted stereo depths.
- Graph construction: The edge weights are obtained by constrained quadratic optimization with weights summing to one and zero outside each point’s neighbor set.For k > 3 and points in general position, the selected solution minimizes the L2 norm, with slight L2 regularization.
- Depth propagation: GDC fixes the n LiDAR-supported depths and optimizes the remaining depths so local weighted reconstructions remain faithful, propagating corrections through neighbors and their neighbors.The final corrected depths can be solved directly with a quadratic optimization; sparse matrix solvers solve both optimization problems efficiently.
- Depth propagation: For a single corrected point with update δ, the optimal solution shifts all remaining depths by the same δ; with multiple measurements, least squares provides soft diffusion.The single-point case follows from Z = WZ and W1 = 1.
5 EXPERIMENTS
Experiments on KITTI evaluate PL++ across depth-estimation components, sparse-LiDAR integration, detectors, object categories, and validation/test settings. PL++ improves localization, especially for faraway objects, while approaching dense-LiDAR performance with only four beams.
- Experimental setup: PL++ combines SDN and GDC and is evaluated with P-RCNN, AVOD, and PIXOR⋆ on KITTI car detection and BEV localization.The experiments use KITTI validation data primarily, with additional test-set results and analyses across detectors.
- KITTI validation results: 73.4 APBEV is achieved by PL++ P-RCNN with L# + S, compared with 52.7 APBEV for PL at the hard setting under IoU 0.7.The stereo-only PL++ version reaches 57.3 APBEV, indicating that most of this gain comes from GDC.
- Overall results: PL++ with four-beam LiDAR is comparable to 64-beam LiDAR at IoU 0.5, and its validation-set gap from LiDAR is at most 13% APBEV at IoU 0.7.On the KITTI test set, pure SDN ranks highest among image-based algorithms, with a similar gap to LiDAR.
- Depth estimation ablation: 6%/2% improvement comes from the depth loss on APBEV/AP3D (moderate), while the depth cost volume adds another 2∼3% gain.The relative contribution depends on the detector, and combining both approaches gives the best performance.
- Sparse LiDAR ablation: GDC improves sparse-LiDAR integration by propagating accurate measurements, outperforming both four-beam LiDAR alone and landmark-only depth replacement.Four-beam LiDAR locates faraway objects but cannot precisely capture nearby objects, while landmark-only replacement does not accurately detect faraway objects.
- Other object categories: PL++ significantly boosts pedestrian and cyclist detection using F-POINTNET on KITTI validation.These results are reported at IoU 0.5 for BEV and 3D average precision.
- Qualitative analysis: PL++ improves localization of faraway objects, where PSEUDO-LIDAR produces depth deviations and false negatives, while nearby-object localization is fairly strong for all point clouds.The qualitative comparison shows SDN and SDN + GDC boxes aligning more closely with ground truth.
- Runtime: GDC runs in 90 ms/frame on a single GPU, including 7.7 ms for KD-tree construction and search.The reported runtime uses simple optimizations.
6 CONCLUSION
The paper improves stereo-based 3D detection by learning depth directly and correcting systematic stereo bias with sparse LiDAR. The resulting PSEUDO-LIDAR++ system approaches 64-beam LiDAR performance using four beams and commodity cameras.
- 6 CONCLUSION: The paper identifies disparity estimation as a main stereo error source and proposes learning depth directly end-to-end.This replaces the conventional disparity-to-depth route with depth-oriented estimation.
- 6 CONCLUSION: A cheap sparse LiDAR signal corrects systematic bias in dense stereo depth estimates through graph-based propagation.The algorithm integrates the two modalities and uses sparse matrix solvers to propagate accurate depth measurements.
- 6 CONCLUSION: PSEUDO-LIDAR++ performs almost on par with 64-beam LiDAR systems while requiring four beams and two commodity cameras.The conclusion states that the 64-beam system costs $75,000, whereas the combined setup could cost less than $1,000.
C EXPERIMENTAL SETUP
The experiments simulate sparse LiDAR by selecting points from KITTI’s 64-beam scans according to elevation angle. The setup also specifies the training and sparsification procedures used for detection models.
- C.1 SPARSE LIDAR GENERATION: The 2-beam signal uses elevation intervals [−2.4°,−2.0°) and [−0.8°,−0.4°), while the 4-beam signal adds [−1.6°,−1.2°) and [0.0°,0.4°).The selected lines follow the specifications of the cheap four-beam ScaLa LiDAR.
- C.1 SPARSE LIDAR GENERATION: Points are ordered by elevation angle and sliced from −23.6° in 0.4° steps before selecting the sparse-beam subsets.The starting angle is close to the Velodyne 64-beam LiDAR specification.
- Experimental setup: Detection models are trained on KITTI’s 3,712 training images, with pseudo-LiDAR point clouds sparsified into 64 beams for P-RCNN.AVOD, PIXOR⋆, and P-RCNN use the described implementations or modifications, and pseudo-LiDAR replaces LiDAR inputs.
D.1 ABLATION STUDY
Ablations show consistent gains from depth-aware stereo estimation and GDC, while direct sparse-LiDAR replacement is model-dependent. Two-beam GDC already outperforms either sparse LiDAR or pseudo-LiDAR alone.
- AVOD and PIXOR⋆ show similar improvements from depth loss, depth cost volume construction, and GDC.
- P-RCNN with SDN + L# outperforms L# in 5 of 6 comparisons, whereas AVOD and PIXOR⋆ perform worse with SDN + L# at moderate and hard settings.
- Dense-input detectors can suppress the four replaced LiDAR beams, explaining why SDN + L# underperforms L# for AVOD and PIXOR⋆.
- Even with 2 beams, GDC combines sparse LiDAR and stereo signals to outperform 2-beam LiDAR or pseudo-LiDAR alone.
D.3 DEPTH CORRECTION VS. DEPTH COMPLETION
GDC addresses a different problem from conventional depth completion: it corrects dense stereo depth using sparse LiDAR rather than densifying LiDAR measurements. It is inference-time, training-free, and substantially outperforms PNP in the reported comparison.
- Most depth completion methods densify 64-beam LiDAR using a single image, whereas GDC fuses 4-beam LiDAR with stereo depths.
- GDC is a general inference-time approach requiring no training, unlike prior learning-based depth-completion methods.
- GDC outperforms PNP by a large margin when evaluated on median absolute depth errors from held-out LiDAR points.
- GDC runs in 90 ms/frame on a single GPU, including KD-tree construction and search, solving W, and solving Z′_PL.
D.5 STEREO DEPTH VS. DETECTION
Depth quality improvements translate most strongly to faraway-object detection, especially near sparse LiDAR measurements. Residual localization errors remain the main gap to LiDAR at high IoU thresholds.
- SDN (+ GDC) improves over PSMNET increasingly for pixels at greater truth depths.
- SDN (+ GDC) significantly outperforms PSMNET for detecting faraway cars, although 50–70 m training instances are extremely sparse.
- GDC provides larger improvements for pseudo-LiDAR points located closer to 4-beam LiDAR points.
- With k ≥9, the SDN pseudo-LiDAR KNN graph has fewer than 2 average connected components on KITTI validation data.
- For low IoU thresholds of 0.2–0.5, SDN (+GDC) is on par with LiDAR; the gap increases significantly at high IoU thresholds.
D.8 QUALITATIVE RESULTS
Qualitative KITTI scenes show that PSEUDO-LIDAR++ produces point clouds and detections that align better with ground truth, particularly for faraway objects where PSMNET is noisy and mislocalized.
- PSEUDO-LIDAR++ point clouds from SDN or SDN +GDC align better with LiDAR than PSMNET point clouds.
- Nearby objects are localized fairly well with all tested point clouds, while faraway objects expose PSMNET’s larger deviations from ground truth.
- Noisy PSMNET points produce several false positives or negatives in the qualitative scenes.
- SDN and SDN +GDC detections align relatively well with ground-truth boxes in the shown KITTI validation examples.