PaperlayerView exact revision on arXiv

Source-linked AI summary

Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving

Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger

arXiv:1812.07179v6cs.CV

TL;DR

Image-based 3D detection is cheaper than LiDAR-based detection but has historically lagged because estimated depth is represented poorly for convolutional detectors. The paper converts image depth maps into pseudo-LiDAR point clouds, applies existing LiDAR-based pipelines, and reports large KITTI gains, while noting data-overlap and runtime limitations.

  • Problem

    Image-based 3D detection has substantially lower accuracy than LiDAR-based detection, and this gap is commonly attributed to poor image-based depth estimation.

  • Method

    The method back-projects dense stereo or monocular depth maps into 3D point clouds called pseudo-LiDAR, then applies existing LiDAR-based detection pipelines.

  • Results

    45.3% 3D AP is achieved on KITTI for moderately hard cars at IoU 0.7, nearly a 350% improvement over the previous image-based approach.

  • Takeaways & Limitations

    The results suggest that representation is a major cause of the stereo-versus-LiDAR detection gap and that pseudo-LiDAR can substantially improve stereo-based detection.

  • Takeaways & Limitations

    The evaluation used overlapping KITTI stereo and detection images in released depth models, so the authors retrained PSMNet to avoid that overlap.

Abstract

from arXiv · show

3D object detection is an essential task in autonomous driving. Recent techniques excel with highly accurate detection rates, provided the 3D input data is obtained from precise but expensive LiDAR technology. Approaches based on cheaper monocular or stereo imagery data have, until now, resulted in drastically lower accuracies --- a gap that is commonly attributed to poor image-based depth estimation. However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations --- essentially mimicking the LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance --- raising the detection accuracy of objects within the 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo-image-based approaches. Our code is publicly available at https://github.com/mileyan/pseudo_lidar.

1. Introduction

Image-based 3D detection offers cheaper and potentially redundant sensing than LiDAR, but historically performs much worse. The paper argues that representation, rather than depth quality alone, is the major source of this gap and proposes pseudo-LiDAR to address it.

  • Reliable 3D object detection is fundamental for autonomous driving because vehicles must detect road users to avoid collisions.
  • Stereo and monocular cameras are substantially cheaper than LiDAR while offering high frame rates and dense depth maps.Cameras could also provide a secondary sensing modality if a primary sensor fails.
  • 10% AP is achieved by the image-only state of the art, compared with 66% for LiDAR and 73% for LiDAR fused with monocular images.
  • Stereo depth can match LiDAR point clouds closely, including for faraway objects, challenging poor depth precision as the sole explanation for weak detection.
  • Image-based depth channels make distant objects smaller and mix spatially separated 3D points within 2D neighborhoods, hindering ConvNet localization.
  • 45.3% 3D AP is obtained on KITTI for moderately hard cars at IoU 0.7, nearly a 350% improvement over the prior image-based approach.The method converts estimated depth into pseudo-LiDAR and trains existing LiDAR-based detection pipelines on it.
  • Consistent gains across multiple depth-estimation and detection combinations suggest that pseudo-LiDAR representation, rather than particular component innovations, drives the improvements.
  • The paper reports state-of-the-art stereo-based detection and describes the improvement as effectively tripling prior art.The results point toward stereo cameras becoming viable for self-driving cars, with potential cost or safety benefits.

2. Related Work

Prior work combines LiDAR with image information, estimates depth from monocular or stereo images, and adapts 2D detection pipelines for image-based 3D detection.

  • LiDAR-based detectors exploit point-cloud representations through frustum processing, bird’s-eye and frontal views, voxel encoding, or 3D convolutions.
  • Monocular and stereo depth estimation methods provide increasingly accurate pixel-depth predictions for replacing LiDAR inputs.Examples include DORN for monocular depth and PSMNet for stereo disparity estimation.
  • Image-based 3D detection methods commonly build on 2D object detection, adding geometric constraints or using per-pixel 3D coordinates as channels or handcrafted features.

3. Approach

The approach argues that image-based 3D detection suffers mainly from an unsuitable representation rather than inaccurate depth, and introduces pseudo-LiDAR to align image and LiDAR detection pipelines. Estimated depth is back-projected into 3D and processed with existing LiDAR-based detectors, while convolutional operations motivate using physically meaningful 3D or bird’s-eye-view representations.

  • Data representation matters: The paper attributes much of the image–LiDAR detection gap to data representation rather than depth quality or physical sensing differences.The authors argue that convolutional networks process image-based depth in ways that distort physical relationships.
  • Pseudo-LiDAR generation: Pseudo-LiDAR converts estimated stereo or monocular depth into a 3D point cloud in the LiDAR coordinate system.All pixels are back-projected into 3D coordinates, producing a point cloud that can be transformed into a cyclopean coordinate frame.
  • Motivation: Image-based depth can be visually well aligned with LiDAR points, including for far-away objects, despite stereo depth error increasing quadratically with depth.This observation supports questioning depth accuracy as the primary explanation for inferior detection.
  • Detection pipelines: Existing LiDAR-based 3D object detectors can process pseudo-LiDAR, including point-cloud and bird’s-eye-view pipelines.Frustum PointNet operates on pseudo-LiDAR as a 3D point cloud, while AVOD uses a bird’s-eye-view representation that encodes width, depth, and height.

4. Experiments

Experiments on KITTI evaluate pseudo-LiDAR across depth estimators, detection architectures, object categories, and validation/test settings. Pseudo-LiDAR substantially improves image-based detection, while remaining gaps are largest for difficult, distant, or small objects.

  • Experimental setup: KITTI experiments use 7,481 training images and 7,518 test images, with a 3,712/3,769 training-validation split and accompanying LiDAR, stereo images, and calibration.Evaluation focuses on car AP3D and APBEV at IoU thresholds 0.5 and 0.7, across easy, moderate, and hard cases.
  • Car detection: Pseudo-LiDAR stereo approaches significantly outperform image-based alternatives and double prior state-of-the-art performance for moderate cars at IoU = 0.7.The representation also benefits two detection algorithms with different architectures, indicating broad compatibility.
  • Representation: At IoU = 0.7, pseudo-LiDAR using DISPNET outperforms MLF-STEREO by at least 16% on both APBEV and AP3D.Both methods use DISPNET, so the comparison attributes the gain to representing depth as pseudo-LiDAR rather than frontal-view feature maps.
  • Depth and detection: Detection accuracy does not necessarily track disparity accuracy because disparity errors can translate into depth errors differently, while detectors process 3D points differently.AVOD voxelizes points, whereas F-POINTNET processes them directly and may be more vulnerable to noise.
  • Depth and detection: PSMNET⋆ produces the most accurate detection results after training on the KITTI detection training set, suggesting potential for further end-to-end pipeline improvement.The authors also note that the released stereo models had validation-image overlap, motivating their retrained PSMNET⋆ version.
  • Comparison to LiDAR: Pseudo-LiDAR approaches approach LiDAR performance for easy cars at IoU = 0.5 but lose accuracy on harder and distant instances, especially at IoU = 0.7.The remaining gap is linked to greater stereo depth error at distance and the low resolution of the 0.4-megapixel images.
  • Pedestrian and cyclist detection: Pedestrian and cyclist detection remains substantially harder than car detection, with larger pseudo-LiDAR–LiDAR gaps in hard cases.The approach nevertheless provides a starting point for image-based detection of these categories.
  • Test-set results: The pseudo-LiDAR–LiDAR performance gap is similar on the test set, and the method ranked first among image-based KITTI algorithms at submission.This test-set comparison suggests the validation results were not simply overfit to validation data.

5. Discussion and Conclusion

The paper argues that representing image-derived 3D information as pseudo-LiDAR can substantially narrow the gap with LiDAR-based detection. It identifies image-based detection as a potentially lower-cost alternative while noting directions for improving resolution, speed, and sensor fusion.

  • 5. Discussion and Conclusion: Pseudo-LiDAR representation may be the key component for closing the gap between image- and LiDAR-based 3D object detection.The authors characterize the contribution as correcting a systemic representation inefficiency rather than introducing a wholly novel algorithm.
  • 5. Discussion and Conclusion: Image-based detection could reduce autonomous-driving hardware costs because LiDAR is currently described as the most expensive additional component.The paper also notes potential benefits from image-based detection even when LiDAR equipment is present.
  • 5. Discussion and Conclusion: Higher-resolution stereo images could improve accuracy for faraway objects beyond the paper’s 0.4-megapixel setting.The authors identify current image resolution as substantially below state-of-the-art camera technology.
  • 5. Discussion and Conclusion: The reported pipeline classifies all objects in one image in approximately 1s, leaving real-time image processing as an improvement direction.The paper suggests faster depth estimation, model distillation, and anytime prediction as possible routes to acceleration.
  • 5. Discussion and Conclusion: Future work could combine LiDAR and pseudo-LiDAR because pseudo-LiDAR is denser and the modalities may have complementary strengths.This is presented as a possible direction for improving 3D object detection.

Supplementary Material

The supplementary material adds methodological details, further stereo and depth analyses, additional test-set and qualitative results, and ground-plane estimation details for AVOD.

  • Supplementary Material: Section A provides additional details on the approach described in Section 4.2 of the main paper.
  • Supplementary Material: Section B reports results using SPS-STEREO, while Section C provides further analysis of depth estimation.
  • Supplementary Material: Section D presents additional results on the test set, and Section E presents additional qualitative results.
  • A. Additional approach details: For AVOD, the supplementary method estimates a ground plane using pseudo-LiDAR points near the camera and within a plausible ground-height range.The plane is fit with RANSAC under the constraint w_y = −1, followed by unit-ℓ2 normalization of w.

A.2. Pseudo disparity ground truth

The pseudo-disparity ground truth procedure trains stereo estimation on detection images by projecting LiDAR points into image space and converting pixel depth to disparity.

  • A.2. Pseudo disparity ground truth: PSMNET⋆ is trained on 3,712 detection-training images rather than the 200 KITTI stereo images.The supplementary procedure uses LiDAR-derived pseudo-disparity ground truth for this training set.
  • A.2. Pseudo disparity ground truth: LiDAR points are projected into 2D image space, then pixel depth is converted to disparity using the main paper’s Eq. (1).When multiple LiDAR points map to one pixel, one is randomly retained; pixels without depth are ignored during training.
  • A.2. Pseudo disparity ground truth: Table 6 compares pseudo-LiDAR detection accuracy across stereo disparity methods for moderate cars at IoU = 0.7.
  • A.2. Pseudo disparity ground truth: Table 7 evaluates how over-smoothing depth estimates affects pseudo-LiDAR detection results for moderate cars at IoU = 0.7.

B. Results Using SPS-STEREO [35]

Pseudo-LiDAR detection remains effective when stereo disparity is estimated with the non-learning-based SPS-STEREO method. Its results are on par with those obtained using PSMNET and DISPNET.

  • B. Results Using SPS-STEREO [35]: 3.84% disparity error on KITTI stereo 2015 makes SPS-STEREO less accurate than PSMNET’s 1.86% but more accurate than DISPNET-C’s 4.32%.
  • B. Results Using SPS-STEREO [35]: SPS-STEREO produces pseudo-LiDAR detection results on par with PSMNET and DISPNET despite being non-learning-based.

C. Further Analysis on Depth Estimation

Over-smoothing estimated depth degrades pseudo-LiDAR 3D detection, indicating that high-quality depth estimates matter for accuracy.

  • Over-smoothing depth estimates degrades pseudo-LiDAR 3D object detection accuracy.AVOD and F-POINTNET were evaluated with PSMNET⋆ depth maps smoothed using an 11 × 11 average kernel.

D. Additional Results on the Test Set

Additional test-set results compare pseudo-LiDAR and LiDAR for pedestrian and cyclist detection using KITTI’s standard AP metrics.

  • The test-set evaluation applies Mask R-CNN trained on MS COCO for pedestrian and cyclist 2D detections.Because MS COCO lacks a cyclist category, bicycle detections substitute for cyclist detections.
  • Table 8 reports APBEV / AP3D at IoU = 0.5 for pedestrian and cyclist detection on the KITTI test set.The comparison includes pseudo-LiDAR with PSMNET⋆ and LiDAR.
  • Pedestrian detection shows a similar pseudo-LiDAR versus LiDAR gap on the test set as on the validation set.The passage attributes the cyclist performance drop to cyclist rarity and algorithm over-fitting, among other factors.
  • Cyclist detection experiences a drastic performance drop with pseudo-LiDAR.For F-POINTNET, bicycle detections may provide inaccurate cyclist heights because cyclists include riders and bicycles.
  • No image-based algorithms had reported test-set 3D results for pedestrians and cyclists at that point.

E. Additional Qualitative Results

Qualitative comparisons show strong pseudo-LiDAR and LiDAR alignment, while depth-estimation quality affects agreement at distance.

  • Pseudo-LiDAR points generated by PSMNET⋆ align very well with LiDAR points.The comparison is shown in Figure 5 using qualitative visualizations.
  • PSMNET and PSMNET⋆ perform similarly at nearby distances but diverge at far-away distances.PSMNET pseudo-LiDAR points show notable deviation from the LiDAR signal farther away.
  • The far-distance deviations suggest that larger training sets could further improve learned disparity estimation.The passage also points toward further improvements through learning disparity on a larger training set or an enhanced approach.

E.3. Visualization and failure cases

Pseudo-LiDAR supports accurate nearby-object predictions but struggles with distant, occluded, and monocular-depth cases, where missed detections and localization errors remain prominent.

  • Visualization and failure cases: LiDAR and stereo or monocular pseudo-LiDAR produce accurate predictions for nearby objects.
  • Visualization and failure cases: Pseudo-LiDAR detects far-away objects less precisely or misses them, especially with monocular depth.The reported errors include mislocalization and missed detections; monocular pseudo-LiDAR misses four middle-distance cars in one example.
  • Visualization and failure cases: Pseudo-LiDAR-based detection has much lower recall than LiDAR-based detection, particularly for moderate and hard cases.Moderate and hard cases involve far-away or occluded objects, making missed detections a major issue.
  • Visualization and failure cases: Stereo pseudo-LiDAR misses a partially occluded car even when it is close to the observer.The authors suggest noisy stereo disparity estimation around occlusion boundaries as a likely explanation.
  • Visualization and failure cases: The frontal-view stereo approach makes extremely inaccurate predictions, including for nearby objects.
Loading 1812.07179v6…