PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection

Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger, Wei-Lun Chao

arXiv:2004.03080v2cs.CVeess.IV

TL;DR

Accurate 3D object detection is important for safe autonomous driving, but LiDAR is prohibitively expensive and pseudo-LiDAR’s depth and detection networks are separately trained. The paper introduces differentiable CoR modules for end-to-end pseudo-LiDAR training, improving benchmark performance and achieving the best KITTI image-based result at submission.

  • Problem

    LiDAR is expensive, while pseudo-LiDAR requires separately trained depth and detection networks whose objectives may be misaligned.

  • Method

    The framework uses differentiable subsampling and soft quantization CoR modules to train pseudo-LiDAR depth estimation and detection end-to-end.

  • Results

    The method improves over pseudo-LiDAR and PL++ across evaluation settings, and PointRCNN achieves the best KITTI image-based 3D detection leaderboard result at submission.

  • Takeaways & Limitations

    End-to-end pseudo-LiDAR narrows the remaining accuracy gap between stereo cameras and LiDAR-based sensors while remaining compatible with multiple detector input types.

  • Takeaways & Limitations

    The framework relies on depth loss because detection loss may influence only parts of the pixels and detection-only training can break the detector’s accurate-point-cloud inductive bias.

Abstract

from arXiv · show

Reliable and accurate 3D object detection is a necessity for safe autonomous driving. Although LiDAR sensors can provide accurate 3D point cloud estimates of the environment, they are also prohibitively expensive for many settings. Recently, the introduction of pseudo-LiDAR (PL) has led to a drastic reduction in the accuracy gap between methods based on LiDAR sensors and those based on cheap stereo cameras. PL combines state-of-the-art deep neural networks for 3D depth estimation with those for 3D object detection by converting 2D depth map outputs to 3D point cloud inputs. However, so far these two networks have to be trained separately. In this paper, we introduce a new framework based on differentiable Change of Representation (CoR) modules that allow the entire PL pipeline to be trained end-to-end. The resulting framework is compatible with most state-of-the-art networks for both tasks and in combination with PointRCNN improves over PL consistently across all benchmarks -- yielding the highest entry on the KITTI image-based 3D object detection leaderboard at the time of submission. Our code will be made available at https://github.com/mileyan/pseudo-LiDAR_e2e.

1 Cornell Univeristy 2 Cornell Tech 3 The Ohio State University

LiDAR enables accurate 3D detection but is prohibitively expensive, motivating stereo-based pseudo-LiDAR. This paper makes pseudo-LiDAR trainable end-to-end through differentiable representation changes, improving detection while preserving compatibility with existing models.

  • LiDAR provides precise 3D point clouds for safe object detection, but its cost can make autonomous driving prohibitively expensive.
  • Pseudo-LiDAR converts stereo-derived depth maps into 3D point clouds and applies LiDAR-based detectors, narrowing the image-based accuracy gap.
  • Separately trained depth estimation and detection objectives can be misaligned, with depth losses overemphasizing nearby or non-object pixels.
  • The proposed framework uses differentiable subsampling and quantization CoR modules, including soft quantization, to enable end-to-end training while retaining modularity.
  • The approach improves over pseudo-LiDAR and PL++ across evaluation settings, with PointRCNN achieving the best KITTI image-based 3D detection leaderboard results at submission.

2. Related Work

Prior 3D detection work primarily uses LiDAR point clouds, while pseudo-LiDAR reduces the image-based gap by converting estimated pixel depths into detector-ready 3D points.

  • LiDAR-based detectors process either unordered 3D point clouds or quantized 3D/4D tensor representations.
  • Solely image-based 3D detectors largely developed from frontal-view detection pipelines are generally less competitive at 3D localization.
  • Figure 2 reports that 90% of pixels are background, while 10% correspond to cars and people and are primarily within 20m depth.
  • Pseudo-LiDAR estimates pixel depths from images, converts them into 3D points, and applies existing LiDAR-based detectors.

3. End-to-End Pseudo-LiDAR

The paper extends pseudo-LiDAR with differentiable change-of-representation modules so depth estimation and 3D object detection can be trained jointly while retaining detector compatibility. It develops soft quantization and subsampling pathways that transmit detection gradients back to estimated depths.

  • Motivation: Pseudo-LiDAR’s two-step training can misalign generic depth estimation with the object detector’s need for accurate points near object surfaces, especially for distant objects.Depth losses treat pixels equally, although cars and pedestrians/cyclists occupy only about 10% of KITTI pixels.
  • End-to-End Framework: A differentiable CoR layer connects the depth estimator to the 3D detector, allowing detection errors to influence the pixels most affecting object predictions.The framework preserves compatibility with different LiDAR-based detector types.
  • Quantization: Soft quantization replaces binary voxel occupancy with RBF-weighted point counts and local neighboring-bin influence, enabling gradients to adjust estimated points.A detection gradient can affect neighboring bins and can pull points from another bin even when the target bin is empty.
  • Subsampling: For point-based detectors, angular subsampling retains one point per discretized spherical-coordinate bin, producing a sparser cloud that mimics true LiDAR beams.Pseudo-LiDAR averages 300,000 points versus 18,000 points for LiDAR in the frontal car view, motivating sparsification.
  • Training: Direct coordinate gradients can be propagated through subsampling and the differentiable depth-to-3D conversion, but sparse detection gradients require retaining the initial depth loss.The depth loss remains important because quantization or subsampling may allow detection loss to influence only part of the image.
  • Scope: The resulting framework is presented as a general end-to-end approach applicable to various point-cloud-based object detectors.The authors describe the section as providing practical details needed to make end-to-end training successful rather than claiming a separate technical contribution for it.

4. Experiments

Experiments on KITTI evaluate E2E-PL with stereo depth estimation and two detector input types, showing consistent gains from joint training across quantitative, ablation, qualitative, and speed analyses.

  • Setup: KITTI contains 3,712 training, 3,769 validation, and 7,518 test images, with evaluation focused on car detection using APBEV and AP3D at IoU thresholds 0.5 and 0.7.
  • Quantitative results: E2E-PL consistently improves detection for both P-RCNN with subsampled points and PIXOR⋆ with quantized inputs on the KITTI validation set.End-to-end training also reduces the moderate APBEV gap at IoU 0.5 between the two detector types from 5.4% to −0.4%.
  • Quantitative results: At submission, E2E-PL with P-RCNN achieves state-of-the-art results among image-based models on the KITTI test set.
  • Ablation studies: Jointly training Depth, RPN, and RCNN with their corresponding losses produces the best P-RCNN ablation results except in two cases, whereas independent extra training does not improve accuracy.
  • Ablation studies: For PIXOR⋆, jointly training Depth and Detector significantly improves performance, while soft quantization alone does not outperform hard-quantized PL++: PIXOR⋆.Soft quantization allows Detector loss gradients to influence 70% of predicted-depth pixels.

5. Conclusion and Discussion

The paper presents end-to-end pseudo-LiDAR training that supports both direct point-cloud and quantized inputs, achieving state-of-the-art image-based 3D detection while narrowing the stereo–LiDAR gap.

  • E2E-PL supports 3D detectors using either direct point-cloud inputs or quantized structured inputs.
  • The resulting models set a new state of the art in image-based 3D object detection.
  • E2E-PL further narrows the remaining accuracy gap between stereo and LiDAR-based sensors.
  • Higher-resolution images with more labeled far-away cars could further improve detection, especially for far-away and heavily occluded cases.

S1. Results on Pedestrians and Cyclists

The evaluation extends beyond cars to pedestrians and cyclists on KITTI, where E2E-PL outperforms the non-end-to-end PL++ baseline for image-based 3D detection.

  • E2E-PL outperforms PL++ by a notable margin for image-based 3D detection of pedestrians and cyclists.
  • The pedestrian and cyclist results use the KITTI object-detection validation set and P-RCNN as the object detector.
  • The reported metrics are APBEV and AP3D at IoU=0.5.

S2. Evaluation at Different Depth Ranges

Across near and far depth ranges, E2E-PL generally performs better than the baseline, with larger far-range APBEV gains in the hard setting.

  • E2E-PL achieves better performance at both 0–30-meter and 30–70-meter ranges, except for 30–70 meters at moderate AP3D.
  • On APBEV, the relative gain between E2E-PL and the baseline becomes larger for far-away objects in the hard setting.
  • The point-cloud-based evaluation uses P-RCNN and reports car APBEV/AP3D at IoU=0.7.
  • The quantization-based evaluation uses PIXOR⋆ and reports APBEV without AP3D because PIXOR⋆ lacks height prediction.

S3. On KITTI Test Set

On KITTI test-set precision–recall curves, E2E-PL suppresses the high-confidence false positives observed with PSEUDO-LiDAR++ across easy, moderate, and hard cars.

  • PSEUDO-LiDAR++ shows a notable precision drop even at low recalls, indicating many high-confidence false-positive predictions.
  • E2E-PL suppresses these false positives across easy, moderate, and hard cars.

S4. Additional Qualitative Results

Qualitative comparisons show that E2E-PL improves depth estimates in car-related regions and produces fewer detection errors than PL++.

  • E2E-PL improves estimated depth in car-related areas compared with PL++.Figure S8 highlights these depth improvements using red bounding boxes.
  • E2E-PL produces fewer false-positive and false-negative predictions than PL++.The detection comparison is shown in Figure S9, with ground-truth and predicted boxes distinguished by color.
  • Figure S9 uses red bounding boxes for ground truth and green bounding boxes for predictions.

S5. Gradient Visualization on Depth Maps

Gradient visualizations examine how the detection loss affects depth maps and compare E2E-PL with PL++ through detection curves. The gradients focus heavily around cars.

  • Figure S10 visualizes relative absolute gradient values with a JET colormap, using red for higher and blue for lower values.
  • Figure S7 compares E2E-PL with PL++ using precision-recall curves on KITTI 3D object detection and bird’s eye view detection tracks.
  • Detection-loss gradients focus heavily around cars in the depth map.

S6. Other results

Additional experiments report depth-estimation and detection results across KITTI and Argoverse, alongside qualitative comparisons and gradient visualizations. E2E-PL outperforms PL++ on Argoverse validation results.

  • Detection-loss improvements to overall depth estimation are limited because foreground objects occupy merely 10% of pixels.Improvements appear at certain depth ranges around objects, and the authors hypothesize that detection loss sharpens object boundaries rather than directly improving metric depth.
  • Table S9 reports quantitative depth-estimation results with and without end-to-end training.
  • Figures S8 and S9 provide qualitative comparisons of depth estimation and detection results between PL++ and E2E-PL.Figure S9 distinguishes ground truth with red boxes and predictions with green boxes.
  • Figure S10 visualizes absolute detection-loss gradients with a JET colormap, where red indicates larger values and blue indicates smaller values.
  • Table S10 reports car-category APBEV / AP3D percentages for a point-cloud-based P-RCNN pipeline on Argoverse, organizing methods by stereo, LiDAR, and pseudo-LiDAR inputs.
  • E2E-PL outperforms PL++ on Argoverse validation detection results.The Argoverse data are converted to KITTI format, and the experiments use the same training scheme and hyperparameters as KITTI.
Loading 2004.03080v2…