PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud

Xinshuo Weng, Kris Kitani

arXiv:1903.09847v4cs.CV

TL;DR

The paper addresses the gap between monocular and sensor-based 3D object detection. It lifts images into pseudo-LiDAR, applies LiDAR-based detection with two noise-reduction innovations, and reports top-ranked KITTI performance among monocular methods.

  • Problem

    Monocular 3D object detection performs substantially worse than methods using explicit 3D sensors, despite the availability and low cost of single cameras.

  • Method

    The method lifts image pixels using monocular depth into pseudo-LiDAR, extracts mask-based frustums, and predicts 3D boxes with 2D-3D consistency refinement.

  • Results

    The method achieves top-ranked KITTI bird’s-eye-view and 3D detection performance among monocular methods, including 21.0% AP versus 5.7% at IoU 0.7.

  • Takeaways & Limitations

    The results demonstrate that LiDAR-based 3D detection can be adapted to single-image input through pseudo-LiDAR and targeted handling of its noise.

  • Takeaways & Limitations

    The approach assumes that the camera matrix is known.

Abstract

from arXiv · show

Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor. On the other hand, single image based methods have significantly worse performance. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, which we call pseudo-LiDAR point cloud. Then we can train a LiDAR-based 3D detection network with our pseudo-LiDAR end-to-end. Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal. Then an oriented 3D bounding box is detected for each frustum. To handle the large amount of noise in the pseudo-LiDAR, we propose two innovations: (1) use a 2D-3D bounding box consistency constraint, adjusting the predicted 3D bounding box to have a high overlap with its corresponding 2D proposal after projecting onto the image; (2) use the instance mask instead of the bounding box as the representation of 2D proposals, in order to reduce the number of points not belonging to the object in the point cloud frustum. Through our evaluation on the KITTI benchmark, we achieve the top-ranked performance on both bird's eye view and 3D object detection among all monocular methods, effectively quadrupling the performance over previous state-of-the-art. Our code is available at https://github.com/xinshuoweng/Mono3D_PLiDAR.

1. Introduction

Monocular 3D detection could reduce reliance on costly 3D sensors, but inaccurate depth creates noisy pseudo-LiDAR and weak performance. The paper extends LiDAR-based detection to images using consistency constraints and instance-mask proposals, achieving strong KITTI results.

  • Monocular 3D detection is important for autonomous driving and robot vision because inexpensive onboard cameras are widely available.
  • 3D sensors provide explicit scene depth, but depth cameras have limited range, stereo systems require difficult calibration and synchronization, and LiDAR is expensive.
  • 13.6% AP versus 86.5% AP on KITTI’s moderate car set illustrates the performance gap between monocular and LiDAR-based methods.
  • The method lifts every image pixel using estimated depth into a 3D pseudo-LiDAR point cloud, then applies LiDAR-based detection to image-derived data.
  • Inaccurate monocular depth causes frustum misalignment and boundary-induced long tails, making object-center and 3D-size estimation difficult.
  • The proposed remedies enforce projected 2D-3D box overlap and use instance masks instead of boxes to reduce irrelevant frustum points.
  • 21.0% AP at IoU 0.7 on KITTI moderate 3D detection nearly quadruples the prior 5.7% result, while 48.3% AP at IoU 0.5 exceeds 42.3% by 6.0%.

2. Related Work

Prior work processes LiDAR through bird’s-eye-view images, 3D voxels, or point-based methods, while monocular approaches infer 3D structure from a single image. Pseudo-LiDAR methods narrow this gap, and this paper addresses their noise with instance masks and bounding-box consistency.

  • LiDAR-Based 3D Object Detection: LiDAR-based 3D detectors process point clouds through bird’s-eye-view images, 3D voxels, or point-based architectures.These approaches exploit explicit 3D information from LiDAR data.
  • Monocular 3D Object Detection: Monocular detectors use a single image and have explored 3D proposal scoring, ROI lifting, instance depth estimation, and fine-grained part-based models.These methods avoid precise LiDAR but must infer missing depth or object extent from image evidence.
  • Supervision via Consistency: Consistency-based methods have used 2D–3D box constraints, but this paper trains a PointNet-based detector to predict the complete 3D box rather than deriving center location from partial estimates.The cited prior method estimates size and orientation with a 2D CNN, then computes the object center during testing.
  • Monocular 3D Object Detection: This work differs from earlier pseudo-LiDAR methods by addressing local misalignment and long-tail noise with bounding-box consistency and instance-mask proposals.It reports absolute AP improvements of up to 21.2% over and 6.0% over [52].

4. Experiments

Experiments on KITTI evaluate the monocular pipeline against prior methods and isolate the effects of its proposal, consistency, segmentation, and training choices. The method leads monocular results, while ablations show benefits from instance masks and bounding-box consistency.

  • Comparison with State-of-the-Art Methods: The method outperforms all monocular methods on KITTI bird’s-eye-view and 3D detection across difficulty levels and metrics.At IoU = 0.7 on the moderate set, AP3D rises from 5.7 by MLF-MONO to 21.0 by ours.
  • Comparison with State-of-the-Art Methods: 21.0 AP3D nearly quadruples the prior 5.7 AP3D at IoU = 0.7 on the moderate KITTI set.The comparison is with MLF-MONO.
  • Comparison with State-of-the-Art Methods: 48.3% AP3D improves over PL-MONO’s 42.3% by up to 6.0% at IoU = 0.5 on the moderate set.This is reported as an absolute AP3D improvement over the best-performing concurrent work.
  • Results on Pedestrian and Cyclist: Pedestrian and cyclist detection remains significantly worse than car detection, although the authors present it as a starting point for future monocular work.Prior monocular works had not reported pedestrian and cyclist results in the cited comparison.
  • Ablation Study: Instance-mask proposals improve performance by about 1–2% AP over bounding-box proposals on nearly all subsets.The exception is the easy set at IoU = 0.5.
  • Ablation Study: Bounding-box consistency improves moderate-set AP3D from 42.5% to 46.6% at IoU = 0.5.Both the training loss BBCL and testing optimization BBCO are reported to improve performance significantly.
  • Ablation Study: Removing the 3D segmentation loss causes a significant performance drop, whereas increasing sampled points from 512 to 4096 does not improve performance.Removing TNet yields a mild improvement at IoU = 0.7 but no obvious difference at IoU = 0.5.
  • Ablation Study: Training from scratch slightly outperforms initialization from Frustum PointNets’ pre-trained weights.The authors attribute the drop to difficulty adapting weights over-fitted to LiDAR input.

5. Conclusion

The paper proposes monocular 3D detection by lifting images into pseudo-LiDAR and refining noisy predictions with consistency constraints and instance-mask proposals. On KITTI, it ranks first among monocular methods for bird’s-eye-view and 3D detection.

  • Conclusion: The pipeline lifts a single image into a pseudo-LiDAR point cloud and enhances LiDAR-based detection without 3D sensors.It addresses pseudo-LiDAR noise using 2D–3D box consistency and instance-mask proposals.
  • Conclusion: The method ranks first among monocular methods on KITTI bird’s-eye-view and 3D object detection, quadrupling prior state-of-the-art performance.The conclusion also states that the approach can be extended to stereo image input.

Supplementary Material

The supplementary material identifies Xinshuo Weng as affiliated with Carnegie Mellon University and gives the paper’s arXiv record dated 31 August 2019.

  • Xinshuo Weng is affiliated with Carnegie Mellon University.
  • The paper is identified as arXiv:1903.09847v4 in the cs.CV category.
  • The cited arXiv version is dated 31 August 2019.

0. Overview

The overview material states that the supplementary document expands the paper with technical details, experiments, visualizations, and justification aligned to the main approach.

  • Overview: Each supplementary section corresponds to a subsection of the main paper’s approach section.The supplement covers additional technical details, experiments, visualizations, and justification.

1. Pseudo-LiDAR Generation

The method compares LiDAR and pseudo-LiDAR point clouds, highlighting local misalignment and long-tail artifacts in pseudo-LiDAR.

  • Pseudo-LiDAR exhibits local misalignment and long-tail issues relative to LiDAR point clouds.These artifacts are demonstrated through an additional visual comparison.

2. 2D Instance Mask Proposal Detection

Instance mask proposals generate point-cloud frustums with less background contamination than bounding-box proposals, improving downstream 3D segmentation and box estimation despite weaker 2D proposal performance.

  • 2. 2D Instance Mask Proposal Detection: Instance mask proposals produce point-cloud frustums without the long tail present in bounding-box frustums.The reduced tail makes subsequent 3D point-cloud segmentation easier and improves 3D box estimation.
  • 2. 2D Instance Mask Proposal Detection: The proposal comparison evaluates car-category AP2D on the KITTI validation set at IoU thresholds of 0.5 and 0.7.The evaluation uses the minimum bounding rectangle of the mask proposals.
  • 2. 2D Instance Mask Proposal Detection: The mask proposal’s minimum bounding rectangle performs worse for 2D detection because KITTI has far fewer instance-mask annotations than bounding-box annotations.Only 200 images have instance masks, compared with 7500 images annotated with bounding boxes.
  • 2. 2D Instance Mask Proposal Detection: Despite weaker 2D proposal performance, instance mask proposals achieve higher bird's eye view and 3D object detection performance than bounding-box proposals.The comparison is reported between Ours (baseline) and Ours+Mask in Table 3 of the main paper.

3. Amodal 3D Object Detection

The 3D box estimation and correction modules use PointNet-based processing of point clouds, with the correction module predicting residual parameter adjustments; qualitative figures compare predictions with ground truth and BBC variants.

  • 3. Amodal 3D Object Detection: Qualitative results visualize predicted and ground-truth 3D boxes on frontal images and pseudo-LiDAR point clouds.Predictions are shown in blue and ground truth in red.
  • 3. Amodal 3D Object Detection: Additional visualizations compare 3D box estimates with and without bounding box consistency against ground truth.The comparison isolates the effect of BBC on the box estimates.
  • 3. Amodal 3D Object Detection: The 3D box correction module uses a PointNet-based network and predicts residuals rather than complete 3D box parameters.It concatenates estimation-module features with correction-module global features to predict the residual parameters.
  • 3. Amodal 3D Object Detection: Both the 3D box estimation and correction modules take a point cloud as input.The estimation module outputs full 3D box parameters, while the correction module outputs parameter residuals.

4. 2D-3D Bounding Box Consistency (BBC)

Bounding box consistency improves predicted 3D boxes by increasing their 3D IoU with ground truth in qualitative comparisons.

  • 4. 2D-3D Bounding Box Consistency (BBC): Bounding box consistency improves 3D IoU compared with estimates produced without BBC.The improved results use both bounding-box consistency loss during training and consistency optimization during post-processing.

5. Experiments

Additional qualitative results in Figure 4 show accurate car 3D bounding-box detection from a single RGB image, including challenging faraway objects.

  • Figure 4 provides additional qualitative results for car 3D bounding-box detection.
  • Single-RGB-image input can produce very accurate 3D bounding-box detection for cars.
  • The qualitative results include challenging faraway objects, such as examples in the sixth-row first-column and eighth-row second-column positions.
Loading 1903.09847v4…