PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Pseudo-Lidar needed for Monocular 3D Object detection?

Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, Adrien Gaidon

arXiv:2108.06417v1cs.CV

TL;DR

Pseudo-lidar methods benefit from improved monocular depth but remain vulnerable to depth errors, generalization issues, and limited scalability in end-to-end detectors. DD3D combines depth pre-training with single-stage end-to-end detection and achieves state-of-the-art accuracy on KITTI-3D and nuScenes.

  • Problem

    Pseudo-lidar detectors improve with monocular depth quality, but depth errors and generalization issues remain, while single-stage detectors lack comparable unsupervised pre-training scalability.

  • Method

    DD3D is a fully convolutional single-stage monocular 3D detector that leverages large-scale monocular depth pre-training and end-to-end fine-tuning.

  • Results

    DD3D achieves state-of-the-art monocular 3D detection on KITTI-3D and nuScenes, including 16.34% 3D AP on Moderate Cars on KITTI-3D.

  • Takeaways & Limitations

    Depth pre-training allows DD3D to scale with unlabeled data while retaining the simplicity and end-to-end optimization of monocular detection.

  • Takeaways & Limitations

    Pseudo-lidar depth pre-training gains do not transfer to benchmark results without in-domain fine-tuning, creating a practical data-curation hurdle.

Abstract

from arXiv · show

Recent progress in 3D object detection from single images leverages monocular depth estimation as a way to produce 3D pointclouds, turning cameras into pseudo-lidar sensors. These two-stage detectors improve with the accuracy of the intermediate depth estimation network, which can itself be improved without manual labels via large-scale self-supervised learning. However, they tend to suffer from overfitting more than end-to-end methods, are more complex, and the gap with similar lidar-based detectors remains significant. In this work, we propose an end-to-end, single stage, monocular 3D object detector, DD3D, that can benefit from depth pre-training like pseudo-lidar methods, but without their limitations. Our architecture is designed for effective information transfer between depth estimation and 3D detection, allowing us to scale with the amount of unlabeled pre-training data. Our method achieves state-of-the-art results on two challenging benchmarks, with 16.34% and 9.28% AP for Cars and Pedestrians (respectively) on the KITTI-3D benchmark, and 41.5% mAP on NuScenes.

1. Introduction

Monocular 3D detection seeks accurate 3D localization from ubiquitous cameras, while pseudo-lidar methods gain scalability from depth estimation but face depth-error and generalization problems. DD3D addresses this trade-off with a single-stage architecture that combines depth pre-training with end-to-end detection and achieves state-of-the-art benchmark performance.

  • Pseudo-lidar detectors first estimate depth, convert it into an intermediate pointcloud, and then apply a 3D detection network.
  • Depth estimation quality enables pseudo-lidar methods to improve through large-scale training on raw data.
  • Depth regression from single images is ill-posed, and its errors account for much of the gap between pseudo-lidar and lidar-based detectors.
  • DD3D is a fully convolutional single-stage detector designed to combine pseudo-lidar scalability with end-to-end simplicity and generalization.
  • DD3D scales with large unlabeled depth datasets, and depth pre-training outperforms equal-sized COCO pre-training.
  • DD3D sets a new state of the art on KITTI-3D and nuScenes, with significant improvements over previous state-of-the-art methods.

2. Related work

Related work develops monocular 3D detection through geometric and shape-based cues, while pseudo-lidar methods use monocular depth to construct pointclouds for 3D detection. Monocular depth estimation itself includes supervised and self-supervised approaches, including training on large raw datasets without human annotations.

  • Image-based 3D detectors lift 2D detections into 3D using object-shape and scene-geometry cues.
  • Some methods align 2D keypoints with projective 3D counterparts or learn low-dimensional shape representations.
  • Other approaches enforce geometric consistency between 2D and 3D structures through constrained optimization.
  • Pseudo-lidar methods use monocular depth prediction to transform dense depth maps into 3D pointclouds for lidar-based 3D detection.
  • Monocular depth estimation supports both DD3D pre-training and pseudo-lidar pipelines, using supervised and self-supervised training methods.
  • The cited supervised monocular depth training requires no human annotations, allowing scaling to large amounts of raw data.

3. Dense depth pre-training for 3D detection

DD3D is a camera-aware, fully convolutional single-stage detector that jointly predicts 3D boxes and dense depth while sharing nearly all parameters between the two paths. Its depth design supports metric decoding, stable training, and transfer from depth pre-training.

  • Architecture: DD3D extends FCOS into a fully convolutional single-stage network for monocular 3D detection and dense depth prediction.
  • Architecture: The classification, 2D box, and 3D detection heads operate on every multi-scale FPN feature map.
  • 3D detection head: The 3D head predicts box geometry, orientation, confidence, and two depth quantities: foreground center depth and per-pixel closest-surface depth.
  • Camera awareness: Camera focal lengths and pixel size are incorporated into metric-depth decoding, making DD3D camera-aware and supporting stable training under image resizing.
  • Depth pre-training: Depth pre-training uses per-pixel predictions from all FPN levels and an L1 loss on pixels with valid projected Lidar depth.Using all FPN levels supports stable training, particularly from scratch and with large-batch, high-resolution inputs.
  • Information transfer: The image-to-box and image-to-depth paths differ only in their final 3×3 convolution, enabling transfer from the pre-trained representation to 3D detection.

4. Pseudo-Lidar 3D detection

Pseudo-Lidar uses a modular two-stage pipeline: monocular depth prediction produces a dense point cloud, which a Lidar-based detector then converts into 3D boxes. The setup isolates the contribution of large-scale image-Lidar data to depth prediction.

  • Pipeline: Pseudo-Lidar first predicts per-pixel depth from an image, transforms the dense depth map into a 3D point cloud, and applies a Lidar-based 3D detector.
  • Motivation: Its modularity enables measurement of how large-scale image-Lidar data improves the depth network component.
  • Depth estimation: The reference depth network is PackNet, trained with the SILog loss because it performs better than L1 for this model.
  • 3D detection: The PL detector links predicted 3D confidence to 3D box-coordinate error through a binary cross-entropy loss.

5. Experimental Setup

The experiments use KITTI-3D and nuScenes for evaluation, KITTI-Depth for debiased depth fine-tuning, and DDAD15M for large-scale pre-training. Training and evaluation include specified augmentation and repeated-run procedures.

  • Benchmarks: KITTI-3D evaluates Car, Pedestrian, and Cyclist detection using revised R40 3D AP and BEV AP metrics.
  • Benchmarks: nuScenes contains 1000 six-camera videos split into 700 training, 150 validation, and 150 testing videos, evaluating ten object classes.
  • Datasets: KITTI-Depth contains over 93 thousand depth maps, and the Eigen-clean split removes training images geographically close to KITTI-3D images to reduce overlap bias.
  • Pre-training data: DDAD15M provides approximately 15M image frames from 25,000 multi-camera urban-driving videos with synchronized six-camera and high-resolution Lidar data.
  • Training: DD3D initializes its backbone from COCO 2D detection and pre-trains dense depth using DDAD15M.
  • Evaluation procedure: Test-time resizing and flipping improves KITTI validation Car BEV AP by 2.3% but provides no nuScenes validation improvement.
  • Evaluation procedure: DD3D metrics are averaged over four training runs, with observed BEV AP variance of 0.5 ∼1.2%.

6. Results

DD3D achieves strong monocular 3D detection results on KITTI-3D and nuScenes while benefiting from large-scale depth pre-training in an end-to-end architecture. Ablations show that depth pre-training improves detection, scales with more data, and avoids the in-domain fine-tuning requirement of pseudo-lidar detectors.

  • KITTI-3D: 16.34% 3D AP on Moderate Cars represents a 23% improvement over the previous best KITTI-3D method.The previous best method achieved 13.25% 3D AP.
  • KITTI-3D: 9.30% vs. 5.14% 3D AP on KITTI-3D Pedestrians gives DD3D an 80.5% improvement over the previous best method.DD3D outperforms all other approaches on the Pedestrian category.
  • nuScenes: 41.8% vs. 35.8% mAP on nuScenes gives DD3D a 17% improvement over the previously best published method.DD3D also surpasses PointPillars, a lidar-based detector.
  • nuScenes: 103.7% and 35.5% relative improvements over the previous best method occur at 0.5m and 1.0m distance thresholds, respectively, across three nuScenes classes.The stricter 0.5m criterion shows the larger average gain.
  • Depth pre-training: Removing depth pre-training reduces Car Moderate BEV AP by 5.3% for DLA-34 and 10.7% for V2-99, while removing initial COCO pre-training costs 2.0%.These results come from the DDAD15M ablation on DD3D.
  • Depth pre-training: Depth pre-training improves detection as the pre-training corpus grows from 0.6M to 15M images, with DD3D and PL performing similarly at each checkpoint.The experiment uses subsampled DDAD15M videos and evaluates both detectors after KITTI-3D fine-tuning.
  • Pseudo-lidar comparison: 30.1% →19.1% BEV AP shows that in-domain depth fine-tuning is crucial for the evaluated pseudo-lidar detector.Without adaptation, the depth network is trained only on DDAD15M and directly applied to KITTI-3D; end-to-end DD3D does not require this step.

7. Conclusion

DD3D is an end-to-end, single-stage detector designed to retain pseudo-Lidar’s depth-data scaling while avoiding its impractical training and generalization limitations. It achieves strong accuracy on two challenging 3D detection benchmarks.

  • DD3D is an end-to-end, single-stage 3D object detector that uses large-scale depth data during pre-training.
  • DD3D and pseudo-Lidar methods are compared using pre-training subsets containing 0.6M, 3M, 6M, and 15M DDAD15M images.
  • DD3D aims to combine pseudo-Lidar’s accuracy scaling with large-scale depth data and end-to-end detectors’ simpler training and better generalization.
  • DD3D achieves excellent accuracy on the KITTI-3D and nuScenes 3D detection benchmarks.

A. Details of training DD3D and PL

The training setup specifies separate pre-training and 3D-detection schedules for DD3D and PL, including batch sizes, optimization steps, learning-rate decay, and input resizing.

  • During depth pre-training, DD3D uses batch size 512 for 375K steps, with learning-rate decay at 305K and 365K steps.The learning rate starts at 0.02 and is reduced by 0.1 at each decay point.
  • Depth pre-training uses 1600 × 900 inputs resized to 910 × 512 while preserving sparse non-zero depth values.
  • For 3D detection, the learning rate starts at 0.002 and decays by 0.1 at 85% and 95% of training.
  • DD3D uses batch size 64 and trains for 25K steps on KITTI-3D and 120K steps on nuScenes.

B. DD3D architecture details

DD3D uses multiscale feature pyramids whose resolution levels are completed differently for DLA-34 and V2-99 backbones. Depth performance is evaluated alongside PL detection performance on KITTI-3D.

  • DLA-34 FPN features use strides 8, 16, and 32 before two additional convolutions add strides 64 and 128.
  • Figure 5 compares depth absolute relative error with PL Car Mod. 3D AP|R40 at each pre-training step on KITTI-3D validation.
  • V2-99 provides strides 4, 8, 16, and 32, so one additional convolution completes the five-level feature pyramid.

C. Pseudo-Lidar 3D confidence head

The PL 3D confidence head predicts distance-specific 3D boxes and confidence scores using three separate heads for near, medium, and far objects.

  • The PL detector separates predictions into near-, medium-, and far-distance heads.
  • Each head outputs 3D bounding-box parameters and a 3D confidence trained through the 3D bounding-box loss.
  • Each estimation head contains fully connected layers with dimensions 512 → 512 → 256 → (δ, γ).Here, δ denotes bounding-box parameters and γ denotes 3D bounding-box confidence.

D. The impact of data on Pseudo-Lidar depth and 3D detection accuracy

The analysis finds an almost perfect linear relationship between monocular depth quality and 3D detection accuracy for the Pseudo-Lidar detector.

  • Depth quality, measured by abs rel, has an almost perfect linear relationship with 3D detection accuracy in the Pseudo-Lidar detector.
Loading 2108.06417v1…