PaperlayerView exact revision on arXiv

Source-linked AI summary

DSGN: Deep Stereo Geometry Network for 3D Object Detection

Yilun Chen, Shu Liu, Xiaoyong Shen, Jiaya Jia

arXiv:2001.03398v3cs.CV

TL;DR

Image-based 3D detection has lagged LiDAR-based methods because forming effective 3D prediction representations is difficult. DSGN uses a differentiable 3D geometric volume transformed from a plane-sweep volume to jointly estimate depth and detect objects end to end. It outperforms prior stereo-based detectors and achieves comparable performance with several LiDAR-based methods on KITTI.

  • Problem

    Image-based 3D detection trails LiDAR-based methods, with effective 3D representation formation identified as a central challenge.

  • Method

    DSGN transforms a plane-sweep volume into a differentiable 3D geometric volume that jointly learns stereo correspondence, depth, and 3D object detection.

  • Results

    10 points higher in AP: DSGN outperforms other stereo-based 3D object detectors and achieves comparable performance with several LiDAR-based methods on KITTI.

  • Takeaways & Limitations

    End-to-end stereo-based 3D object detection is feasible and effective, with DSGN learning geometric structure alongside pixel- and high-level features.

  • Takeaways & Limitations

    The authors state that understanding how 3D volume transformation works remains limited and requires further study.

Abstract

from arXiv · show

Most state-of-the-art 3D object detectors heavily rely on LiDAR sensors because there is a large performance gap between image-based and LiDAR-based methods. It is caused by the way to form representation for the prediction in 3D scenarios. Our method, called Deep Stereo Geometry Network (DSGN), significantly reduces this gap by detecting 3D objects on a differentiable volumetric representation -- 3D geometric volume, which effectively encodes 3D geometric structure for 3D regular space. With this representation, we learn depth information and semantic cues simultaneously. For the first time, we provide a simple and effective one-stage stereo-based 3D detection pipeline that jointly estimates the depth and detects 3D objects in an end-to-end learning manner. Our approach outperforms previous stereo-based 3D detectors (about 10 higher in terms of AP) and even achieves comparable performance with several LiDAR-based methods on the KITTI 3D object detection leaderboard. Our code is publicly available at https://github.com/chenyilun95/DSGN.

1. Introduction

DSGN addresses the representation gap in image-based 3D detection with a differentiable 3D geometric volume that jointly supports stereo depth estimation and object detection. Its one-stage stereo pipeline outperforms prior stereo-based detectors on KITTI.

  • Motivation: LiDAR enables accurate 3D information retrieval but has sparse resolution and high device cost, motivating image-based alternatives.Cameras offer cheaper sensing with denser resolutions, but image-based 3D detection remains behind LiDAR-based approaches.
  • Challenges: Image-based detectors struggle to form effective 3D representations because 3D geometry must be inferred from 2D projections with pose-dependent appearance changes.Existing approaches either separate depth prediction from detection or transform intermediate point clouds through non-differentiable pipelines.
  • Our Solution: DSGN transforms a plane-sweep volume from the camera frustum into a 3D geometric volume defined in world space.The plane-sweep volume supports learned pixel correspondence, while the 3D geometric volume supports learning features for real-world objects.
  • Our Solution: The proposed volume encodes 3D geometry and semantic cues in regular 3D space while preserving differentiable transformation for joint optimization.This representation supports both stereo matching and object detection within one jointly trained pipeline.
  • Results: 10 points higher in AP: DSGN outperforms all other stereo-based 3D object detectors on the official KITTI leaderboard.The reported result comes from a simple, fully differentiable network without additional bells and whistles.

2. Related Work

Prior work spans stereo matching, multi-view stereo, LiDAR-based detection, and image-based 3D detection. Image-based methods differ in whether they use intermediate depth or 3D representations before detection.

  • Stereo Matching: Stereo matching methods construct 3D cost volumes from Siamese image processing, correlation, concatenation, and 3D convolution.GC-Net regresses disparity from a concatenation-based cost volume, while PSMNet improves accuracy with pyramid pooling.
  • Multi-View Stereo: Multi-view stereo methods reconstruct 3D objects from multiple views using plane-sweep volumes, point clouds, or differentiable projection and unprojection.MVSNet generates per-view depth from camera-frustum plane-sweep volumes, while Point-MVSNet transforms them into point clouds.
  • LiDAR-based 3D Detection: LiDAR-based 3D detectors process point clouds with either voxel-based or point-based architectures.These methods exploit LiDAR's strong 3D sensing capability for object detection.
  • Image-based 3D Detection: Image-based 3D detectors include depth-predictor pipelines and methods that detect from intermediate 3D representations.The former combine 2D detection with monocular or stereo depth extraction; the latter include point-cloud, Pseudo-LiDAR, and bird's-eye-view representations.

3. Our Approach

DSGN addresses the difficulty of representing image-derived 3D structure by learning stereo correspondence in a plane-sweep volume and transforming it into a differentiable 3D geometric volume. This representation supports joint depth estimation and 3D object detection in regular 3D space.

  • Motivation: Image-based 3D detection is difficult because perspective, varying object pose, and foreshortening make stable 3D information hard to extract from 2D features.These effects make similarly categorized objects appear with different sizes, orientations, and distorted shapes.
  • Motivation: Point-based pipelines can require independent depth and detection networks, potentially lose information during transformation, and produce streaking artifacts near object edges.Voxel-based OFT-NetMono also does not explicitly encode the 3D geometry of its data.
  • Network Overview: DSGN extracts image features, constructs a plane-sweep volume, learns pixel correspondence and depth there, then differentiably warps it into a 3D geometric volume for detection.The pipeline uses a Siamese feature extractor for binocular images and a subsequent 3D network for object detection.
  • Volume Representation: The 3D geometric volume is formed by warping a plane-sweep feature map into a regular voxel grid defined in 3D world space.The grid dimensions represent width, height, and depth along camera-view directions, with fixed voxel sizes.
  • Volume Representation: The plane-sweep volume preserves pixel-correspondence learning along projection rays, while the transformed volume provides geometric structure for learning real-world 3D features.The transformation lifts geometric information from the cost volume rather than directly unprojecting image features, and preserves depth distribution instead of reducing it to a depth map.
  • 3D Detection: The 3D detector progressively downsamples the 3D volume along height to produce a bird’s-eye-view feature map for anchor-based object prediction.Anchors at each bird’s-eye-view location use different orientations and sizes for box regression.

4. Experiments

Experiments evaluate DSGN on KITTI under official and validation protocols, comparing its detection performance, volume-construction choices, supervision, and depth estimation.

  • Evaluation setup: The KITTI training split contains 7,481 stereo image-pairs and point clouds, while testing uses 7,518 samples annotated for Car, Pedestrian, and Cyclist.Ground-truth depth maps are generated from point clouds.
  • Evaluation setup: DSGN is evaluated on KITTI using easy, moderate, and hard settings with class-specific IoU thresholds for 2D, BEV, and 3D detection.The main index is the moderate difficulty level; the official leaderboard uses 40 recall positions.
  • Main results: DSGN outperforms other image-based methods on KITTI 3D and BEV detection, despite training one unified network from scratch on roughly 7K training samples.Competing Pseudo-LiDAR approaches use separate networks and extra Scene Flow pre-training, while Stereo R-CNN uses ImageNet-pretrained ResNet-101.
  • Main results: DSGN achieves comparable BEV and better 3D detection performance than LiDAR-only MV3D on the KITTI easy regime.The paper presents this as the first such comparison for that setting and notes low-speed autonomous driving as a promising application.
  • Supervision: Point-cloud supervision consistently improves performance, while stereo approaches outperform monocular approaches under supervision.The comparisons indicate that 3D-box-only supervision is insufficient for learning 3D geometry.
  • 3D volume construction: Plane-sweep cost volume reaches 54.27 AP versus 45.89 AP for disparity-based cost volume, indicating the importance of balanced feature mapping during 3D-volume transformation.The ablation also reports that plane-sweep volume more effectively contains depth information.
  • Depth estimation: Joint training yields similar depth errors for DSGN and PSMNet-PSV*—0.5586 meter versus 0.5606 meter—while DSGN gains 7.86 AP in 3D detection and 6.34 AP in BEV detection.The result indicates stronger high-level semantic features for object detection in DSGN’s 2D network.

5. Conclusion

DSGN demonstrates that end-to-end stereo-based 3D detection is feasible by transforming plane-sweep volumes into 3D geometric volumes and jointly learning stereo correspondence and object detection. The one-stage approach outperforms image-based methods and reaches comparable performance with some LiDAR-based detectors.

  • Conclusion: DSGN is a binocular-image 3D detector that transforms a plane-sweep volume into a 3D geometric volume for learning geometric structure features.The representation supports joint learning of pixel-level stereo correspondence and high-level 3D object detection features.
  • Conclusion: End-to-end stereo-based 3D object detection is feasible and effective within DSGN’s unified network.The conclusion attributes this capability to encoding 3D geometry in the transformed volume.
  • Conclusion: DSGN outperforms other image-based approaches and achieves comparable performance with a few LiDAR-based approaches on 3D object detection.The paper also reports ablation evidence for several key components of 3D-volume training.
  • Limitations and future work: Further work is needed to understand how 3D-volume transformation works, despite clear and explained improvements in the ablation study.This is the conclusion’s stated future direction.

A.1. More Experiments

The experiments examine how depth estimation, geometric representations, optimization choices, and distance affect stereo-based 3D detection. They also compare performance across object categories and analyze relationships between depth precision and detection accuracy.

  • Depth and detection accuracy: At an outlier threshold of 0.3 meter, depth estimation precision has the strongest linear correlation with detection accuracy.The analysis uses Pearson’s correlation coefficients across several outlier thresholds.
  • Depth and detection accuracy: Over 0.7 detected precision is achieved by some predictions despite larger depth estimation errors.The 3D location regression can compensate for stereo depth estimation errors through back-propagation.
  • Distance and detection accuracy: Average detection accuracy remains above 80% within 25 meters, while 3D AP declines fastest as distance increases.Beyond 20 meters, the reported observation suggests that 3D detection accuracy is determined by BEV location precision.
  • 3D geometric representation: Latent last features outperform explicit occupancy or probability representations by providing geometric and semantic information for the 3D volume.The ablation defines occupancy, probability, and transformed final cost-volume features as alternative geometric representations.
  • Technical details: +4.80 AP comes from joint bounding-box regression optimization, while depth-probability-weighted image features add +1.01 AP.Adding an hourglass module and flip augmentation yields a final 54.27 AP on 3D object detection.
  • Pedestrian and Cyclist detection: The approach performs better on Pedestrian but worse on Cyclist than PL: F-PointNet.PL: F-PointNet used Scene Flow pre-training, while PL: AVOD performs best on Car and Cyclist in the cited comparisons.

A.2. More Implementation Details

The implementation uses a multi-stage stereo-to-3D pipeline with bird’s-eye-view anchors and differentiable warping from camera-frustum features into world-space geometry.

  • Network Architecture: The full architecture includes 2D feature extraction, plane-sweep and 3D geometric volume construction, stereo matching, and 3D object detection.These components are listed as the networks comprising the full architecture.
  • Implementation Details of 3D Object Detector: Four orientation-specific anchors are placed at every bird’s-eye-view feature-map location for 3D detection.The orientations are 0, π/2, π, and 3π/2, with category-specific predefined box sizes.
  • Implementation Details of 3D Object Detector: Anchor centers lie on bird’s-eye-view grid centers, with category-specific vertical positions and positive-negative sample balancing factors.The detector applies NMS with IoU threshold 0.6 to filter predicted boxes.
  • Implementation Details of Differentiable Warping: Differentiable warping projects 3D voxel centers into camera-frustum coordinates and samples plane-sweep features at the projected locations.The projected coordinates are formed using the projection matrix and concatenated depth coordinates.
  • Implementation Details of Differentiable Warping: Trilinear interpolation constructs the 3D geometric volume, while out-of-image voxels receive zero features and backward gradients use the same coordinate volume.The resulting volume has the spatial dimensions of the predefined 3D volume and the feature channel size of the plane-sweep representation.

A.3. Future Work

Future work targets the remaining performance gap with LiDAR methods, more effective and efficient 3D volume construction, computational bottlenecks, architecture trade-offs, and broader deployment settings.

  • Gap with state-of-the-art LiDAR-based approaches: A nearly 12 AP BEV detection gap remains at moderate and hard levels compared with state-of-the-art LiDAR-based approaches.The paper reports comparable performance with some LiDAR-based methods but a large gap with state-of-the-art approaches.
  • Gap with state-of-the-art LiDAR-based approaches: High-resolution stereo matching is proposed to improve depth robustness for highly occluded, truncated, and distant objects.This is presented as one possible direction for narrowing stereo-based detection weaknesses.
  • 3D Volume Construction: More analysis is needed to determine how 3D volume construction balances depth and semantic information.The paper identifies multi-view or binocular volume construction as an essential component for 3D object understanding.
  • Computation Bottleneck: Computing 3D convolutions for cost-volume processing is identified as DSGN’s main computational bottleneck.Future work could explore more efficient feature encoding for plane-sweep cost volumes designed for depth rather than disparity regression.
  • Network Architecture Design: The stereo-matching and 3D-detection networks require a feature-extraction trade-off, motivating possible use of neural architecture search.The paper also identifies application to low-speed scenarios as a future direction.

A.4. Qualitative Results

The qualitative results visualize DSGN’s 3D detections and predicted depth, while also documenting depth-map noise associated with the modeled range and sparse training supervision.

  • Qualitative Results: The video demo shows detected 3D boxes from DSGN in both front-view and bird’s-eye-view visualizations.The model is trained on the KITTI training split only, and the depth-map unit is meters.
  • Qualitative Results: Predicted depth noise appears near the modeled range boundaries because 3D volumes are constructed from 2 to 40.4 meters.The passage describes noise in both near and far regions of the predicted depth map.
  • Qualitative Results: Noise and a large white region above 3 meters are associated with training the stereo branch on sparse ground-truth depth from a 64-ray LiDAR.The sparse supervision covers 64 lines around the gravitational z-axis from -1 to 3 meters.
Loading 2001.03398v3…