PaperlayerView exact revision on arXiv

Source-linked AI summary

Stereo R-CNN based 3D Object Detection for Autonomous Driving

Peiliang Li, Xiaozhi Chen, Shaojie Shen

arXiv:1902.09738v2cs.CVcs.RO

TL;DR

Image-based 3D detection needs accurate depth without relying on costly, sparse LiDAR or uncertain monocular depth. Stereo R-CNN combines stereo detection and association with sparse geometric constraints and dense RoI photometric alignment to estimate and refine 3D boxes, outperforming existing image-based methods on KITTI. Its performance nevertheless decreases as object distance increases because depth error grows as disparity decreases.

  • Problem

    The paper targets accurate autonomous-driving 3D detection from images, where LiDAR is costly and sparse and monocular depth can be inaccurate on unseen scenes.

  • Method

    Stereo R-CNN simultaneously detects and associates stereo objects, predicts boxes, keypoints, dimensions, and viewpoint, then estimates and photometrically aligns their 3D boxes.

  • Results

    Stereo R-CNN outperforms all existing image-based methods on KITTI 3D detection and localization and is better than a baseline LiDAR method.

  • Takeaways & Limitations

    The approach demonstrates that stereo imagery can support accurate image-based 3D detection without depth input or 3D position supervision.

  • Takeaways & Limitations

    KITTI image-based 3D performance decreases as object distance increases because depth error grows as disparity decreases.

Abstract

from arXiv · show

We propose a 3D object detection method for autonomous driving by fully exploiting the sparse and dense, semantic and geometry information in stereo imagery. Our method, called Stereo R-CNN, extends Faster R-CNN for stereo inputs to simultaneously detect and associate object in left and right images. We add extra branches after stereo Region Proposal Network (RPN) to predict sparse keypoints, viewpoints, and object dimensions, which are combined with 2D left-right boxes to calculate a coarse 3D object bounding box. We then recover the accurate 3D bounding box by a region-based photometric alignment using left and right RoIs. Our method does not require depth input and 3D position supervision, however, outperforms all existing fully supervised image-based methods. Experiments on the challenging KITTI dataset show that our method outperforms the state-of-the-art stereo-based method by around 30% AP on both 3D detection and 3D localization tasks. Code has been released at https://github.com/HKUST-Aerial-Robotics/Stereo-RCNN.

1. Introduction

Stereo R-CNN addresses image-based 3D detection by combining stereo imagery’s semantic and geometric information with sparse box/keypoint constraints and dense photometric alignment. It simultaneously detects and associates stereo objects, estimates a coarse 3D box, and refines localization.

  • Motivation: Stereo vision offers more precise depth than monocular inference while remaining lower-cost than LiDAR and potentially supporting longer perception ranges.Stereo perception range depends on focal length and baseline, allowing different stereo modules to cover different ranges.
  • Approach: Stereo R-CNN simultaneously detects and associates objects in left and right images using stereo image information.Its Stereo RPN produces corresponding left-right RoI proposals, which support stereo box, viewpoint, and dimension prediction.
  • Approach: Stereo R-CNN combines stereo boxes, keypoints, dimensions, and viewpoint predictions to estimate a coarse 3D bounding box.The 3D box estimator formulates projection relations between 3D box corners, 2D left-right boxes, and keypoints.
  • Approach: Dense region-based photometric alignment refines the 3D box by finding the center depth that minimizes photometric error across the object RoI.The method warps dense pixels from the left RoI to the right image using depth relations and then rectifies the 3D box.
  • Results: KITTI evaluation shows Stereo R-CNN outperforms state-of-the-art image-based methods and is comparable with a LiDAR-based method.The reported evaluation concerns the proposed image-based stereo approach and its 3D detection performance.

2. Related Work

Prior 3D detection methods primarily use LiDAR or monocular imagery, while relatively few methods exploit stereo vision. Existing stereo approaches use priors, depth cues, or temporal information for proposal generation and pose estimation.

  • LiDAR-based 3D Object Detection: LiDAR-based methods represent point clouds through bird’s-eye/front views, voxel grids, fused modalities, or raw-point processing.These approaches rely on LiDAR to provide accurate 3D information.
  • Monocular-based 3D Object Detection: Monocular methods use ground-plane, shape, contextual, segmentation, geometric, keypoint, or wireframe constraints to estimate 3D objects.The reviewed approaches include proposal generation, 3D box estimation, and regular-shape vehicle modeling.
  • Stereo-based 3D Object Detection: Stereo-based methods include 3DOP, which encodes priors and depth cues in an energy function, and SfM-based tracking that fuses spatial and temporal information.The stereo literature is described as limited compared with LiDAR- and monocular-based work.

3. Stereo R-CNN Network

Stereo R-CNN jointly detects and associates objects across stereo images, then predicts box, dimension, viewpoint, and keypoint information for 3D estimation. Its stereo RPN uses shared anchors and union-box targets to produce corresponding proposals without additional association computation.

  • Stereo RPN: Stereo R-CNN concatenates left-right feature maps in a modified pyramid RPN to generate corresponding stereo proposals.Weight-shared ResNet-101 and FPN extract consistent left-right features before stereo proposal generation.
  • Stereo RPN: Union GT boxes train objectness while separate left-right regressors predict six stereo box offsets, causing positive anchors to contain both object regions.Rectified images share vertical and height offsets, while horizontal coordinates and widths are predicted separately.
  • Stereo RPN: Shared anchors and objectness scores associate left-right proposals naturally, with separate NMS retaining candidates kept in both views.The method selects the top 2,000 candidates for training and top 300 for testing.
  • Stereo R-CNN: Stereo regression predicts object class, stereo boxes, dimensions, and viewpoint angle from concatenated left-right RoI features.The viewpoint is α = θ + β, represented during training by [sin α, cos α] because raw angles are discontinuous.
  • Keypoint Prediction: Keypoint prediction adds four semantic bottom-corner projections and two boundary keypoints to constrain 3D estimation and select pixels for dense alignment.Only one semantic keypoint can project to the box middle; boundary keypoints delimit the object region used later.
  • Keypoint Prediction: The keypoint branch predicts horizontal locations from left RoI features, using softmax outputs for exclusive perspective-keypoint and independent boundary-keypoint predictions.The implementation reduces the output to horizontal coordinates because only u provides information beyond the 2D box.

4. 3D Box Estimation

The method estimates a coarse 3D box by jointly fitting stereo box measurements and a perspective keypoint, then uses the aligned projection for dense refinement. Seven normalized measurements produce geometric constraints solved with Gauss-Newton.

  • Sparse 3D Box Estimation: The coarse 3D box state contains center position and horizontal orientation, estimated by minimizing reprojection error from stereo boxes and a perspective keypoint.The inputs include left-right 2D boxes, the perspective keypoint, and regressed object dimensions.
  • Sparse 3D Box Estimation: Seven normalized measurements encode the left box edges, right-box horizontal edges, and perspective-keypoint horizontal coordinate.Normalizing by camera intrinsics simplifies the representation and allows 3D-box corner correspondences to be inferred.
  • Sparse 3D Box Estimation: Projection equations relate the measurements to stereo baseline, dimensions, depth, and orientation, with corner-dependent signs and truncated edges omitted.The resulting multivariate equations are solved using the Gauss-Newton method.
  • Sparse 3D Box Estimation: Jointly using stereo boxes and regressed dimensions recovers depth more robustly than using a single 2D box and size prior.The viewpoint angle compensates for otherwise unobservable states in some configurations.
  • Sparse 3D Box Estimation: The solved coarse box has accurate 2D projection and image alignment, enabling subsequent dense alignment.Its role is to provide a projection-consistent initialization for later refinement rather than guarantee accurate 3D position by itself.

5. Dense 3D Box Alignment

Dense 3D box alignment refines a coarse stereo-derived box by matching high-resolution pixel information between left and right RoIs. It estimates center depth through photometric consistency and then rectifies the full 3D box.

  • Dense 3D Box Alignment: Stereo boxes provide coarse object-level disparity, but convolutional RoI features lose pixel-level details such as corners and edges.The method retrieves raw image information to support sub-pixel matching accuracy.
  • Dense 3D Box Alignment: The valid RoI is restricted between boundary keypoints and to the bottom halves of the 3D box to reduce background and other-object pixels.The bottom halves of vehicles fit the 3D box more tightly.
  • Dense 3D Box Alignment: Photometric alignment compares left-RoI pixels with right-image samples warped using each pixel’s depth relation to the 3D box center.The method uses RGB vectors, the baseline, and bilinear interpolation for sub-pixel right-image values.
  • Dense 3D Box Alignment: The center depth z is estimated by minimizing the total Sum of Squared Difference matching cost over pixels in the valid RoI.Depth enumeration first uses 50 values at 0.5-meter intervals, then 20 values at 0.05-meter intervals around the rough estimate.
  • Dense 3D Box Alignment: After alignment, the entire 3D box is rectified by fixing the estimated center depth in the 3D box estimator.The RoI is treated as a geometric-constraint entirety rather than as independent pixels.

6. Implementation Details

Implementation uses stereo-specific feature and channel configurations, a multi-task uncertainty-weighted loss, and stereo augmentation. Evaluation is reported on KITTI validation data with 2D and 3D metrics.

  • Implementation Details: Stereo R-CNN uses five scale anchors, three anchor ratios, 600-pixel shorter-side resizing, and expanded channel dimensions for stereo feature concatenation.The final Stereo RPN classification and regression layer uses 1024 input channels, while the R-CNN regression head uses 512.
  • Implementation Details: Table 1 reports RPN average recall and 2D detection average precision on the KITTI validation set.It compares two Stereo R-CNN fusion methods with Faster R-CNN under matched backbone, hyper-parameter, and augmentation settings; recall uses the moderate set.
  • Implementation Details: Table 2 compares bird’s eye view average precision and 3D-box average precision on the KITTI validation set.The metrics are APbv and AP3d.
  • Implementation Details: Training uses a multi-task loss covering stereo boxes, viewpoint, dimensions, and keypoints, with each loss weighted by its uncertainty.The notation distinguishes RPN and R-CNN losses for these prediction targets.
  • Implementation Details: Stereo flip augmentation flips and exchanges the left and right images while mirroring viewpoint angles and keypoints, doubling the training set with different targets.Training keeps one stereo pair and 512 sampled RoIs per mini-batch and uses SGD with weight decay 0.0005.

7. Experiments

Experiments on KITTI evaluate stereo recall, detection, association, 3D detection, and localization. Stereo R-CNN substantially outperforms image-based baselines, with dense alignment and keypoints improving 3D localization.

  • Evaluation setup: KITTI experiments evaluate 2D stereo recall, 2D detection, stereo association, 3D detection, and 3D localization across three difficulty regimes.The benchmark uses separate easy, moderate, and hard settings.
  • Stereo detection and association: Stereo R-CNN preserves single-image detection quality while producing high-quality left-right associations without additional computation.Stereo metrics jointly evaluate detection and association, and accurate stereo association supplies box-level constraints for 3D estimation.
  • 3D detection and localization: Over 30% AP improvement over 3DOP is reported for both APbv and AP3d on easy and moderate KITTI sets.The hard set shows approximately 25% improvements; against VeloFCN, the method exceeds both metrics by approximately 10% AP at IoU = 0.5 on moderate data.
  • Ablation studies: Keypoints improve APbv and AP3D across all difficulty regimes by providing pixel-level constraints beyond 2D box measurements.The ablation compares models with and without keypoints in the 3D box estimator.
  • Ablation studies: Dense alignment and subsequent 3D box rectification improve 3D detection and localization beyond the coarse box estimated from box-level disparity and 2D size.Small disparity or 2D box errors can create large distance errors for distant objects, while alignment and rectification produce further gains.
  • Ablation studies: Stereo flip augmentation and uncertainty weighting each improve performance, and their combination yields strong results on both 3D detection and localization.The two strategies are evaluated through ablations on the KITTI validation set.

8. Conclusion and Future Work

The paper concludes that Stereo R-CNN uses semantic properties and dense geometric constraints for image-based 3D detection without 3D supervision. Its modular framework is presented as extensible to tracking, segmentation-based RoI selection, and general objects.

  • Conclusion: Without 3D supervision, Stereo R-CNN outperforms existing image-based methods on 3D detection and localization and exceeds a baseline LiDAR method.The conclusion frames localization as a learning-aided geometry problem combining semantic properties with dense constraints.
  • Future work: The framework can be extended to multiple-object detection and tracking, instance-segmentation-based RoI selection, and general objects through learned shape.These extensions are proposed as future applications of the modular framework.
Loading 1902.09738v2…