Source-linked AI summary
3D Object Proposals using Stereo Imagery for Accurate Object Class Detection
Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Huimin Ma, Sanja Fidler, Raquel Urtasun
TL;DR
Autonomous-driving systems need accurate 3D object detection from affordable visual sensors, while existing proposals have limited performance under KITTI’s challenging conditions. The paper generates stereo-based 3D proposals with depth-aware energy features and applies a context- and depth-aware CNN to regress 3D boxes and pose. On KITTI, the approach achieves higher proposal recall, outperforms published detection and orientation results across three classes, and performs best when stereo and LIDAR are combined.
Problem
The paper addresses accurate 3D object detection for autonomous driving using stereo imagery, where KITTI’s small, occluded objects and strict localization requirements challenge existing proposals.
Method
The method minimizes an energy function for stereo-based 3D proposals, then uses a contextual CNN with depth information to jointly regress 3D boxes and object pose.
Results
The method outperforms published KITTI detection and orientation results for Car, Cyclist, and Pedestrian, while stereo-LIDAR fusion achieves the highest 3D detection accuracy.
Takeaways & Limitations
Stereo-derived 3D proposals combined with CNN scoring provide accurate object localization, distance, and pose estimation on KITTI.
Takeaways & Limitations
The proposal model assumes a road plane and estimates object height using class-conditional Gaussian statistics.
Abstract
from arXiv · showhide
The goal of this paper is to perform 3D object detection in the context of autonomous driving. Our method first aims at generating a set of high-quality 3D object proposals by exploiting stereo imagery. We formulate the problem as minimizing an energy function that encodes object size priors, placement of objects on the ground plane as well as several depth informed features that reason about free space, point cloud densities and distance to the ground. We then exploit a CNN on top of these proposals to perform object detection. In particular, we employ a convolutional neural net (CNN) that exploits context and depth information to jointly regress to 3D bounding box coordinates and object pose. Our experiments show significant performance gains over existing RGB and RGB-D object proposal methods on the challenging KITTI benchmark. When combined with the CNN, our approach outperforms all existing results in object detection and orientation estimation tasks for all three KITTI object classes. Furthermore, we experiment also with the setting where LIDAR information is available, and show that using both LIDAR and stereo leads to the best result.
1 INTRODUCTION
The paper targets autonomous-driving object detection, where conventional proposals struggle with strict localization requirements, small objects, and occlusion. It introduces stereo-based 3D proposals and a CNN that jointly predicts 3D boxes and pose, with extensive KITTI evaluation.
- Motivation: Conventional object proposals have very low achievable recall on the autonomous-driving benchmark.This limitation forms a bottleneck for object detectors such as R-CNN on KITTI.
- Motivation: Autonomous-driving detection demands stricter localization than PASCAL VOC because distance estimates require accurate vehicle overlap.KITTI objects are also typically small, heavily occluded, or truncated.
- Approach: The proposed method places 3D candidate boxes on the ground plane and scores them using size priors, point density, free space, visibility, and height features.The features can be computed in constant time using 3D integral images.
- Approach: The detection CNN uses contextual information and a multi-task loss to jointly regress 3D bounding-box coordinates and object orientation.The method extends the proposal stage with learned CNN scoring for accurate 3D detections.
- Results: 25% higher recall is achieved than the state-of-the-art RGB-D method MCG-D with 2K proposals.The proposals obtain higher recall across overlap thresholds and occlusion or truncation levels.
- Results: The combined system outperforms published results for object detection and orientation estimation across Car, Cyclist, and Pedestrian classes.Combining stereo and LIDAR produces the highest 3D object detection accuracy.
2 RELATED WORK
Prior work develops object proposals and 3D detection across RGB, RGB-D, and CNN-based approaches. These methods provide context for the paper’s stereo-driven 3D proposal and detection pipeline.
- Object Proposal Generation: RGB proposal methods group superpixels or score densely sampled windows using cues such as color, texture, edges, location, and size.Other approaches learn diverse regions with segmentation models or parametric energies.
- Object Proposal Generation: RGB-D methods extend proposal generation with depth cues, including 3D cubes, RGB-D contours, and depth features.Examples include CPMC-based cuboid proposals, MCG-D 2.5D proposals, and mesh-based candidate objects.
- 3D Object Detection: Earlier 3D detection methods model viewpoint, 3D part geometry, occlusion, context, or voxel-based appearance patterns.These include 3D deformable part models, poselet-based candidates, And-Or models, and 3DVP.
- CNN-Based Detection: The paper’s network can accept either RGB input or a 6-channel RGB-HHA image.RGB-HHA combines appearance with disparity, height above ground, and surface-normal angle channels.
- CNN-Based Detection: CNN-based autonomous-driving detectors use convolutional features for pedestrian detection, part modeling, and complexity-aware cascades.The cited approaches primarily address detection under occlusion or computational constraints.
3 3D OBJECT PROPOSALS
The method generates diverse 3D object proposals from stereo-derived point clouds by sampling candidate boxes and minimizing a learned energy over depth-informed features and object priors.
- Proposal Generation: Stereo imagery is converted into a point cloud, and proposals are represented as 3D boxes placed in autonomous-driving scenes.Proposal generation uses depth information rather than appearance features.
- Energy Function: Each proposal is scored with point-cloud density, free-space occupancy, height prior, and height contrast potentials.These terms favor boxes containing occupied space, avoiding free space, matching class height statistics, and contrasting with surrounding heights.
- Candidate Sampling: Candidate boxes use three size templates per class and two orientations, with ground-plane placement reducing the search space.Templates are clustered from training-set ground-truth 3D boxes, while ground-plane estimation uses superpixel classification and RANSAC plane fitting.
- Inference: For objects beyond 20m, additional sampling planes address noisy stereo depth, while exhaustive scoring and NMS produce diverse top-K proposals.Greedy NMS selects low-energy boxes whose IoU with previously selected proposals stays below δ; inference takes 1.2s per image for 2K proposals.
- Energy Function: The energy weights are learned with a structured SVM, using strict 3D IoU as the training loss.The model can learn class-specific weights for its energy terms.
4 3D OBJECT DETECTION NETWORKS
The network scores 3D proposals using shared convolutional features and jointly predicts object categories, box coordinates, and orientation. It incorporates proposal context and depth features, with specialized regression for full 3D boxes.
- Joint 2D Object Detection and Pose Estimation: Context-aware scoring extends Fast R-CNN with a second branch that processes regions enlarged by a factor of 1.5.The original and context branches use ROI pooling and fully connected layers before feature concatenation.
- Joint 2D Object Detection and Pose Estimation: The network jointly predicts class labels, bounding-box offsets, and object orientation using an equally weighted multi-task loss.Category loss uses cross entropy, while orientation and box-offset losses use smooth ℓ1 loss.
- 3D Object Detection: For 3D detection, the network replaces 2D regressors with 3D regressors that estimate box centers and sizes using scale normalization and log-space shifts.The predicted 3D box is combined with estimated orientation to compute its azimuth angle.
- CNN Scoring with Depth Features: RGB-HHA depth integration is evaluated with both a six-channel single-stream network and a separate two-stream RGB-HHA architecture.HHA encodes disparity, height above ground, and surface-normal angle; the two-stream design has almost double the single-stream model’s parameters.
- Implementation Details: Training uses class-specific proposal weights and KITTI-aligned IoU thresholds of 0.7 for Car and 0.5 for Pedestrian and Cyclist.The same thresholds define proposal-ground-truth overlap for selecting training samples.
5 EXPERIMENTAL EVALUATION
On KITTI, the proposal method is evaluated for recall, efficiency, and downstream 2D, orientation, and 3D detection across object classes and difficulty regimes. It maintains strong recall with relatively few proposals, improves detection and orientation results, and benefits from combining stereo and LIDAR for 3D detection.
- Evaluation Setup: The evaluation uses KITTI’s Car, Pedestrian, and Cyclist classes across Easy, Moderate, and Hard regimes, with oracle recall, AP2D, AOS, AP3D, and ALP metrics.Proposal recall uses class-specific IoU thresholds of 70% for Car and 50% for Pedestrian and Cyclist; 3D recall uses a 0.25 IoU threshold.
- 5.1 Proposal Recall: Using 1000 proposals, the method reaches almost 90% 2D recall for Car in Moderate and Hard regimes, while other methods saturate or require far more proposals.Class-specific proposals slightly outperform the class-independent Ours-G variant, and the method’s recall advantage persists across IoU levels, especially for Cyclist.
- 5.1 Proposal Recall: At distances beyond 40m, the method maintains very high recall while competing proposal methods drop significantly.For 3D recall with 2000 proposals, it achieves around 90% for Car, 80% for Pedestrian, and 60% for Cyclist.
- 5.2 2D Object Detection and Orientation Estimation: The detector improves AP2D and AOS over baseline proposals, achieving around 20% higher scores than Selective Search and EdgeBoxes and 70% AP with 100 proposals.EdgeBoxes and Selective Search reach 25.9% and 16.5% AP, respectively, with 100 proposals, while EdgeBoxes needs 5000 proposals for 78.7% AP and the method needs only 200.
- 5.3 3D Object Detection Performance: For 3D Car detection, the hybrid stereo-LIDAR model achieves 81.21% AP3D and 75.44%/88.83% ALP within 1m/2m in the Moderate regime.Depth-aware RGB-HHA models improve 3D detection over RGB, with the hybrid two-stream model producing the highest reported 3D accuracy in Moderate and Hard settings.
6 CONCLUSION
The paper presents a stereo-based 3D object detection pipeline that generates 3D proposals and refines them with a CNN using appearance, depth, and context. On KITTI, it achieves higher proposal recall and outperforms prior published detection methods across all three object classes.
- The pipeline generates 3D object proposals by minimizing an energy function encoding object size priors, ground-plane context, and depth-informed features.The depth features include reasoning about point-cloud structure and object placement.
- A CNN scores proposals using appearance, depth, and context while jointly predicting 3D bounding-box coordinates and object pose.
- 25% higher recall than the state-of-the-art RGB-D method MCG-D is achieved for 2K proposals on KITTI.
- The full pipeline significantly outperforms all previous published object detection methods for all three object classes on KITTI.The evaluation includes 2D detection, joint 2D detection and orientation estimation, and 3D object detection.