Source-linked AI summary
PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud
Shaoshuai Shi, Xiaogang Wang, Hongsheng Li
TL;DR
3D object detection from point clouds must handle irregular data and a large 6-DoF search space, while prior projections and voxelization can lose information. PointRCNN directly generates proposals from raw points through foreground segmentation, then refines them in canonical coordinates using semantic and local spatial features. On KITTI, it outperforms prior state-of-the-art methods with point clouds alone.
Problem
3D detection remains challenging because point clouds are irregular and object localization spans a large 6-DoF search space, while projections and voxelization lose information.
Method
PointRCNN segments foreground points to generate bottom-up proposals, then transforms pooled proposal points into canonical coordinates for semantic and local-spatial box refinement.
Results
PointRCNN outperforms previous state-of-the-art methods by remarkable margins on KITTI’s challenging 3D detection benchmark using only point clouds.
Takeaways & Limitations
The framework provides a point-cloud-only two-stage detector that combines high-recall proposal generation with canonical-coordinate refinement.
Takeaways & Limitations
Canonical transformation loses object depth information, so the method includes distance to the sensor to compensate for this loss.
Abstract
from arXiv · showhide
In this paper, we propose PointRCNN for 3D object detection from raw point cloud. The whole framework is composed of two stages: stage-1 for the bottom-up 3D proposal generation and stage-2 for refining proposals in the canonical coordinates to obtain the final detection results. Instead of generating proposals from RGB image or projecting point cloud to bird's view or voxels as previous methods do, our stage-1 sub-network directly generates a small number of high-quality 3D proposals from point cloud in a bottom-up manner via segmenting the point cloud of the whole scene into foreground points and background. The stage-2 sub-network transforms the pooled points of each proposal to canonical coordinates to learn better local spatial features, which is combined with global semantic features of each point learned in stage-1 for accurate box refinement and confidence prediction. Extensive experiments on the 3D detection benchmark of KITTI dataset show that our proposed architecture outperforms state-of-the-art methods with remarkable margins by using only point cloud as input. The code is available at https://github.com/sshaoshuai/PointRCNN.
1. Introduction
PointRCNN addresses the challenges of 3D detection from irregular point clouds by directly generating proposals from raw points and refining them in canonical coordinates. Its bottom-up, two-stage design uses segmentation-derived proposals and features while avoiding large-scale 3D anchor enumeration.
- 3D point-cloud detection is difficult because irregular data and the large 6-DoF search space complicate object localization.
- Prior methods project point clouds into bird’s view, frontal view, or voxels, introducing information loss during quantization.
- PointRCNN uses annotated 3D boxes to derive foreground masks, simultaneously segmenting points and generating a small number of proposals bottom-up.
- Stage-2 pools proposal points, transforms them into canonical coordinates, and combines local spatial features with stage-1 semantic features for box refinement.
- The framework uses canonical refinement and bin-based losses, with learned segmentation representations supporting both proposal generation and later box refinement.
- PointRCNN uses only point clouds and ranked first among published KITTI 3D detection methods as of Nov. 16 2018.
2. Related Work
Related work represents 3D objects using images, projections, voxels, or raw point clouds. PointRCNN builds on direct point-cloud representation learning while targeting the limitations of prior detection pipelines.
- Image-based methods recover 3D pose using geometry, CAD similarity, or predefined-box energy functions, but depth limitations produce coarse detections.
- Bird’s-view and voxel-based methods use 2D or 3D CNNs to learn features for generating 3D boxes from point clouds.
- PointRCNN extends direct point-cloud representation learning to 3D object detection while avoiding projection and voxelization-based representations.
- PointNet directly learns point features from raw point clouds instead of voxels or multi-view formats, improving classification and segmentation speed and accuracy.
3. PointRCNN for Point Cloud 3D Detection
PointRCNN detects 3D objects directly from irregular point clouds using bottom-up proposal generation followed by canonical-coordinate refinement. Point-wise segmentation supplies foreground cues for proposals, while bin-based localization and canonical features support box refinement.
- PointRCNN uses a two-stage architecture: bottom-up 3D proposal generation followed by canonical 3D box refinement.
- Bottom-up 3D Proposal Generation: Stage 1 segments whole-scene point clouds into foreground and background while generating proposals from foreground points.The approach avoids a large set of predefined 3D boxes and constrains the proposal search space.
- Bottom-up 3D Proposal Generation: Foreground segmentation uses focal loss to address the imbalance between sparse foreground points and abundant background points.The segmentation mask is obtained from 3D ground-truth boxes, and the focal-loss settings are αt = 0.25 and γ = 2.
- Bottom-up 3D Proposal Generation: The proposal head predicts box centers with X/Z bins plus residuals, directly regresses Y, and estimates orientation with bins and dimensions with residuals.X/Z bin classification is reported as more accurate and robust than direct smooth L1 regression for center localization.
- Bottom-up 3D Proposal Generation: Stage 1 applies oriented bird’s-view NMS, retaining top 300 proposals for training and top 100 for inference before refinement.The stated IoU thresholds are 0.85 for training and 0.8 for inference.
4. Experiments
Experiments on KITTI evaluate PointRCNN’s implementation, detection performance, proposal recall, and ablations of refinement inputs, context pooling, and regression loss. The results show strong proposal recall and improvements from canonical transformation, stage-1 features, contextual points, and the full bin-based loss.
- Implementation: PointRCNN is evaluated on KITTI using separate stage-1 and stage-2 training, with augmentation and category-specific training details reported.The benchmark includes train, validation, and test splits; implementation details focus on the car category.
- 3D Object Detection: 8.28% AP improvement over the previous best AP is reported for the car class at hard difficulty on the KITTI validation split.The paper also reports large margins over previous state-of-the-art methods on the validation split.
- Proposal Generation: 96.01% recall at IoU threshold 0.5 is achieved with 50 proposals on moderate car examples, compared with 91% for AVOD.With 300 proposals, recall reaches 98.21% at IoU threshold 0.5.
- Ablation Study: Canonical transformation substantially improves refinement performance by reducing rotation and location variation in the stage-2 input.Removing stage-1 segmentation and proposal features decreases moderate-difficulty mAP by 2.71%.
- Ablation Study: η = 1.0m gives the best context-aware pooling performance, while removing contextual points particularly reduces accuracy on hard examples.Hard cases often contain fewer proposal points because objects are occluded or far from the sensor.
- Ablation Study: The full bin-based regression loss produces higher recall and faster convergence than alternative losses, especially at IoU threshold 0.7.The partial-bin loss has similar recall but slower convergence, while improved residual-cosine loss outperforms residual loss for angle regression.
5. Conclusion
PointRCNN detects 3D objects directly from raw point clouds using bottom-up proposal generation and canonical-coordinate refinement. On KITTI, it combines semantic and local spatial features and outperforms previous state-of-the-art methods.
- Conclusion: PointRCNN directly generates 3D proposals from point clouds by segmenting foreground objects and background in a bottom-up stage.The stage-1 network produces a small number of high-quality proposals.
- Conclusion: The stage-2 network refines proposals in canonical coordinates by combining semantic features with local spatial features.This refinement operates on proposals generated by stage 1.
- Conclusion: PointRCNN outperforms previous state-of-the-art methods with remarkable margins on the KITTI 3D detection benchmark.The reported detector uses raw point cloud input.
- Conclusion: Qualitative KITTI test examples display detected objects with green 3D bounding boxes and orientations marked by an X in images and red tubes in point clouds.Each sample pairs an image view with a representative point-cloud view.