Source-linked AI summary
PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, Hongsheng Li
TL;DR
3D object detection from sparse, irregular point clouds requires effective integration of voxel-based and point-based feature learning. PV-RCNN uses voxel-to-keypoint scene encoding followed by keypoint-to-grid RoI abstraction, and experiments on KITTI and Waymo show improved performance over previous state-of-the-art methods. Its scope is bounded by low-resolution, sparse voxel feature volumes and the computational cost of directly pooling many scene voxels.
Problem
Combining 3D sparse voxel CNNs with PointNet-based set abstraction is challenging because directly using many voxels and proposal grid points is memory-intensive.
Method
PV-RCNN encodes multi-scale voxel features into a small set of keypoints, then applies multi-radius keypoint set abstraction to proposal RoI-grid points.
Results
PV-RCNN significantly improves 3D object detection performance on KITTI and Waymo compared with previous state-of-the-art methods.
Takeaways & Limitations
The voxel-to-keypoint and keypoint-to-grid strategy integrates efficient voxel learning with flexible point-based receptive fields for accurate proposal refinement.
Takeaways & Limitations
Voxel CNN feature volumes can be low-resolution and sparse, while directly pooling scene voxels for RoI grids can be inefficient and memory-intensive.
Abstract
from arXiv · showhide
We present a novel and high-performance 3D object detection framework, named PointVoxel-RCNN (PV-RCNN), for accurate 3D object detection from point clouds. Our proposed method deeply integrates both 3D voxel Convolutional Neural Network (CNN) and PointNet-based set abstraction to learn more discriminative point cloud features. It takes advantages of efficient learning and high-quality proposals of the 3D voxel CNN and the flexible receptive fields of the PointNet-based networks. Specifically, the proposed framework summarizes the 3D scene with a 3D voxel CNN into a small set of keypoints via a novel voxel set abstraction module to save follow-up computations and also to encode representative scene features. Given the high-quality 3D proposals generated by the voxel CNN, the RoI-grid pooling is proposed to abstract proposal-specific features from the keypoints to the RoI-grid points via keypoint set abstraction with multiple receptive fields. Compared with conventional pooling operations, the RoI-grid feature points encode much richer context information for accurately estimating object confidences and locations. Extensive experiments on both the KITTI dataset and the Waymo Open dataset show that our proposed PV-RCNN surpasses state-of-the-art 3D detection methods with remarkable margins by using only point clouds. Code is available at https://github.com/open-mmlab/OpenPCDet.
1. Introduction
PV-RCNN integrates voxel-based and PointNet-based feature learning through voxel-to-keypoint scene encoding and keypoint-to-grid RoI feature abstraction. This design addresses the challenge of combining efficient voxel processing with flexible point-based receptive fields while limiting memory use.
- Framework motivation: PV-RCNN combines voxel-based and point-based feature learning to improve 3D object detection from irregular point clouds.The framework is designed to retain efficient voxel processing while learning more discriminative point-cloud features.
- Framework motivation: Uniformly applying set abstraction to proposal grid points is memory-intensive because both voxel and grid-point counts can be large.The introduction identifies this as the central challenge in unifying 3D sparse voxel CNNs with PointNet-based set abstraction.
- Keypoint-to-grid RoI feature abstraction: Keypoint-to-grid RoI abstraction uses multiple receptive fields to aggregate richer context for proposal box refinement and confidence prediction.Each proposal grid point aggregates features from keypoints through a multi-scale keypoint set abstraction layer.
- Voxel-to-keypoint scene encoding: Voxel-to-keypoint encoding summarizes multi-scale voxel features of the whole scene into a small set of keypoints.The keypoints preserve accurate location and encode rich scene context for later proposal processing.
- Experimental outcome: PV-RCNN ranks first on the KITTI 3D detection benchmark and surpasses previous methods on the Waymo Open dataset by a large margin.The reported results use point clouds and compare against previous 3D detection methods.
2. Related Work
Prior 3D object detectors mainly regularize point clouds into grids for CNN processing or directly process points with point-based networks. PV-RCNN combines voxel-based feature learning with PointNet-based set abstraction to obtain efficient proposal generation and flexible receptive fields.
- Grid-based methods: Grid-based methods transform irregular point clouds into 3D voxels or 2D bird-view maps for efficient CNN-based feature learning.These representations support processing with either 3D or 2D convolutional neural networks.
- Point-voxel integration: PV-RCNN combines 3D sparse convolution with PointNet-based set abstraction to support high-quality proposals and flexible receptive fields.Its representation integrates advantages associated with both voxel-based and point-based feature learning.
3. PV-RCNN for Point Cloud Object Detection
PV-RCNN combines an efficient voxel CNN for multi-scale scene encoding and proposal generation with PointNet-based set abstraction for location-aware, multi-context proposal refinement. It compresses scene features into keypoints before aggregating them to RoI grids, addressing the cost and sparsity of direct feature-volume pooling.
- PV-RCNN integrates voxel-based and point-based feature learning to combine efficient encoding and proposal generation with accurate contextual features and flexible receptive fields.The voxel CNN serves as the backbone, while PointNet-based set abstraction supports fine-grained box refinement.
- Direct RoI pooling from voxel feature volumes is limited by up to 8× downsampling, sparsity, small interpolation neighborhoods, and wasted computation and memory.A naive set-abstraction design can also require 2,700 × 18,000 pairwise distances and feature aggregations for a representative KITTI scene.
- The voxel-to-keypoint scene encoding step uses Furthest-Point-Sampling to summarize the full scene into a small, representative set of keypoints.The method samples 2,048 keypoints for KITTI and 4,096 for Waymo, then aggregates neighboring voxel features from multiple CNN levels.
- Voxel set abstraction encodes neighboring voxel features using relative locations, learned transformations, max pooling, and multiple radii to capture multi-scale context.Features from different neural levels are concatenated to form multi-scale semantic keypoint representations while retaining keypoint coordinates.
- Predicted Keypoint Weighting reweights keypoint features using foreground supervision, while quality-aware confidence prediction is reported to outperform traditional classification targets.The weighting module predicts foreground confidence with a three-layer MLP and sigmoid, trained using focal loss.
4. Experiments
Experiments evaluate PV-RCNN on KITTI and Waymo using standard detection metrics, comparisons, and ablations. The method achieves strong benchmark performance, while component studies support the voxel-to-keypoint, multi-scale feature, and RoI-grid designs.
- Experimental Setup: PV-RCNN is evaluated on KITTI and Waymo Open, including benchmark comparisons and ablation studies of its components.KITTI uses mAP-based evaluation, while Waymo reports mAP and heading-weighted mAP under distance and difficulty splits.
- KITTI Results: PV-RCNN ranks first on the KITTI car 3D detection leaderboard and first among published LiDAR-only methods for cyclist 3D detection.The ranking is reported as of Nov. 15, 2019.
- Waymo Results: 7.37% mAP and 2.56% mAP gains are reported on Waymo vehicle 3D and bird-view detection, respectively, over previous state-of-the-art methods.The Waymo evaluation uses official mAP and mAPH metrics and reports results across distance ranges.
- Ablation Studies: Voxel-to-keypoint scene encoding improves performance across difficulty levels by bridging the voxel CNN and RoI-grid points with broader receptive fields.Ablations also report substantial degradation when only raw-point features are aggregated.
- Ablation Studies: Removing predicted keypoint weighting or replacing RoI-grid pooling with RoI-aware pooling significantly reduces performance.The study attributes these effects to foreground-focused aggregation and richer contextual features from larger search radii.
5. Conclusion
The conclusion presents PV-RCNN as a point-cloud 3D detection framework that integrates voxel and PointNet-based features. Its voxel-to-keypoint and keypoint-to-grid abstraction strategy improves detection performance on KITTI and Waymo relative to prior state-of-the-art methods.
- Conclusion: Keypoint features are aggregated to RoI-grid points with multiple receptive fields to capture context for fine-grained proposal refinement.This is the paper’s keypoint-to-grid RoI feature abstraction strategy.
- Conclusion: Experiments on KITTI and Waymo show significant 3D detection improvements over previous state-of-the-art methods.The conclusion attributes the reported gains to the voxel-to-keypoint and keypoint-to-grid abstraction strategy.