Source-linked AI summary
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection
Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, Hongsheng Li
TL;DR
3D object detection from point clouds must handle sparse, irregular data while retaining accurate localization. The paper proposes PV-RCNN and PV-RCNN++, which deeply integrate voxel and point feature learning, adding efficient keypoint sampling and VectorPool aggregation. PV-RCNN++ achieves state-of-the-art Waymo performance with 10 FPS inference over a 150m × 150m range.
Problem
Point-cloud detection is important but challenging because sparse, irregular points complicate efficient processing and voxelization introduces localization-degrading quantization errors.
Method
PV-RCNN integrates sparse-convolution voxel features with point-based set abstraction through voxel-to-keypoint encoding and keypoint-to-grid RoI abstraction; PV-RCNN++ adds sectorized proposal-centric sampling and VectorPool aggregation.
Results
PV-RCNN++ achieves state-of-the-art performance on the Waymo Open Dataset, outperforming previous state-of-the-art methods in LEVEL 2 mAPH by +1.88% for vehicles, +2.40% for pedestrians, and +1.59% for cyclists.
Takeaways & Limitations
PV-RCNN++ provides a more accurate, efficient, and practical point-cloud detector, with 10 FPS inference across a 150m × 150m detection range.
Takeaways & Limitations
Sectorized keypoint sampling is tailored for LiDAR sensors, and spatial partitioning is required because random grouping harms model performance.
Abstract
from arXiv · showhide
3D object detection is receiving increasing attention from both industry and academia thanks to its wide applications in various fields. In this paper, we propose Point-Voxel Region-based Convolution Neural Networks (PV-RCNNs) for 3D object detection on point clouds. First, we propose a novel 3D detector, PV-RCNN, which boosts the 3D detection performance by deeply integrating the feature learning of both point-based set abstraction and voxel-based sparse convolution through two novel steps, i.e., the voxel-to-keypoint scene encoding and the keypoint-to-grid RoI feature abstraction. Second, we propose an advanced framework, PV-RCNN++, for more efficient and accurate 3D object detection. It consists of two major improvements: sectorized proposal-centric sampling for efficiently producing more representative keypoints, and VectorPool aggregation for better aggregating local point features with much less resource consumption. With these two strategies, our PV-RCNN++ is about $3\times$ faster than PV-RCNN, while also achieving better performance. The experiments demonstrate that our proposed PV-RCNN++ framework achieves state-of-the-art 3D detection performance on the large-scale and highly-competitive Waymo Open Dataset with 10 FPS inference speed on the detection range of 150m * 150m.
1 Introduction
PV-RCNNs address the tension between efficient voxel-based processing and location-precise point-based feature learning through deep point-voxel integration. PV-RCNN++ extends this design with faster, more representative keypoint sampling and efficient local feature aggregation.
- 3D point-cloud detection is important but difficult because point clouds are sparse and irregular compared with images.These properties make directly applying 2D detection techniques challenging.
- Voxel methods efficiently encode multi-scale features but introduce quantization errors, whereas point methods preserve accurate locations with flexible receptive fields.
- PV-RCNN deeply integrates voxel-based sparse convolution and point-based set abstraction through voxel-to-keypoint scene encoding and keypoint-to-grid RoI feature abstraction.The first step summarizes multi-scale voxel features into accurately located keypoints; the second aggregates keypoint features into proposal-specific RoI grids.
- PV-RCNN++ improves the two-stage detector with sectorized proposal-centric sampling that concentrates keypoints near proposals while parallelizing sampling across spatial sectors.The strategy accelerates sampling and maintains uniform keypoint distribution, addressing the quadratic complexity of vanilla farthest point sampling.
- VectorPool aggregation encodes position-sensitive local features by partitioning local space into compact sub-voxels and combining their features into hyper feature vectors.Its compact representation handles many centric points with lower resource consumption than set abstraction.
- PV-RCNN++ achieves state-of-the-art Waymo Open Dataset results at 10 FPS for a 150m × 150m detection range.The framework is presented as faster, more practical, and less resource-intensive while maintaining or improving performance.
2 Related Work
Prior 3D detectors use image, voxel, point, or hybrid representations to handle irregular point clouds, trading efficiency, receptive-field flexibility, and localization precision. PV-RCNN combines voxel sparse convolution with point set abstraction and proposal-aligned feature refinement.
- Image-based 3D Detection: Image-based methods estimate 3D boxes from monocular, stereo, or surrounding-camera imagery using geometric, depth-aware, or bird’s-eye-view representations.
- Voxel-based Methods: Voxel-based methods project points into regular 2D or 3D grids, enabling convolutional processing and efficient proposal generation.Their receptive fields can remain constrained by convolution kernel size.
- Point-based Methods: Point-based methods operate directly on raw or cropped points and use set abstraction to provide flexible receptive fields for feature learning.Representative approaches include F-PointNet, PointRCNN, and 3DSSD.
- Point-Voxel Methods: Hybrid methods combine point-wise and voxel features, but earlier approaches generally transform features between representations without deeply intertwining their basic operators.
3 PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
PV-RCNN combines sparse voxel convolution with point-based set abstraction in a two-stage detector, using voxel features for efficient proposal generation and point-voxel features for refinement. Its keypoint and RoI-grid modules preserve location information while aggregating multi-scale context.
- PV-RCNN framework: PV-RCNN uses a 3D sparse-convolution voxel CNN as its backbone for efficient feature encoding and initial proposal generation.The backbone produces multi-scale voxel features and bird-view maps for detection heads.
- Voxel-to-keypoint scene encoding: Voxel Set Abstraction summarizes multi-scale voxel features at farthest-sampled keypoints while retaining their accurate raw-point coordinates.The module replaces neighboring raw points with voxel-wise semantic features and aggregates them using set abstraction.
- Voxel-to-keypoint scene encoding: Keypoint features combine four-level voxel semantics with raw-point features and bird-view features to compensate for voxelization loss and enlarge receptive fields.Raw-point features help recover location information, while bird-view interpolation provides broader context along the height axis.
- Keypoint-to-grid RoI feature abstraction: For each proposal, PV-RCNN uniformly samples 216 RoI-grid points and aggregates neighboring keypoint features using multiple receptive-field radii.The resulting grid features are concatenated and transformed to represent the proposal box.
- Keypoint-to-grid RoI feature abstraction: RoI-grid pooling captures contextual keypoint features beyond proposal boundaries, unlike prior pooling methods that average features or include many uninformative zeros.Overlapping neighborhood balls allow one keypoint to contribute to multiple grid points.
4 PV-RCNN++: Faster and Better 3D Detection With PV-RCNN Framework
PV-RCNN++ replaces key PV-RCNN components with sectorized proposal-centric keypoint sampling and VectorPool aggregation to improve efficiency and local feature representation for large-scale point clouds.
- PV-RCNN++ introduces sectorized proposal-centric sampling and VectorPool aggregation to make PV-RCNN more accurate, efficient, and practical for large-scale 3D detection.The modules replace PV-RCNN’s corresponding keypoint sampling and local feature aggregation components.
- Sectorized Proposal-Centric Sampling: Farthest point sampling is time-consuming because of quadratic complexity and produces background keypoints that are generally useless for proposal refinement.These drawbacks hinder training and inference speed, especially on large-scale point clouds.
- Sectorized Proposal-Centric Sampling: SPC filtering restricts candidates to proposal neighborhoods, reducing the candidate set from |P| to |P′| and concentrating keypoints on important regions.For Waymo, P is generally about 180k while P′ can be smaller than 90k in most cases.
- Sectorized Proposal-Centric Sampling: Sectorized sampling divides the filtered points into spatial sectors, enabling parallel local sampling while preserving a roughly uniform keypoint distribution.The sector partition considers LiDAR point radial distribution to balance group sizes and reduce the effective sampling scale.
- Sectorized Proposal-Centric Sampling: The algorithm reduces sampling scale from |P| to the much smaller maxk∈{1,...,s} |S′k|, accelerating sampling and improving representation near 3D proposals.The method retains farthest point sampling within each sector to produce uniformly distributed keypoints.
- Local Vector Representation: VectorPool uses position-specific kernels and a concatenated local vector representation to preserve spatial point distributions with less memory and computation than set abstraction.It is adopted as a basic module for local feature aggregation in PV-RCNN++.
5 Experiments
PV-RCNN++ improves 3D detection across Waymo and KITTI evaluations while substantially increasing processing speed over PV-RCNN. Ablations attribute these gains to its feature aggregation and keypoint-sampling designs.
- Main Results: +1.88% vehicle, +2.40% pedestrian, and +1.59% cyclist mAPH gains over prior state-of-the-art methods on Waymo LEVEL 2 validation.These results use the single-frame setting.
- Main Results: +1.57% vehicle and +2.00% pedestrian mAPH gains over prior state-of-the-art on the Waymo test set in the single-frame setting.The multi-frame setting gains +2.93% for vehicle and +2.03% for pedestrian; cyclist performance is comparable.
- PV-RCNN versus PV-RCNN++: With a center-based head, PV-RCNN++ gains +0.93% vehicle, +1.58% pedestrian, and +1.83% cyclist mAPH over PV-RCNN at LEVEL 2.The anchor-based setting also improves by +1.54%, +3.33%, and 4.24% for vehicle, pedestrian, and cyclist, respectively.
- Main Results: 10 FPS on a 150m × 150m detection range, up from 3.3 FPS for PV-RCNN.The comparison used a single TITAN RTX GPU on Waymo Open Dataset.
- Main Results: PV-RCNN++ outperforms PV-RCNN across all KITTI categories, indicating gains across different scenes and LiDAR sensors.The passage reports a broad category-level comparison but does not provide the individual KITTI values.
- Ablation Study: Removing predicted keypoint weighting slightly reduces performance, while shallow semantic features slightly improve performance but greatly increase training cost.The weighting module is optional because its gains are limited; shallow semantic features are therefore omitted.
- Ablation Study: Proposal-centric keypoint filtering improves average LEVEL 2 performance by 1.12 mAPH, from 65.87% to 66.99%, versus vanilla farthest point sampling.The filtering also reduces sampling time from 133ms to 27ms.
- Ablation Study: Sectorized farthest point sampling matches vanilla FPS coverage, 84.76% versus 84.78%, while reducing sampling time from 27ms to 9ms.Its detection performance is 66.99% versus 66.87% for vanilla FPS.
6 Conclusion
PV-RCNN and PV-RCNN++ integrate voxel- and point-based features for accurate 3D object detection from point clouds. PV-RCNN++ further improves keypoint generation and local feature encoding, achieving better performance and faster running speed than PV-RCNN, including state-of-the-art results on Waymo Open Dataset.
- PV-RCNN integrates multi-scale voxel CNN and PointNet-based features through voxel set abstraction and RoI-grid pooling.These modules produce discriminative keypoint features and aggregate richer contextual information for proposal refinement.
- PV-RCNN++ generates more representative keypoints through sectorized proposal-centric sampling and learns structure-preserved local features with VectorPool aggregation.The two improvements target keypoint-generation efficiency and local feature encoding in both major aggregation modules.
- PV-RCNN++ achieves better performance with much faster running speed than the original PV-RCNN framework.
- PV-RCNN++ significantly outperforms previous 3D detection methods and achieves state-of-the-art performance on both validation and testing sets of the Waymo Open Dataset.The paper reports extensive experiments investigating the individual components of both proposed frameworks.