Source-linked AI summary
LiDAR R-CNN: An Efficient and Universal 3D Object Detector
Zhichao Li, Feng Wang, Naiyan Wang
TL;DR
LiDAR-based 3D detection needs to estimate object states from sparse point clouds, but naive point-based R-CNN features can ignore proposal size. LiDAR R-CNN addresses this with a PointNet-based, size-aware second-stage detector that improves existing detectors across datasets and baselines, while remaining fast and extensible to aggregated inputs.
Problem
Naive point-based R-CNN features can ignore proposal-size information because sparse point clouds omit spacing cues, creating size ambiguity.
Method
LiDAR R-CNN is a PointNet-based R-CNN-style second-stage detector that analyzes size ambiguity and applies practical size-aware solutions to proposals.
Results
The method consistently improves baseline models on Waymo Open Dataset and achieves state-of-the-art performance, running at 200fps for 128 proposals on a 2080Ti GPU.
Takeaways & Limitations
As a plug-and-play second stage, LiDAR R-CNN can improve existing 3D detectors without retraining their base detectors.
Takeaways & Limitations
Point-based methods remain constrained by large-scene point counts and the costly hierarchical grouping often needed for feature extraction.
Abstract
from arXiv · showhide
LiDAR-based 3D detection in point cloud is essential in the perception system of autonomous driving. In this paper, we present LiDAR R-CNN, a second stage detector that can generally improve any existing 3D detector. To fulfill the real-time and high precision requirement in practice, we resort to point-based approach other than the popular voxel-based approach. However, we find an overlooked issue in previous work: Naively applying point-based methods like PointNet could make the learned features ignore the size of proposals. To this end, we analyze this problem in detail and propose several methods to remedy it, which bring significant performance improvement. Comprehensive experimental results on real-world datasets like Waymo Open Dataset (WOD) and KITTI dataset with various popular detectors demonstrate the universality and superiority of our LiDAR R-CNN. In particular, based on one variant of PointPillars, our method could achieve new state-of-the-art results with minor cost. Codes will be released at https://github.com/tusimple/LiDAR_RCNN .
1. Introduction
LiDAR-based 3D detection addresses essential 7-DoF object-state estimation but must handle sparse, irregular point clouds and large 3D search spaces. LiDAR R-CNN introduces a PointNet-based second-stage detector that exposes proposal-size ambiguity and improves existing detectors without retraining.
- 7-DoF object-state estimation is vital for autonomous vehicles and robots, while LiDAR detection directly measures 3D structure.
- LiDAR-based 3D detection remains difficult because point clouds are sparse and irregular, unlike well-developed 2D image detection.
- Voxel and projected-view methods enable regular convolution but incur quantization or projection information loss, whereas point-based methods face costly local aggregation.
- LiDAR R-CNN is an R-CNN-style PointNet second-stage detector that is plug-and-play with existing 3D detectors and requires no base-detector retraining.
- The paper identifies size ambiguity in point-based R-CNNs and proposes size-aware solutions that substantially improve performance.
- 200fps for 128 proposals on a 2080Ti GPU accompanies consistent improvements across Waymo and KITTI experiments with various base detectors.
2. Related Work
Prior LiDAR 3D detectors represent irregular point clouds with voxels, projected views, or raw points, trading computational convenience against information loss and efficiency. Point-based methods preserve raw geometry but commonly require downsampling and costly hierarchical grouping.
- Voxel-based methods regularize irregular point clouds for convolution, but hand-crafted voxel features can generalize poorly in complicated real-world environments.
- BEV and range-view methods efficiently process projected point clouds, but projection can lose 3D information and range view introduces object-scale variation.
- PointNet directly processes raw unordered points, while PointRCNN, Part A2, and PV-RCNN extend point-based detection through proposals, part supervision, or voxel aggregation.
- More than 100K points per HDL-64E scan make downsampling necessary for point-based methods operating on large scenes.
- Hierarchical grouping usually improves individual-point feature extraction but is costly compared with pure PointNet, limiting point-based efficiency.
3. Methods
LiDAR R-CNN is a point-based second-stage detector that refines proposal boxes and scores using original point clouds. It addresses size ambiguity in point-based R-CNNs by adding proposal-size information through several size-aware feature designs.
- Point-based R-CNN: LiDAR R-CNN refines 7DoF proposal parameters and scores in an R-CNN framework using original point clouds rather than high-level DNN features.The framework is intended to be plug-and-play and avoid further quantization or interpolation errors.
- Point-based R-CNN: The method enlarges each proposal to include contextual points, transforms those points into the proposal coordinate system, and processes them with a PointNet backbone.The backbone uses an MLP, max pooling, and separate classification and regression branches.
- Size ambiguity problem: Direct point-based R-CNN suffers size ambiguity because sparse point pooling can give differently sized proposals identical features despite different IoU targets.The issue is most significant for mixed-size classes or jointly trained multiple classes.
- Size-aware point features: The paper evaluates normalization, anchors, voxelization, boundary offsets, and virtual grid points as ways to expose proposal size or boundary information to the R-CNN model.Normalization can distort object shape and erase category scale differences, while voxelization provides only coarse proposal-size information.
- Contribution and evaluation: The size ambiguity analysis is presented as the core contribution, while alternative solutions are expected to work whenever they provide proposal-size information.Table 1 reports vehicle detection results on Waymo validation sequences for several LiDAR R-CNN variants and first-stage detectors.
4. Experiments
Experiments on WOD and KITTI evaluate LiDAR R-CNN with multiple base detectors, datasets, and scale-aware variants. The method consistently improves baselines, generalizes across models, and offers a speed–accuracy trade-off with marginal computational demands.
- Datasets and setup: Experiments use WOD and KITTI with multiple base detectors, including PointPillars, SECOND, RCD, and a range-view model.WOD includes large-scale multi-class data; KITTI experiments use training or trainval configurations.
- 3D detection results: LiDAR R-CNN improves all evaluated baseline models across distance ranges and surpasses prior state-of-the-art comparisons on WOD vehicle detection.With a strong PointPillars baseline, it exceeds PV-RCNN by 5.31% AP on 3D vehicle detection.
- 3D detection results: Multi-class WOD experiments show strong detection performance for VEHICLE, PEDESTRIAN, and CYCLIST, including small objects.The experiments use PointPillars and SECOND with multi-class classification and class-agnostic regression losses.
- 3D detection results: On KITTI, LiDAR R-CNN improves PointPillars by 2.8 AP on the easy vehicle-detection set, although gains are smaller than on WOD.The paper attributes reduced effectiveness partly to KITTI’s limited training data and high baseline performance.
- Ablation studies: Ablations show that point-based R-CNN variants outperform the PointPillars baseline, while virtual-point and offset methods perform best in multi-class detection.Size normalization and anchor methods have difficulty distinguishing categories when classes have different scales.
- Efficiency: Doubling point-encoding channels improves pedestrian AP by 0.6% and cyclist AP by 1.8%, illustrating a speed–accuracy trade-off.Inference time can also be adjusted by changing the number of sampled points per object.
- Ablation studies: Boundary offsets resolve size ambiguity sufficiently to refine proposal size, whereas vanilla PointNet refinement does not improve the size sub-task.This comparison is conducted on the PointPillars model.
5. Conclusions and Future Work
LiDAR R-CNN is a fast second-stage 3D detector that steadily improves baseline models and state-of-the-art performance on the Waymo Open Dataset. Its second-stage design also supports generalization to multi-frame LiDAR and RGB+LiDAR inputs, while multimodal fusion remains future work.
- LiDAR R-CNN steadily improves all baseline models and achieves state-of-the-art performance on the Waymo Open Dataset.
- As a second-stage framework, LiDAR R-CNN is easy to generalize to multi-frame LiDAR and RGB+LiDAR inputs.
- The authors identify developing LiDAR R-CNN into a multimodal fusion framework as future work.