PaperlayerView exact revision on arXiv

Source-linked AI summary

Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection

Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, Chunjing Xu

arXiv:2109.02499v1cs.CV

TL;DR

Pyramid R-CNN addresses the difficulty of extracting reliable RoI features from sparse and non-uniform 3D point clouds. It introduces a pyramid RoI head with expanded point collection, unified attention operations, and density-aware radius prediction, achieving strong results across KITTI and Waymo.

  • Problem

    Existing RoI feature extraction methods struggle with sparse and non-uniform Points of Interest, limiting detection in difficult point-cloud conditions.

  • Method

    Pyramid R-CNN uses a pyramid RoI head combining RoI-grid Pyramid, RoI-grid Attention, and Density-Aware Radius Prediction across diverse 3D backbones.

  • Results

    Pyramid R-CNN consistently outperforms baselines, reaches 82.08% moderate car mAP on KITTI, and ranks first among LiDAR-only methods for Waymo vehicle detection.

  • Takeaways & Limitations

    The framework is robust to sparse and imbalanced point distributions and generalizes across multiple 3D backbones and benchmark settings.

Abstract

from arXiv · show

We present a flexible and high-performance framework, named Pyramid R-CNN, for two-stage 3D object detection from point clouds. Current approaches generally rely on the points or voxels of interest for RoI feature extraction on the second stage, but cannot effectively handle the sparsity and non-uniform distribution of those points, and this may result in failures in detecting objects that are far away. To resolve the problems, we propose a novel second-stage module, named pyramid RoI head, to adaptively learn the features from the sparse points of interest. The pyramid RoI head consists of three key components. Firstly, we propose the RoI-grid Pyramid, which mitigates the sparsity problem by extensively collecting points of interest for each RoI in a pyramid manner. Secondly, we propose RoI-grid Attention, a new operation that can encode richer information from sparse points by incorporating conventional attention-based and graph-based point operators into a unified formulation. Thirdly, we propose the Density-Aware Radius Prediction (DARP) module, which can adapt to different point density levels by dynamically adjusting the focusing range of RoIs. Combining the three components, our pyramid RoI head is robust to the sparse and imbalanced circumstances, and can be applied upon various 3D backbones to consistently boost the detection performance. Extensive experiments show that Pyramid R-CNN outperforms the state-of-the-art 3D detection models by a large margin on both the KITTI dataset and the Waymo Open dataset.

1. Introduction

Pyramid R-CNN is a general two-stage 3D detection framework designed to address sparse, non-uniform point distributions through a pyramid RoI head. Its components improve point collection and feature extraction, with reported gains on KITTI and Waymo.

  • Framework: Pyramid R-CNN is a general two-stage framework that can be applied across multiple 3D backbones for adaptable and accurate detection.The framework is intended to handle environmental changes and improve detection performance.
  • Motivation: Two-stage 3D detectors benefit from RoI refinement, but their Points of Interest are sparse, non-uniformly distributed, and limited relative to the full point cloud.More than 7% of KITTI objects have fewer than 10 points, object-point counts range from fewer than 10 to more than 500, and 2k keypoints may represent 15k input points.
  • Pyramid RoI head: The pyramid RoI head combines RoI-grid Pyramid, RoI-grid Attention, and Density-Aware Radius Prediction to address sparsity and non-uniform point distributions.The three components expand point collection, unify graph- and attention-based operators, and adapt the RoI focusing range to density.
  • Results: 82.08% moderate car mAP is achieved on KITTI, while Pyramid R-CNN ranks first among LiDAR-only methods for vehicle detection on the Waymo test leaderboard.The reported results also state consistent improvement over baselines.

2. Related Work

Related work includes point-, voxel-, and pillar-based single-stage detectors and two-stage approaches that exploit fine-grained point information for box refinement. Single-stage methods differ in representation and generally lack the second-stage refinement used by two-stage detectors.

  • Single-stage 3D Object Detection: Single-stage 3D detectors use point-based, voxel-based, or pillar-based representations for proposal generation and box prediction.Examples include set abstraction for points, sparse convolutions for voxels, and BEV pillars with 2D CNNs.
  • Two-stage 3D object detection: Single-stage methods cannot benefit from the fine-grained point information that two-stage approaches use for accurate box prediction.Two-stage methods add a refinement stage based on Points of Interest and their extracted features.

3. Pyramid R-CNN

Pyramid R-CNN is a general two-stage 3D detection framework whose pyramid RoI head addresses sparse, non-uniform Points of Interest through hierarchical context collection, unified feature aggregation, and density-adaptive neighborhoods.

  • Overall Architecture: Pyramid R-CNN uses a two-stage architecture compatible with point-based, voxel-based, and point-voxel-based backbones.Backbones generate 3D proposals and Points of Interest in stage one, while the pyramid RoI head refines boxes in stage two.
  • RoI-grid Pyramid: The RoI-grid Pyramid enlarges RoIs across levels to capture contextual Points of Interest while preserving fine-grained internal structure.The lowest level maintains the original RoI size, whereas higher levels expand the RoI and reduce grid resolution to control computation.
  • RoI-grid Pyramid: The RoI-grid Pyramid combines interior grid points for shape detail with exterior grid points for context needed to recognize incomplete objects.This design targets sparse, non-uniform Points of Interest whose limited object coverage makes object size and category difficult to infer from interior points alone.
  • RoI-grid Attention: RoI-grid Attention unifies graph-based and attention-based point operators through learnable gated functions that adaptively select geometric and semantic information.The formulation can recover graph, standard attention, and Point Transformer operators under different gate settings.
  • Density-Aware Radius Prediction: Density-Aware Radius Prediction makes the RoI neighborhood differentiable and learns an adaptive radius for feature extraction under changing point densities.The radius determines which neighboring Points of Interest participate, while a smooth sampling approximation enables gradient-based learning and limits computation to a sphere slightly larger than the radius.

4. Experiments

Pyramid R-CNN is evaluated across Waymo and KITTI using point-, voxel-, and point-voxel-based architectures. It consistently improves baseline detection performance, including difficult and distant cases, while retaining computational efficiency.

  • Experimental Setup: Pyramid R-CNN provides three architectures compatible with point-based, voxel-based, and point-voxel-based backbones.Pyramid-P, Pyramid-V, and Pyramid-PV replace the respective baseline second-stage heads while keeping backbone configurations comparable.
  • Waymo Open Dataset: On Waymo validation, Pyramid-P, Pyramid-V, and Pyramid-PV achieve 2.0%, 4.1%, and 6.0% mAP gains over their baselines, respectively.The models achieve superior mAP across all difficulty levels and distance ranges.
  • Waymo Open Dataset: Pyramid-V surpasses PV-RCNN by 12.3% mAP for objects beyond 50m, while Pyramid-PV reaches 76.30% LEVEL 1 mAP and 67.23% LEVEL 2 mAP.Pyramid-PV also achieves 81.77% LEVEL 1 mAP on the Waymo test leaderboard and ranks first among LiDAR-only approaches as of March 10th, 2021.
  • KITTI Dataset: On KITTI moderate-car detection, Pyramid-P, Pyramid-V, and Pyramid-PV improve baseline mAP by 4.66%, 2.79%, and 0.65%, with Pyramid-PV reaching 82.08% mAP.Validation gains are reported for both moderate and hard car cases, with gains mainly from hard cases.
  • Ablation Studies: The ablation study attributes gains to the RoI-grid Pyramid, RoI-grid Attention, and DARP, which add 1.20%, 0.51%, and 0.37% mAP improvements, respectively.The RoI-grid Pyramid captures larger context, RoI-grid Attention improves over pooling, and adaptive radius supports density-aware processing.
  • Efficiency and Design Analysis: Using enlarged RoI pyramids improves performance by placing grid points outside RoIs, while the pyramid RoI head adds little per-frame latency compared with baselines.The evaluated pyramid uses 409 grid points, comparable to 432 points in a referenced baseline.

5. Conclusion

Pyramid R-CNN is a general two-stage framework that addresses sparse and non-uniform point-cloud distributions through a pyramid RoI head and supports diverse backbones.

  • Pyramid R-CNN is a general two-stage framework applicable to diverse 3D detection backbones.
  • The pyramid RoI head addresses sparse and non-uniform point-cloud distributions.
  • The authors plan to optimize Pyramid R-CNN for efficient inference in future work.

A. Approximation in Radius Prediction

The method uses s(i|r) as a soft approximation to p(i|r), with approximation quality controlled by temperature τ. The temperature is decreased during training to sharpen this approximation.

  • A. Approximation in Radius Prediction: s(i|r) approximates p(i|r) softly, with the curve’s sharpness controlled by temperature τ.As τ approaches 0, s(i|r) becomes more similar to p(i|r).
  • A. Approximation in Radius Prediction: The initial temperature is 0.02 and is gradually decreased to 0.0001 for a better approximation.

B. Implementation of the DARP Module

DARP predicts dynamic radius offsets for grid points at each RoI-grid Pyramid level. It first aggregates context with a fixed sphere, then uses that embedding to predict the offsets.

  • B. Implementation of the DARP Module: DARP first aggregates context embeddings with a fixed sphere before predicting dynamic radius offsets.The design is inspired by deformable convolutions, which use standard convolutions to predict deformable offsets.
  • B. Implementation of the DARP Module: At each RoI-grid Pyramid level, two spheres are centered at the RoI as part of the radius-prediction process.
  • B. Implementation of the DARP Module: Figure 7 presents the 3D backbones of Pyramid-PV and Pyramid-PV†.

C. Backbones of Pyramid R-CNN

Pyramid R-CNN keeps the pyramid RoI head unchanged across backbones while supporting backbone variants with different point and voxel configurations. The implementations include a Waymo-adapted PointRCNN backbone and a larger Pyramid-PV† backbone.

  • C. Backbones of Pyramid R-CNN: The pyramid RoI head is kept the same across all backbones used in the paper.
  • C. Backbones of Pyramid R-CNN: For Waymo, Pyramid-P re-implements PointRCNN with 40k input points and 18,024, 2,048, 512, and 128 downsampled points across layers.This modification keeps more points because the input point count is larger than in the KITTI setting.
  • C. Backbones of Pyramid R-CNN: Pyramid-PV† uses a larger backbone with a voxel size of [0.2m, 0.2m, 0.2m].
  • C. Backbones of Pyramid R-CNN: Pyramid R-CNN supports either a small backbone for fair baseline comparison or a large backbone for further enhanced detection performance.

D. Qualitative Results

Qualitative results on KITTI and Waymo Open show that Pyramid R-CNN accurately detects distant 3D objects with few points. The visualizations compare blue ground-truth boxes with red Pyramid-PV predictions.

  • Pyramid R-CNN accurately detects far-away 3D objects represented by only a few points.
  • The KITTI visualization uses blue boxes for ground truth and red boxes for Pyramid-PV predictions.
  • The Waymo Open visualization uses blue boxes for ground truth and red boxes for Pyramid-PV predictions.
Loading 2109.02499v1…