PaperlayerView exact revision on arXiv

Source-linked AI summary

Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, Houqiang Li

arXiv:2012.15712v2cs.CV

TL;DR

3D detectors face a trade-off between point-based accuracy and voxel-based efficiency because point processing preserves precise positions but is computationally costly. Voxel R-CNN uses a two-stage voxel-based design with direct voxel RoI pooling, achieving strong accuracy on KITTI and Waymo while maintaining efficient processing.

  • Problem

    Point-based 3D detectors preserve precise point positions but incur high computation overhead, while voxelization is efficient yet can lose positional information.

  • Method

    Voxel R-CNN combines a 3D backbone, BEV-based proposal generation, and voxel RoI pooling that extracts proposal features directly from 3D voxel volumes.

  • Results

    Voxel R-CNN achieves state-of-the-art Waymo LEVEL 1 3D mAP of 75.59% and exceeds PV-RCNN by 5.29% mAP.

  • Takeaways & Limitations

    The results support using coarse voxel features as sufficient spatial context for high-performance 3D detection while balancing accuracy and efficiency.

  • Takeaways & Limitations

    Voxel RoI pooling still has large computation complexity because local aggregation groups many voxel features and applies fully connected layers.

Abstract

from arXiv · show

Recent advances on 3D object detection heavily rely on how the 3D data are represented, \emph{i.e.}, voxel-based or point-based representation. Many existing high performance 3D detectors are point-based because this structure can better retain precise point positions. Nevertheless, point-level features lead to high computation overheads due to unordered storage. In contrast, the voxel-based structure is better suited for feature extraction but often yields lower accuracy because the input data are divided into grids. In this paper, we take a slightly different viewpoint -- we find that precise positioning of raw points is not essential for high performance 3D object detection and that the coarse voxel granularity can also offer sufficient detection accuracy. Bearing this view in mind, we devise a simple but effective voxel-based framework, named Voxel R-CNN. By taking full advantage of voxel features in a two stage approach, our method achieves comparable detection accuracy with state-of-the-art point-based models, but at a fraction of the computation cost. Voxel R-CNN consists of a 3D backbone network, a 2D bird-eye-view (BEV) Region Proposal Network and a detect head. A voxel RoI pooling is devised to extract RoI features directly from voxel features for further refinement. Extensive experiments are conducted on the widely used KITTI Dataset and the more recent Waymo Open Dataset. Our results show that compared to existing voxel-based methods, Voxel R-CNN delivers a higher detection accuracy while maintaining a real-time frame processing rate, \emph{i.e}., at a speed of 25 FPS on an NVIDIA RTX 2080 Ti GPU. The code is available at \url{https://github.com/djiajunustc/Voxel-R-CNN}.

1 Introduction

Voxel R-CNN addresses the accuracy–efficiency trade-off in 3D detection by using voxel features in a two-stage framework. It argues that coarse voxel representations can retain sufficient spatial context without the computational cost of point-based processing.

  • Point-based detectors preserve precise positions but incur high computation costs from unordered point-neighbor searches.Voxel methods offer better memory locality and efficient feature extraction, but voxelization can lose positional information.
  • The method targets accuracy comparable to advanced point-based detectors while retaining the efficiency associated with voxel-based processing.Its design is motivated by the goal of matching point-based accuracy with voxel-based speed.
  • Voxel R-CNN directly extracts RoI features from 3D voxel feature volumes instead of aggregating features from point representations.The framework combines a 3D backbone, BEV proposal generation, and voxel RoI pooling for refinement.
  • Experiments support the claim that precise raw-point positioning is unnecessary and coarser voxels can provide sufficient spatial context for high-performance detection.The authors present Voxel R-CNN as a simple baseline for further investigation and downstream tasks.

2 Reflection on 3D Object Detection

The analysis contrasts efficient voxel-based detection with more accurate but slower point-voxel approaches. It motivates Voxel R-CNN by identifying the limits of BEV-only features and the cost of point-voxel interaction.

  • SECOND.: SECOND performs voxel-based one-stage detection by extracting 3D features, converting them to BEV, and applying a 2D backbone with an RPN.This establishes the baseline pipeline examined in the analysis.
  • PV-RCNN.: PV-RCNN adds keypoints and voxel set abstraction to preserve 3D structure before extracting proposal features for box refinement.Its design improves structural representation relative to direct BEV detection.
  • 2.2 Analysis: Adding a BEV detect head to SECOND improves KITTI car moderate AP by 0.6%, but remains less accurate than PV-RCNN.The result indicates that box refinement helps, while BEV representation alone has limited capacity.
  • 2.2 Analysis: Point-voxel interaction takes almost half of PV-RCNN’s running time, making it substantially slower than SECOND.This computation cost motivates a voxel-only detector that directly uses 3D voxel tensors.
  • Voxel R-CNN motivation: Voxel R-CNN is designed to retain 3D structure through voxel features while avoiding the time cost of point-voxel feature interaction.Its voxel RoI pooling extracts spatial context directly from 3D feature volumes.

3 Voxel R-CNN Design

Voxel R-CNN constructs a two-stage detector from sparse voxel features, using BEV proposals and direct 3D voxel RoI pooling for refinement. Its voxel query exploits regular voxel indexing for efficient neighborhood aggregation.

  • Voxel-based framework: Raw point clouds are voxelized and processed by a 3D backbone to produce sparse 3D feature volumes.The volumes are represented through non-empty voxel center points and their feature vectors.
  • Region proposal generation: The 3D voxel features are converted into BEV representations for 2D backbone and RPN-based region proposal generation.This forms the first stage of the detector.
  • Voxel RoI pooling: Voxel RoI pooling divides each proposal into G×G×G sub-voxels and aggregates neighboring voxel features at their grid points.This directly supplies 3D spatial context for proposal feature extraction.
  • Voxel query: The voxel query is illustrated against ball query as an alternative neighborhood operation performed in 3D space.The comparison emphasizes the regular arrangement of voxels rather than unordered point storage.
  • Voxel query: Voxel query finds neighboring voxels through regular voxel-index offsets, enabling efficient grouping in quantized 3D space.For example, neighboring voxels can be computed using offset triplets over voxel indices.

Voxel Query

Voxel RoI pooling aggregates neighboring voxel features around regular sub-voxel grid points, then combines multiscale features for RoI refinement. An accelerated PointNet decomposition reduces the computation required for local aggregation.

  • Voxel RoI Pooling: Voxel RoI pooling divides each proposal into G×G×G sub-voxels and uses each sub-voxel center as a grid point.Because non-empty voxels account for < 3% of 3D feature volumes, neighboring voxel features are integrated rather than directly max-pooled within each sub-voxel.
  • Voxel RoI Pooling: For each grid point, neighboring voxel features and relative coordinates are aggregated with a PointNet module and channel-wise max pooling.The resulting aggregated feature vector is extracted from the last two 3D-backbone stages using multiple Manhattan-distance thresholds.
  • Voxel RoI Pooling: Multiscale features from different backbone stages and distance scales are concatenated to form the final RoI features.These RoI features are passed to the detect head for bounding-box refinement and confidence prediction.
  • Computational Complexity: The original local aggregation costs O(M × K × (C + 3) × C′) FLOPs because M grid points each group K voxels with C-dimensional features and 3D coordinates.Grouped voxel features also consume substantial memory during the fully connected transformation.
  • Computational Complexity: The accelerated PointNet applies the feature transformation before voxel query and processes relative coordinates afterward, reducing cost to O(N × C × C′ + M × K × 3 × C′).Voxel features are independent of grid points, enabling the decomposition into feature and coordinate streams; with M × K much larger than N, this is more efficient than the original module.

4 Experiments

Experiments evaluate Voxel R-CNN on KITTI and Waymo, using established autonomous-driving benchmarks with distinct dataset sizes and annotation coverage.

  • KITTI contains 7,481 training samples and 7,518 testing samples from autonomous-driving scenes.
  • KITTI training data are commonly split into 3,712 training and 3,769 validation samples, with validation and online test-leaderboard results reported.
  • Waymo Open Dataset contains 1,000 sequences, including approximately 158k training and 40k validation point-cloud samples.
  • Unlike KITTI, Waymo provides annotations for objects across the full 360° field of view.
  • Table 3 compares KITTI test-set car performance using average precision calculated at 40 recall positions.

4.2 Implementation Details

Implementation uses dataset-specific voxelization and backbone configurations, end-to-end optimization, and proposal refinement with nonmaximum suppression.

  • KITTI inputs use voxel size (0.05m, 0.05m, 0.1m) within a clipped [0, 70.4]m × [−40, 40]m × [−3, 1]m range.
  • Waymo inputs use voxel size (0.1m, 0.1m, 0.15m) within a clipped [−75.2, 75.2]m × [−75.2, 75.2]m × [−2, 4]m range.
  • The 3D backbone has four stages with filter numbers 16, 32, 48, and 64, while the 2D backbone contains two resolution-specific blocks.
  • Training uses Adam with 80 epochs and batch size 16 on KITTI, versus 30 epochs and batch size 32 on Waymo.
  • Tables 4 and 5 report KITTI validation performance using 40-recall and 11-recall AP settings, respectively.
  • Inference applies RPN NMS at IoU 0.7, retains the top 100 proposals, then applies refinement NMS at IoU 0.1.

4.3 Results on KITTI Dataset

On KITTI, Voxel R-CNN achieves strong car-detection accuracy while retaining real-time efficiency and consistently performing well across validation and test evaluations.

  • KITTI car evaluation uses a 0.7 IoU threshold, with validation AP at 11 recall positions and test-server AP at 40 recall positions.
  • 81.62% AP is achieved for moderate KITTI test-set cars at 25.2 FPS.
  • Voxel R-CNN matches PV-RCNN comparably in accuracy with approximately one-third of its running time.
  • Compared with SA-SSD, Voxel R-CNN improves easy, moderate, and hard AP by 2.15%, 1.83%, and 2.90%, respectively.
  • On the KITTI validation set, Voxel R-CNN achieves the best performance at moderate and hard difficulty levels.
  • Validation and test results consistently show state-of-the-art 3D detection AP alongside high efficiency characteristic of voxel-based models.

4.4 Results on Waymo Open Dataset

On Waymo, Voxel R-CNN surpasses prior methods across object ranges and both difficulty levels, with especially strong gains for distant sparse objects.

  • Waymo objects are divided into LEVEL 1 objects with more than 5 points and LEVEL 2 objects with 1–5 points.
  • 75.59% LEVEL 1 3D mAP establishes new state-of-the-art performance on Waymo.
  • Voxel R-CNN surpasses previous methods across all ranges for both LEVEL 1 and LEVEL 2 objects.
  • Voxel R-CNN outperforms PV-RCNN by 5.29% mAP on the commonly used LEVEL 1 metric.
  • In the 50m-Inf range, Voxel R-CNN exceeds PV-RCNN by 10.98% mAP, demonstrating effectiveness for objects with very sparse points.

4.5 Ablation Study

The ablation study shows that adding 3D voxel-based refinement improves detection accuracy, while voxel querying and accelerated feature aggregation recover much of the computational cost.

  • 40.8 FPS is achieved by the one-stage BEV baseline, but its AP is unsatisfactory for precise 3D object detection.
  • 4.22% moderate AP is gained by adding a detect head for box refinement to the one-stage baseline.This supports using 3D spatial context for more precise object detection.
  • 23.4 FPS is lost when ball query and original PointNet modules extract RoI features for the refined method.
  • 1.7 FPS is recovered by replacing ball query with voxel query, leveraging the neighbor-aware property of voxel representations.
  • 21.4 FPS is reached after using the accelerated PointNet module to aggregate voxel features, up from 17.4 FPS.
  • Voxel R-CNN combines box refinement, voxel querying, and accelerated feature aggregation to achieve state-of-the-art accuracy while maintaining voxel-based efficiency.

5 Related Work

Related work divides point-cloud 3D object detectors into point-based and voxel-based approaches. Point-based methods preserve raw point structure, whereas voxel-based methods discretize points into grids for CNN-based processing and efficiency.

  • Point-based Methods: Point-based methods take raw point clouds as input and construct point representations through iterative sampling and grouping.
  • Point-based Methods: PointRCNN uses a 3D region proposal network and point-cloud RoI pooling to extract features for each proposal.
  • Point-based Methods: STD transfers proposal points into dense voxel representations for 3D CNN refinement, while 3DSSD develops a one-stage anchor-free detector based on representative points.
  • Voxel-based Methods: Voxel-based methods discretize point clouds into equally spaced grids and apply 2D or 3D CNNs for detection.
  • Voxel-based Methods: VoxelNet uses voxel-wise PointNet features followed by 3D CNNs, while SECOND reduces computation with sparse 3D convolution and SA-SSD adds structure-preserving auxiliary objectives.
  • Voxel-based Methods: PointPillars groups points into pillars and applies a simplified PointNet before forming the detection representation.

6 Conclusion

Voxel R-CNN uses voxel representations in a two-stage detector that generates BEV proposals and refines them with voxel RoI pooling. Results on KITTI and Waymo support its use as an effective baseline for 3D detection research and downstream tasks.

  • Voxel R-CNN generates dense proposals from BEV features and extracts proposal features from 3D voxel features using voxel RoI pooling.
  • Results on KITTI and Waymo demonstrate a balance between detection accuracy and efficiency for this voxel-based detector.
  • Voxel R-CNN can serve as a simple but effective baseline for investigating 3D object detection and downstream tasks.
Loading 2012.15712v2…