PaperlayerView exact revision on arXiv

Source-linked AI summary

From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network

Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, Hongsheng Li

arXiv:1907.03670v3cs.CV

TL;DR

3D point-cloud object detection must handle irregular data and large six-degree-of-freedom search spaces. Part-A2 net uses free 3D-box-derived part supervision, RoI-aware pooling, and two-stage aggregation to improve proposals and boxes, achieving state-of-the-art KITTI performance with LiDAR alone. Its supervision assumes separable, well-oriented 3D objects, limiting application to some indoor scenes and categories.

  • Problem

    3D point-cloud detection faces irregular data and large six-degree-of-freedom search spaces, while prior methods do not fully exploit information encoded by 3D box annotations.

  • Method

    Part-A2 net predicts foreground segmentation, intra-object part locations, and 3D proposals, then applies RoI-aware pooling and part aggregation for box scoring and refinement.

  • Results

    Part-A2 net significantly outperforms existing methods and achieves new state-of-the-art performance on KITTI using only LiDAR point clouds.

  • Takeaways & Limitations

    Aggregating predicted intra-object part locations improves proposal quality and supports accurate box scoring and location refinement.

  • Takeaways & Limitations

    The method is less applicable to indoor scenes with overlapping 3D boxes or categories whose orientation is not well-defined.

Abstract

from arXiv · show

3D object detection from LiDAR point cloud is a challenging problem in 3D scene understanding and has many practical applications. In this paper, we extend our preliminary work PointRCNN to a novel and strong point-cloud-based 3D object detection framework, the part-aware and aggregation neural network (Part-$A^2$ net). The whole framework consists of the part-aware stage and the part-aggregation stage. Firstly, the part-aware stage for the first time fully utilizes free-of-charge part supervisions derived from 3D ground-truth boxes to simultaneously predict high quality 3D proposals and accurate intra-object part locations. The predicted intra-object part locations within the same proposal are grouped by our new-designed RoI-aware point cloud pooling module, which results in an effective representation to encode the geometry-specific features of each 3D proposal. Then the part-aggregation stage learns to re-score the box and refine the box location by exploring the spatial relationship of the pooled intra-object part locations. Extensive experiments are conducted to demonstrate the performance improvements from each component of our proposed framework. Our Part-$A^2$ net outperforms all existing 3D detection methods and achieves new state-of-the-art on KITTI 3D object detection dataset by utilizing only the LiDAR point cloud data. Code is available at https://github.com/sshaoshuai/PointCloudDet3D.

1 INTRODUCTION

Part-A2 net addresses the irregularity and large search space of point-cloud 3D detection by exploiting intra-object part information implied by 3D boxes. Its two stages predict part locations and proposals, then pool and aggregate part features for box scoring and refinement.

  • 3D point-cloud detection is difficult because irregular data formats and six-degree-of-freedom object search complicate direct extension of 2D detectors.
  • Existing methods do not fully exploit point distributions implied by 3D box annotations or use proposal-specific pooling for second-stage refinement.
  • Part-A2 net uses free intra-object part locations from 3D boxes to learn discriminative point features in a two-stage detector.
  • Stage-I predicts foreground segmentation, intra-object part locations, and 3D proposals using anchor-free or anchor-based proposal generation.
  • Stage-II uses differentiable RoI-aware pooling and sparse convolutions to aggregate part features for confidence prediction and box refinement.
  • 14 FPS inference speed and first-place performance on KITTI are reported for Part-A2 net as of August 15, 2019.

2 RELATED WORK

Prior 3D detection work uses image fusion, bird-view projections, point-based features, or voxelization, but Part-A2 net combines point-cloud processing with part-aware two-stage refinement. It aggregates predicted part locations to refine proposals and reports strong point-cloud-only results.

  • Image-based 3D methods recover boxes using geometry constraints, CAD similarity, or energy functions, while sensor-fusion methods may face synchronization issues.
  • Using only point clouds, Part-A2 net achieves comparable or better detection results than sensor-fusion methods.
  • Point-cloud-only detectors use bird-view maps, raw-point PointNet features, or voxelized sparse-convolution features, but earlier methods are generally one-stage.
  • Part-A2 net predicts intra-object part locations from 3D box annotations and aggregates them in a second stage for proposal refinement.
  • Unlike related anchor-free instance-segmentation approaches, Part-A2 directly estimates intra-object part locations and generates 3D proposals from individual points.

3 PART-A2 NET: 3D PART-AWARE AND AGGREGATION FOR 3D DETECTION FROM POINT CLOUD

Part-A2 net extends PointRCNN with part-aware supervision and part aggregation for point-cloud 3D detection. The framework uses 3D box-derived locations and segmentation labels to learn features, while voxelized points approximately preserve raw-point geometry.

  • Part-A2 net extends PointRCNN into an end-to-end framework with part-aware and part-aggregation stages.
  • Ground-truth 3D boxes provide accurate segmentation masks and relative locations for foreground points, unlike potentially occluded 2D boxes.
  • The first stage uses free intra-object part-location and segmentation labels as extra supervision for learning better 3D features.
  • Voxelized point clouds use non-empty voxel centers as points and approximately preserve raw-point geometry and 3D object shapes.

3.1 Stage-I: Part-aware 3D proposal generation

Stage-I learns point-wise foreground and intra-object part representations while generating 3D proposals from point clouds. It supports anchor-free and anchor-based proposal strategies for different efficiency and recall needs.

  • Part-aware feature learning: The part-aware network estimates intra-object part locations for foreground points while simultaneously generating 3D proposals.Part locations are derived from 3D ground-truth boxes and encode relative positions within objects.
  • Part-aware feature learning: The sparse convolution encoder-decoder learns voxel-wise features for foreground segmentation and part-location estimation.The 3D space is voxelized, and sparse convolutions and deconvolutions produce point-wise features.
  • Part-aware feature learning: Part locations normalize foreground-point positions within ground-truth boxes to [0, 1], with the object center represented as (0.5, 0.5, 0.5).The coordinates use the box center, dimensions, and orientation to express each point’s relative location.
  • 3D proposal generation: The bin-based center regression strategy replaces direct offset regression with X- and Y-axis bin classification to improve center localization robustness.This addresses the wide variation in center offsets between foreground points at different object locations.
  • 3D proposal generation: The anchor-free strategy generates proposals from predicted foreground points without evaluating large sets of predefined 3D anchors.It is described as more memory efficient, whereas the anchor-based strategy achieves higher object recall.

3.2 RoI-aware point cloud feature pooling

RoI-aware pooling resolves ambiguous proposal representations by normalizing each proposal and encoding its geometry in a fixed voxel grid. Empty voxels are retained for sparse-convolution processing.

  • Canonical transformation: Canonical transformation maps pooled points and refinement targets into each proposal’s local coordinate system.The transformation places the origin at the proposal center and reduces rotation and location variation.
  • Pooling motivation: Previous pooling could map different proposals to the same point set, losing geometric information needed for refinement.The ambiguity arises because it aggregates features solely from points inside each proposal.
  • RoI-aware pooling: RoI-aware pooling divides every proposal into a fixed regular voxel grid, such as 14 × 14 × 14, independent of proposal size.Each voxel corresponds to a fixed spatial region in the proposal’s local coordinates.
  • RoI-aware pooling: Empty voxels are assigned zero features and marked empty so subsequent sparse convolutions can process the proposal geometry efficiently.The operation supports both voxel-wise max pooling and average pooling over points assigned to each voxel.
  • RoI-aware pooling: Position-specific pooling captures proposal geometry and provides an effective representation for box scoring and location refinement.The module is differentiable, allowing integration into the full detection framework.

3.3 Stage-II: Part location aggregation for confidence prediction and 3D box refinement

Stage-II fuses predicted part locations with semantic point features, then aggregates them across each proposal. The resulting representation supports confidence prediction and box refinement.

  • Confidence and refinement: The aggregation subnetwork predicts proposal confidence and refines box locations from the pooled 3D features.Its purpose is to score box proposals and improve their spatial localization.
  • Feature fusion: Stage-II combines predicted part locations and semantic part features into fused RoI features for each 3D proposal.Average pooling summarizes predicted part locations, while max pooling summarizes semantic part features.
  • Feature fusion: The fused features encode both geometric and semantic information in a shared 14 × 14 × 14 spatial shape.A sparse convolution transforms pooled part locations to match the semantic feature dimensions before concatenation.
  • Part aggregation: Sparse convolutions and sparse max-pooling aggregate proposal features from local to global spatial scales.The aggregation network uses 3 × 3 × 3 convolution kernels and intermediate 2 × 2 × 2 max-pooling.
  • Confidence and refinement: IoU-guided box scoring uses normalized 3D IoU as a soft proposal-quality label and performs slightly better than traditional classification-based scoring.The target is 1 for IoU > 0.75, 0 for IoU < 0.25, and 2IoU − 0.5 otherwise.

3.4 Overall loss

The end-to-end objective combines losses for part-aware proposal generation and part-aggregation-based proposal scoring and refinement, with equal stage weights.

  • The overall loss combines the part-aware loss and the part-aggregation loss for end-to-end training.
  • Part-aware stage-I uses equally weighted focal, binary cross entropy, and smooth-L1 terms for foreground segmentation, part-location regression, and proposal generation.
  • Part-A2-free uses bin-based box generation loss, whereas Part-A2-anchor uses residual-based box regression loss for proposal generation.
  • Part-aggregation stage-II uses binary cross entropy for box-quality regression and smooth-L1 loss for 3D proposal refinement.
  • Stage-II box refinement includes center, size, and angle refinement, with an additional corner regularization loss.

3.5 Implementation details

The implementation uses sparse encoder-decoder networks and RoI-aware pooling, with specified training augmentation, proposal sampling, and inference post-processing.

  • The stage-I backbone is a UNet-like sparse convolution and deconvolution encoder-decoder that downsamples spatial resolution by 8 times before upsampling.
  • The Part-A2-anchor backbone uses feature dimensions 16-32-64-64 in the encoder and 64-64-32-16 in the decoder.
  • RoI-aware pooling uses a 14 × 14 × 14 volume, downsampled to 7 × 7 × 7 with 128-dimensional features before vectorization for scoring and refinement.
  • Training uses ADAM for 50 epochs with batch size 6, initial learning rate 0.001, and 128 stage-II proposals per scene sampled with a 1:1 positive-negative ratio.
  • Training augmentation includes random flipping, scaling sampled from [0.95, 1.05], vertical rotation, and copying ground-truth boxes with their inside points between scenes.
  • Inference retains 100 stage-I proposals, applies stage-specific NMS, and takes about 70ms on a Tesla V100 GPU.

3.6 Pros and cons.

Part-A2 introduces part-location learning and RoI-aware feature pooling, but its annotation-derived supervision is constrained in overlapping indoor scenes and poorly oriented categories.

  • The method introduces intra-object part-location learning to improve point-cloud 3D detection performance.
  • RoI-aware pooling removes ambiguity in prior point-cloud pooling and converts sparse point-wise features into regular voxel features for proposal geometry and semantics.
  • The RoI-aware pooling module can transform point-wise features from networks such as PointNet++ into sparse voxel-wise features for sparse convolution processing.
  • Indoor-scene applicability is limited because overlapping 3D boxes may not yield accurate point-wise segmentation labels.
  • Categories with poorly defined orientation, such as round tables, cannot provide accurate intra-object part-location labels.
  • The anchor-free proposal strategy remains promising for indoor scenes because it avoids placing 3D anchors throughout the space.

4 EXPERIMENTS

Experiments on KITTI evaluate the framework’s components, proposal-generation strategies, pooling design, and resource costs. The results show gains from sparse point-wise features, part supervision, RoI-aware pooling, and stage-II refinement, with trade-offs between anchor-free efficiency and anchor-based recall.

  • Backbone and proposal generation: 81.54% recall at 3D IoU 0.7 was achieved by SparseConvUNet-based Part-A2-free, 6.73% higher than PointRCNN with a PointNet++ backbone.This supports the effectiveness of sparse convolution for point-wise feature learning and proposal generation.
  • RoI-aware pooling: Removing RoI-aware pooling substantially decreases detection accuracy, while replacing stage-II sparse convolutions with fully connected layers produces similar performance.The ablation attributes the main improvement to RoI-aware pooling rather than the choice of stage-II aggregation operator.
  • RoI-aware pooling: Pooling sizes above 12 × 12 × 12 show similar performance, indicating robust 3D detection results across larger RoI-aware pooling grids.The default 14 × 14 × 14 setting captures details before sparse max-pooling downsamples the feature volumes.
  • Proposal-generation strategies: 96.01% recall at IoU 0.5 with 50 proposals was achieved by PointRCNN, exceeding AVOD’s 91% recall by 5.01% using only point cloud input.The comparison concerns proposal recall and uses the same number of proposals, while AVOD uses both image and point cloud data.
  • Proposal-generation strategies: 83.71% recall at IoU 0.7 with 50 proposals was achieved by Part-A2-anchor, compared with 77.12% for Part-A2-free.The anchor-based strategy improves spatial coverage but requires class-specific anchors, whereas anchor-free generation can produce proposals across classes from segmented foreground points.
  • Part supervision and stage-II refinement: Part location supervision improves recall and average precision across all car difficulty levels, while stage-II refinement raises performance from 82.92 to 84.33 after refining 100 proposals.The experiments attribute these gains to learning better 3D features and aggregating predicted intra-object part locations for box rescoring and localization refinement.
  • Memory and model complexity: Part-A2-free generates approximately 16k proposals independently of class count, whereas Part-A2-anchor reaches 211.2k anchors for three classes.For inference on one scene, GPU memory usage was 1.16GB for Part-A2-free and 1.63GB for Part-A2-anchor; for six scenes, usage was 3.42GB and 5.46GB, respectively.
  • False-positive analysis: Most false positives come from background confusion and poor localization, with poor-localization errors increasing as the score threshold rises.Sparse point clouds may lack sufficient semantic information for background structures with geometry similar to foreground objects, while rotated 3D IoU imposes stricter localization requirements.

4.2 Main results and comparison with state-of-the-arts on KITTI benchmark

Part-A2 achieves strong KITTI detection results using only LiDAR while accurately predicting intra-object part locations. The reported analyses connect part-location quality with box localization and show a limitation for pedestrian orientation estimation.

  • KITTI test results: Part-A2 outperforms previous peer-reviewed LiDAR-only methods across all difficulty levels and classes on KITTI 3D detection.It also surpasses previous multisensor methods at moderate difficulty for car and cyclist.
  • KITTI validation results: Part-A2 achieves new state-of-the-art performance across all difficulty levels on the KITTI car validation split using only point clouds.It also outperforms state-of-the-art methods on moderate-difficulty 3D detection and localization.
  • Class-specific analysis: Pedestrian performance is slightly inferior because sparse point clouds make pedestrian orientation difficult to recognize, affecting part-location prediction.The authors note that RGB-integrating multisensor methods can have advantages for small pedestrians.
  • Part-location accuracy: 6.28% mean part-location error indicates accurate intra-object predictions for cars, averaging ±6.28 cm per meter.The errors are reported for the car class on the KITTI validation split.
  • Error analysis: 0.531 overall Pearson correlation shows a positive relationship between part-location errors and 3D box errors.The z-axis in the height direction is reported as the most correlated axis.
  • Error analysis: Inaccurately localized false positives have substantially larger part-location errors across all three axes.The authors indicate that inaccurate part locations may accompany unsatisfactory 3D object localization and reduced detection performance.

4.3 Qualitative results

Qualitative KITTI test examples show that Part-A2 estimates intra-object part locations from point clouds and aggregates them to produce 3D bounding boxes.

  • Qualitative results: Part-A2-anchor qualitatively estimates intra-object part locations and aggregates them to generate accurate 3D bounding boxes.The examples are shown on the KITTI test split, with boxes and part locations visualized separately.

5 CONCLUSION

Part-A2 extends PointRCNN with part-aware supervision and part aggregation for point-cloud 3D detection. It achieves new state-of-the-art KITTI performance, with experiments examining individual framework components.

  • Framework: Part-A2 uses ground-truth-derived foreground and intra-object location labels to learn part-aware features while generating proposals with anchor-free or anchor-based strategies.Its second stage pools predicted part locations with a RoI-aware point-cloud scheme for box scoring and location refinement.
  • Conclusion: Part-A2 significantly outperforms existing 3D detection methods and achieves new state-of-the-art performance on the KITTI benchmark.The experiments also investigate the individual components of the proposed framework.
Loading 1907.03670v3…