PaperlayerView exact revision on arXiv

Source-linked AI summary

Group-Free 3D Object Detection via Transformers

Ze Liu, Zheng Zhang, Yue Cao, Han Hu, Xin Tong

arXiv:2104.00678v2cs.CV

TL;DR

Existing 3D point-cloud detectors commonly rely on handcrafted point grouping, whose inaccurate assignments can reduce detection performance. This paper instead uses Transformer attention over all points, with iterative stage refinement and ensemble decoding, and reports state-of-the-art results on ScanNet V2 and SUN RGB-D. The reported SUN RGB-D gain from ensembling is +3.8 mAP@0.25.

  • Problem

    Handcrafted point grouping for object candidates can assign points inaccurately, limiting 3D object detection from irregular point clouds.

  • Method

    The method computes each object candidate’s feature from all point-cloud points using learned Transformer attention, with iterative refinement and stage-wise detection ensembling.

  • Results

    The method achieves state-of-the-art 3D object detection performance on ScanNet V2 and SUN RGB-D, with ensembling improving SUN RGB-D by +3.8 mAP@0.25.

  • Takeaways & Limitations

    Attention-based group-free processing is presented as a way to model irregular 3D point clouds while avoiding handcrafted point grouping.

  • Takeaways & Limitations

    The approach manually assigns object candidates to ground truths, and this assignment may be difficult for the network to learn.

Abstract

from arXiv · show

Recently, directly detecting 3D objects from 3D point clouds has received increasing attention. To extract object representation from an irregular point cloud, existing methods usually take a point grouping step to assign the points to an object candidate so that a PointNet-like network could be used to derive object features from the grouped points. However, the inaccurate point assignments caused by the hand-crafted grouping scheme decrease the performance of 3D object detection. In this paper, we present a simple yet effective method for directly detecting 3D objects from the 3D point cloud. Instead of grouping local points to each object candidate, our method computes the feature of an object from all the points in the point cloud with the help of an attention mechanism in the Transformers \cite{vaswani2017attention}, where the contribution of each point is automatically learned in the network training. With an improved attention stacking scheme, our method fuses object features in different stages and generates more accurate object detection results. With few bells and whistles, the proposed method achieves state-of-the-art 3D object detection performance on two widely used benchmarks, ScanNet V2 and SUN RGB-D. The code and models are publicly available at \url{https://github.com/zeliu98/Group-Free-3D}

1. Introduction

3D point-cloud detection is difficult because sparse, irregular inputs complicate direct use of 2D detection techniques, while handcrafted grouping can assign points inaccurately. Group-Free 3D replaces grouping with attention over all points and improves Transformer decoding through iterative refinement and stage ensembling.

  • Motivation: 3D object detection localizes and recognizes objects from irregular, sparse point clouds, unlike detection on regular 2D images.The task supports applications including autonomous driving, robotics manipulation, and augmented reality.
  • Limitations of Grouping: Existing point-based methods assign points to each object candidate before computing object features, but these handcrafted assignments can be inaccurate.Examples include grouping within frustum or 3D-box proposals, or grouping points voting for the same center.
  • Proposed Approach: Group-Free 3D computes each candidate’s feature from all point-cloud points, learning every point’s contribution through attention instead of handcrafted grouping.The method adapts Transformers to model object-object and object-point relationships.
  • Transformer Improvements: The method iteratively refines object predictions by updating spatial encodings across stages and ensembles detections from all stages during inference.These modifications improve performance with few computational overheads.
  • Results: The approach is validated on ScanNet V2 and SUN RGB-D, with a reported SUN RGB-D improvement of +3.8 mAP@0.25 from the ensemble scheme.The paper reports state-of-the-art performance on both benchmarks.

2. Related Work

Prior 3D detectors either project or voxelize point clouds, incurring quantization or computational costs, or group points before feature extraction. This work uses attention over all points and stacked refinement to avoid grouping while adapting Transformers to irregular 3D data.

  • Grid and Voxel Methods: Projection- and voxelization-based detectors enable convolutional processing but suffer from quantization errors, while voxel methods also incur large 3D-convolution costs.Bird’s-eye-view projection is mainly suited to outdoor scenes with objects distributed on a horizontal plane.
  • Point-Based Methods: Point-based detectors typically assign a point group to each object candidate and then use PointNet-like processing to compute object features.Frustum-PointNet uses 2D-box frustums, Point R-CNN uses 3D-box proposals, and VoteNet uses voted centers.
  • Group-Free Detection: The proposed point-based method uses attention over all points for each candidate and stacks attention modules to iteratively refine detections.This preserves a simple architecture while removing the conventional point-group generation step.
  • Transformers for 3D: Transformers are dominant in NLP and competitive in 2D image recognition, but this work adapts them to model irregular and sparse 3D point clouds.The paper positions attention as naturally suited to point-cloud structure and reports task-specific Transformer improvements.
  • Architecture: The architecture comprises a point-feature backbone, candidate sampling, and stacked attention modules that extract and refine object representations.The backbone can use different point-cloud architectures; the implementation adopts PointNet++ for fair comparison.

3. Methodology

The method detects 3D objects directly from irregular point clouds by using Transformer attention to extract each candidate’s features from all points, avoiding handcrafted grouping. It samples candidates from points, refines predictions across decoder stages, and combines stage outputs for final detection.

  • Overall Architecture: The architecture uses a backbone, point-based candidate sampling, and stacked attention modules to extract and refine object representations from all points.PointNet++ is used as the backbone, producing point features for candidate sampling and attention processing.
  • Candidate Sampling: Initial object candidates are sampled directly from the point cloud because applying dense 3D anchor boxes is impractical.The framework considers FPS, KPS, and KPS-NMS sampling strategies; KPS is used by default because it matches KPS-NMS effectiveness with less complexity.
  • Attention Decoder: Transformer attention computes each candidate’s object feature from all point features, with multi-head self-attention and cross-attention modules operating in parallel.Cross-attention aggregates point-derived values according to learned attention weights, while self-attention models interactions among object features.
  • Iterative Refinement: Iterative box prediction updates each candidate’s spatial encoding stage by stage using predicted 3D box locations before the next decoder stage.The spatial encoding uses box center and size parameters, while point encodings use point coordinates.
  • Iterative Refinement: 1.6 and 5.0 points: iterative refinement improves mAP@0.25 and mAP@0.5 on ScanNet V2 compared with no iterative refinement.The reported gains are 1.6 on mAP@0.25 and 5.0 on mAP@0.5.
  • Multi-Stage Prediction: Predictions from all decoder stages are ensembled and passed through NMS to produce the final detection results.This differs from using only the final decoder stage output.

4. Experiments

Experiments on ScanNet V2 and SUN RGB-D show that the group-free Transformer approach performs strongly across system comparisons and design ablations. Iterative spatial refinement, multi-stage predictions, and attention over all points improve detection while retaining modest computational cost.

  • Experimental Setup: The method is evaluated on ScanNet V2 and SUN RGB-D using standard data splits and PointNet++ as the default backbone for fair comparison.ScanNet V2 contains 1,513 indoor scenes and 18 object categories; evaluation follows the standard mean Average Precision protocol.
  • System-level Comparison: 69.1 mAP@0.25 and 52.8 mAP@0.5 are achieved on ScanNet V2 with a wider backbone and 512 object candidates, outperforming the previous best method by a large margin.The setting uses twice as many channels and 512 candidates.
  • System-level Comparison: 63.0 mAP@0.25 and 45.2 mAP@0.5 are achieved on SUN RGB-D, exceeding previous point-cloud-only state-of-the-art methods; the mAP@0.5 advantage over H3DNet is 6.2 points.The multi-stage ensemble is used by default during inference on SUN RGB-D.
  • Ablation Study: Sampling choices are robust, with the framework performing well across strategies and a wide range of KPS hyperparameters; KPS is selected by default for better performance than FPS.KPS-NMS is more complex but similarly effective to KPS.
  • Ablation Study: Stage-wise spatial refinement raises mAP@0.5 by 4.1 points over naive iterative prediction, while encoding box center and size outperforms encoding centers alone.The reported mAP@0.5 comparison is 47.5 versus 43.4; center-and-size encoding gives 48.5 versus 47.5.
  • Ablation Study: A six-stage model improves mAP@0.25 and mAP@0.5 by 3.0 and 7.8 points over using no attention modules, while one attention module adds 0.95 GFLOPs.The attention cost is reported for 256 object candidates and 1,024 output points and is described as light relative to the backbone.

5. Conclusion

The paper presents a group-free 3D object detector that uses Transformer attention to compute object features from all point-cloud points, achieving state-of-the-art performance on ScanNet V2 and SUN RGB-D.

  • The detector computes each object candidate’s features from all point-cloud points rather than using a grouping step.Attention modules automatically determine each point’s contribution.
  • State-of-the-art performance is reported on the ScanNet V2 and SUN RGB-D benchmarks.

A1. Training Details

The appendix describes dataset-specific training settings, feature-aggregation comparisons, and ScanNet V2 evaluation details for the detector.

  • ScanNet V2: ScanNet V2 training uses 50k points, AdamW, 400 epochs, and learning-rate drops at epochs 280 and 340.The initial learning rate is 0.006, with random flipping, rotation, and scaling augmentations.
  • SUN RGB-D: SUN RGB-D training uses 20k points and adds an orientation prediction branch to every decoder layer.The branch combines orientation classification and offset regression losses.
  • Evaluation: ScanNet V2 evaluation with 40k points yields 66.3 mAP@0.25 and 48.5 mAP@0.5.These results are similar to 66.2 and 48.6 obtained with 50k points, respectively.
  • Feature aggregation: RoI-Pooling aggregates points within each predicted object box before predicting a refined box.Average-pooling and max-pooling are both evaluated as aggregation strategies.
  • Feature aggregation: Voting groups points whose predicted object centers lie within 0.3 meters of an object candidate center.A two-layer MLP with max-pooling forms the candidate aggregation feature.

A2. More Results

The appendix reports per-category quantitative results under multiple IoU thresholds and additional qualitative results on both evaluation datasets.

  • Per-category results are reported for mAP@0.25 and mAP@0.5 on ScanNet V2 and SUN RGB-D.Tables 12–15 contain the dataset- and threshold-specific results.
  • Additional qualitative results are shown for ScanNet V2 in Figure 6 and SUN RGB-D in Figure 7.

GT Ours

Figure 7 presents qualitative results on SUN RGB-D.

  • Figure 7 contains qualitative SUN RGB-D detection results.
Loading 2104.00678v2…