Group-Free 3D Object Detection via Transformers

1. Introduction

• Hand-crafted grouping schemes assign points using 2D frusta, 3D box proposals, or vote proximity, but complex and diverse real-scene objects can cause wrong assignments and degrade detection.The cited examples are Frustum-PointNet [27], Point R-CNN [35], and VoteNet [26].Source: §1 · p. 1
• The method computes each object candidate’s features from all point-cloud points, using learned attention to model object-object and object-pixel relationships without handcrafted grouping.The contribution of each point is determined automatically by an attention module.Source: §1 · p. 2
• +3.8 mAP@0.25 is the reported SUN RGB-D improvement with the ensemble scheme; the approach achieves state-of-the-art performance on ScanNet V2 and SUN RGB-D.The introduction does not specify the baseline for the reported improvement.Source: §1 · p. 2
• It iteratively refines object predictions by updating spatial encodings across stages and ensembles detections from all stages at inference, with few computational overheads.This differs from fixed spatial encoding and using only the final-stage predictions.Source: §1 · p. 2
• The method is reported effective and robust to initial object-candidate quality, with even farthest point sampling producing strong results on both benchmarks.The benchmarks are ScanNet V2 [6] and SUN RGB-D.Source: §1 · p. 2
• The authors argue attention and Transformers naturally address irregular, sparse point-cloud modeling, unlike their challenger or complementary role alongside mature grid-based image tools.The passage names ConvNets variants [16] [32] [46] and RoI Align [2] [5] as examples of those image tools.Source: §1 · p. 2

Grid Projection/Voxelization based Detection

• Projection- and voxelization-based methods use 2D or 3D ConvNets, but suffer quantization errors; voxel methods also incur high memory and computational costs.Bird’s-eye-view projections are mainly used for outdoor driving scenes, where objects lie on a horizontal plane and their projections are occlusion-free; projection methods also address sparse points through pixelization.Source: §2 · p. 2

Point based Detection

• Existing point-based detectors assign point groups to object candidates before feature extraction, using 2D-box frustums [27], points within 3D-box proposals [35], or voted centers [26].Follow-up works improve point-group generation [51] or object box localization and recognition.Source: §2 · p. 2
• Unlike prior point-based approaches, this method computes each candidate’s features from all points using attention, then stacks attention modules to iteratively refine detections.The paper identifies point grouping as an issue for point-cloud detection that this architecture may address.Source: §2 · pp. 2–3

Network architecture for Point Cloud

• The method can use any point-cloud architecture as its backbone and adopts PointNet++ [30], also used in previous methods [26] [25] [51], for a fair comparison.A review [13] provides a taxonomy of point-cloud architectures; discussing them all is beyond the paper’s scope.Source: §2 · p. 2

Attention Mechanism/Transformer in NLP and 2D Image Recognition

• Transformers are established in NLP [7] [21] and have been applied to image recognition [16] [32] [46] and 2D detection [15] [11] [5] [2].The paper finds its most similar 2D detection approach [2] performs significantly worse when directly applied to 3D point clouds, and describes adapting Transformers through multi-stage query-location updates and an ensemble of stage detections.Source: §2 · pp. 2–3

3. Methodology

• The backbone uses PointNet++ to encode 2,048 input points through down- and up-sampling, producing 1,024 per-point feature vectors for candidate sampling and attention.The encoder-decoder downsamples to 256 points through four set-abstraction stages, then upsamples to 1,024 points through feature-propagation layers.Source: §3 · p. 3

3.1. Initial Object Candidate Sampling

• Candidates are sampled bottom-up because 3D anchor boxes are impractical; the framework considers FPS, KPS, and KPS-NMS, using KPS by default.KPS labels points inside a ground-truth box and among the k closest to its center as positive; KPS-NMS additionally suppresses nearby candidates using a 0.05 radius. KPS outperforms FPS and is as effective as KPS-NMS in the reported experiments.Source: §3.1 · pp. 3–4

3.2. Iterative Object Feature Extraction and Box Prediction by Transformer Decoder

• A Transformer decoder computes candidate features from all point-cloud points in parallel, using stacked self-attention, cross-attention, and feed-forward modules.Self-attention models interactions among object features, while cross-attention aggregates point features for each object.Source: §3.2 · p. 4
• Iterative box refinement improves ScanNet V2 mAP@0.25 and mAP@0.5 by 1.6 and 5.0, respectively, while multi-stage ensembling adds +3.8 mAP@0.25 on SUN RGB-D.Each stage's predicted box location refines that object's spatial encoding for the next stage; predictions from different stages are combined through NMS at IoU threshold 0.25.Source: §3.2 · pp. 4–5

3.3. Heads and Loss Functions

• Decoder stages predict objectness, box class, center offset, size class, and size offset through a shared two-layer MLP and task-specific linear layers.The sampling head omits box classification and uses the candidate-label assignment described for KPS; the final loss sums sampling-head and decoder-head losses.Source: §3.3 · p. 5

4.1. Datasets and Evaluation Protocol

• Evaluation uses standard splits and mAP at varying IoU thresholds on ScanNet V2, and mAP on the 10 most common SUN RGB-D categories.ScanNet V2 contains 1,513 scenes and 18 categories; SUN RGB-D comprises about 5K indoor RGB-D images with annotations across 37 categories.Source: §4.1 · p. 5

4.2. Implementation Details

• The ScanNet V2 backbone is PointNet++; SUN RGB-D uses 20K input points and adds orientation prediction branches at all decoder stages.The ScanNet V2 backbone uses four set-abstraction layers and two feature-propagation layers; the SUN RGB-D architecture and augmentation otherwise match ScanNet V2.Source: §4.2 · p. 5

4.3. System-level Comparison

• 69.1 mAP@0.25 and 52.8 mAP@0.5 with stronger backbones and 512 candidates outperform previous best ScanNet V2 results by a large margin; SUN RGB-D scores 63.0 mAP@0.25 and 45.2 mAP@0.5, exceeding prior point-cloud-only methods and H3DNet by 6.2 mAP@0.5.The SUN RGB-D ensemble combines predictions across decoder stages during inference.Source: Table 1 · p. 6, §4.3 · p. 6, Table 2 · p. 6

4.4. Ablation Study

• Different sampling strategies and a wide range of KPS k values perform well, indicating robust detection across sampling choices.For iterative box prediction, stage-by-stage location-encoding refinement improves mAP@0.5 by 4.1 points over the naive implementation (47.5 vs. 43.4); encoding both box center and size also beats center-only encoding (66.3 vs. 65.2 mAP@0.25; 48.5 vs. 47.5 mAP@0.5).Source: §4.4 · pp. 6–7, Table 6 · p. 7
• Multi-stage ensembling improves SUN RGB-D by 3.8 mAP@0.25 and 1.9 mAP@0.5 while maintaining ScanNet V2 performance; group-free detection exceeds RoI-Pooling by 1.2 and 4.1 points on those respective metrics.The grouping comparison changes only feature aggregation and uses a six-stage decoder; RoI-Pooling outperforms voted-center grouping but remains below the group-free approach.Source: §4.4 · p. 7, Table 7 · p. 7, Table 8 · p. 7

4.5. Inference Speed

• With 256 candidates and 1,024 output points, one attention model adds 0.95 GFLOPs, and the method achieves better performance and higher inference speed than compared state-of-the-art methods.The realistic-speed comparison uses the same workstation and environment, official code for other methods, and batch size 1.Source: §4.5 · p. 7, Table 9 · p. 7

4.6. Comparison with DETR

• With 400 training epochs, DETR reaches 39.6 mAP@0.25 and 21.4 mAP@0.5; bridging representation and refinement differences raises results to 59.9 and 42.9, or 61.8 and 45.2 with longer training.The passage attributes the remaining gap to DETR’s set-loss ground-truth assignments versus the manual candidate-to-ground-truth assignments in this approach, which may be difficult for a network to learn.Source: §4.6 · p. 8, Table 10 · p. 8

4.7. Qualitative Results

• On ScanNet V2 and SUN RGB-D, deeper decoder stages produce more accurate detections, while higher-stage attention focuses more on object geometry and extracts higher-quality features.Lower-stage models attend to surrounding points without considering geometry; the visualized attention becomes more geometry-focused with refinement.Source: §4.7 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

• Unlike methods requiring grouping, the detector computes object features from all points, with attention modules automatically determining each point’s contribution.Source: §5 · pp. 8–9
• The detector achieves state-of-the-art performance on the ScanNet V2 and SUN RGB-D benchmarks.Source: §5 · p. 9

A1.1. Our Approach

• ScanNet training uses 50k input points, AdamW from scratch for 400 epochs, staged 10× learning-rate drops, and attention-module learning rates one-tenth of the backbone rate.Augmentation includes random flips, rotations between [−5°, 5°], and scaling by [0.9, 1.1]; weight decay is 5e-4, gradnorm clipping is applied, and the size head is class-aware. The PointNet++ backbone has four set abstraction and two feature propagation layers.Source: A1.1. Our Approach · p. 9
• SUN RGB-D uses 20k input points and adds orientation prediction at every decoder layer, with classification and offset-regression loss weights of 0.1 and 0.04.Training uses AdamW from scratch for 600 epochs by default, with learning-rate drops at epochs 420, 480, and 540; attention-module learning rates are one-twentieth of the backbone rate, and size prediction uses a class-agnostic head.Source: A1.1. Our Approach · p. 9
• On ScanNet V2, 40k-point evaluation is similar to 50k-point evaluation: 66.3(40k) vs. 66.2(50k) on mAP@0.25 and 48.5(40k) vs. 48.6(50k) on mAP@0.5.The paper says the 40k-point evaluation follows previous works.Source: A1.1. Our Approach · p. 9

A1.2. Other Pooling Mechanisms

• RoI-Pooling aggregates points inside predicted boxes for multi-stage refinement, and max-pooling performs better than average-pooling; Voting groups points whose predicted centers are within 0.3 meters of a candidate center.Voting uses a two-layer MLP with max-pooling to form candidate features; for these comparisons, only the feature aggregation mechanism is switched while other settings remain unchanged.Source: A1.2. Other Pooling Mechanisms · p. 9, Table 11 · p. 9

A2. More Results

• Per-category results are reported for ScanNet V2 and SUN RGB-D at mAP@0.25 and mAP@0.5.Tables 12–13 cover ScanNet V2; Tables 14–15 cover SUN RGB-D.Source: A2. More Results · p. 9
• Qualitative detection results are shown for ScanNet V2 and SUN RGB-D.The examples appear in Figures 6 and 7, respectively.Source: A2. More Results · p. 9

• Figure 7 presents qualitative results on SUN RGB-D.Source: Figure 7 · p. 12