PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation

• 3D instance segmentation must predict both semantic labels and instance IDs for every scene point, supporting applications such as autonomous driving and robot navigation.Source: §1 · p. 1
• PointGroup is a bottom-up, end-to-end framework that predicts per-point semantics and offsets toward instance centroids.The offsets gather points from the same object while separating nearby objects of the same class.Source: §1 · pp. 1–2
• Dual-Set Point Grouping clusters same-label points in both original and offset-shifted coordinate sets, whose results complement each other.ScoreNet evaluates candidate groups, and NMS removes duplicate predictions.Source: §1 · pp. 1–2
• 63.6% mAP50 on ScanNet v2 is 8.7% higher than the former best solution, while S3DIS reaches 64.0% mAP50 and outperforms previous approaches.Source: §1 · p. 2
• The method achieves state-of-the-art results across challenging datasets, demonstrating effectiveness and generality.Source: §1 · pp. 1–2

Deep Learning in 3D Scenes

• 3D point clouds are unordered and scattered, creating extra difficulty for scene understanding compared with regular 2D image grids.Source: §2 · p. 2
• Prior methods address point-cloud irregularity with MLPs, dynamic context aggregation, attention, learned ordering, voxelization, or multi-view projections.Source: §2 · p. 2

2D Instance Segmentation

• 2D instance segmentation includes top-down detection methods and bottom-up methods that group pixel-level semantic segmentations into object instances.Source: §2 · p. 2

3D Instance Segmentation

• 3D instance segmentation similarly includes detection-based box-and-mask methods and segmentation-based methods that group points using semantic labels, embeddings, or affinities.Source: §2 · p. 2
• PointGroup combines semantic predictions in original and shifted coordinate spaces with ScoreNet for selecting clusters in a differentiable end-to-end framework.Source: §2 · p. 2

Architecture Overview

• PointGroup separates instance segmentation into backbone feature extraction, point clustering, and ScoreNet-based proposal scoring.The clustering output is passed to ScoreNet, followed by proposal selection through NMS.Source: §3.1 · p. 3

Backbone Network

• The backbone predicts semantic labels and per-point offset vectors that shift points toward their respective instance centroids.The semantic branch uses an MLP and cross-entropy loss, while the offset branch predicts three-dimensional vectors with regression supervision.Source: §3.1 · p. 3, §3.2 · p. 4

Clustering Algorithm

• Dual-Set Point Grouping clusters same-label nearby points separately in original coordinates P and offset-shifted coordinates Q.Shifted coordinates can separate nearby same-category objects, while original coordinates help when boundary offsets are inaccurate.Source: §3.1 · p. 3, §3.3 · p. 4

ScoreNet

• ScoreNet assigns scores to candidate clusters, which are then processed by non-maximum suppression to produce final instance predictions.Candidate clusters comprise the union of clusters found from P and Q.Source: §3.1 · p. 3, §3.4 · p. 4

Backbone Network

• The direction loss constrains predicted offset directions toward instance centroids, addressing inaccurate offset magnitudes for boundary points of large objects.The paper notes that point-to-centroid distances are commonly between 0 and 1m and that precise offsets are difficult for distant boundary points.Source: §3.2 · p. 4, Figure 3 · p. 5

Clustering Algorithm

• The clustering algorithm uses a radius-constrained breadth-first search to form same-label connected components and discards stuff classes such as walls.Its inputs include the clustering radius, a point-count threshold, coordinates, and semantic labels.Source: §3.3 · p. 4

4.2.1 Benchmark Results

• PointGroup achieves the highest reported ScanNet v2 testing performance, with 63.6% AP50 versus 54.9% for the former best solution.It ranks first in 13 of 18 object categories.Source: §4.2.1 · p. 6, Table 1 · p. 6

4.2.2 Ablation Studies

• Using both original coordinates P and shifted coordinates Q gives the best clustering performance, combining separation of nearby objects with robustness to offset errors.P-only clustering can merge nearby same-label objects, while Q-only clustering can be inaccurate near large-object boundaries.Source: §4.2.2 · p. 6, Table 2 · p. 7
• The clustering radius trades off density sensitivity against merging nearby same-class objects, leading to an empirical setting of r = 0.03 meter.Small radii may fail to grow clusters in low-density regions, whereas large radii increase erroneous merging risk.Source: §4.2.2 · p. 6, Table 3 · p. 7
• ScoreNet improves candidate-instance ranking: AP/AP50/AP25 are 34.8/56.9/71.3% with ScoreNet versus 30.2/51.9/68.9% using averaged semantic probability.The comparison evaluates ScoreNet as a source of quality scores for NMS.Source: §4.2.2 · p. 7

4.2.3 Runtime Analysis

• Inference time depends on scene point count and complexity, and clustering on shifted coordinates Q usually takes longer than clustering on original coordinates P.Shifted points can have more neighbors, increasing the clustering workload.Source: §4.2.3 · p. 7, Table 4 · p. 7

4.3. Evaluation on S3DIS

• PointGroup reaches 57.8% AP50, 61.9% mPrec50, and 62.1% mRec50 on S3DIS Area 5.Its mPrec50 and mRec50 exceed ASIS by 6.6 and 19.7 points, respectively.Source: Table 5 · p. 7, §4.3 · p. 8
• On S3DIS six-fold cross validation, PointGroup exceeds SGPN by 9.6 AP50 points and the second-best method by 4 and 21.6 points on mPrec50 and mRec50.The reported comparisons cover all three evaluation metrics.Source: §4.3 · p. 8

• PointGroup focuses on better grouping points by exploring the in-between space and point semantic labels among object instances.Source: §5 · p. 8
• A two-branch network learns per-point semantic labels and offset vectors that move points toward their respective instance centroids.Source: §5 · p. 8
• Clustering both original and offset-shifted coordinates combines their complementary strengths to improve point-grouping precision.Source: §5 · p. 8
• Future work will address semantic inaccuracy affecting instance grouping through progressive refinement and explore weakly or self-supervised techniques.Source: §5 · p. 8