Source-linked AI summary
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia
TL;DR
3D point-cloud instance segmentation remains challenging because unordered points complicate grouping objects while requiring both semantic labels and instance identities. PointGroup predicts semantics and centroid-directed offsets, groups points in original and shifted coordinate sets, and scores candidates before NMS. It achieves 63.6% mAP50 on ScanNet v2 and 64.0% mAP50 on S3DIS, surpassing previous approaches.
Problem
3D point-cloud instance segmentation remains challenging because unordered, unstructured points make 2D methods difficult to extend, while the task requires semantic labels and instance IDs.
Method
PointGroup is an end-to-end bottom-up framework that predicts per-point semantics and centroid-directed offsets, clusters original and shifted coordinates, then uses ScoreNet and NMS to select instances.
Results
PointGroup achieves the highest reported performance on ScanNet v2 and S3DIS, reaching 63.6% mAP50 and 64.0% mAP50, respectively.
Takeaways & Limitations
Dual-coordinate point grouping and candidate scoring provide a framework that achieves state-of-the-art 3D instance segmentation results across two challenging datasets.
Takeaways & Limitations
Future work identifies semantic inaccuracy as affecting instance grouping and proposes progressive refinement, alongside weakly or self-supervised training.
Abstract
from arXiv · showhide
Instance segmentation is an important task for scene understanding. Compared to the fully-developed 2D, 3D instance segmentation for point clouds have much room to improve. In this paper, we present PointGroup, a new end-to-end bottom-up architecture, specifically focused on better grouping the points by exploring the void space between objects. We design a two-branch network to extract point features and predict semantic labels and offsets, for shifting each point towards its respective instance centroid. A clustering component is followed to utilize both the original and offset-shifted point coordinate sets, taking advantage of their complementary strength. Further, we formulate the ScoreNet to evaluate the candidate instances, followed by the Non-Maximum Suppression (NMS) to remove duplicates. We conduct extensive experiments on two challenging datasets, ScanNet v2 and S3DIS, on which our method achieves the highest performance, 63.6% and 64.0%, compared to 54.9% and 54.4% achieved by former best solutions in terms of mAP with IoU threshold 0.5.
1. Introduction
PointGroup addresses challenging 3D point-cloud instance segmentation by grouping points with semantic information and void-space geometry. Its dual-set grouping and ScoreNet achieve top results on ScanNet v2 and S3DIS.
- 3D instance segmentation must predict both semantic labels and instance IDs for every scene point, supporting applications such as autonomous driving and robot navigation.
- PointGroup is a bottom-up, end-to-end framework that predicts per-point semantics and offsets toward instance centroids.The offsets gather points from the same object while separating nearby objects of the same class.
- Dual-Set Point Grouping clusters same-label points in both original and offset-shifted coordinate sets, whose results complement each other.ScoreNet evaluates candidate groups, and NMS removes duplicate predictions.
- 63.6% mAP50 on ScanNet v2 is 8.7% higher than the former best solution, while S3DIS reaches 64.0% mAP50 and outperforms previous approaches.
- The method achieves state-of-the-art results across challenging datasets, demonstrating effectiveness and generality.
2. Related Work
Related work covers methods for irregular 3D point-cloud processing and two main instance-segmentation paradigms: detection-based and segmentation-based approaches. PointGroup differs by combining dual-coordinate clustering with ScoreNet in an end-to-end framework.
- Deep Learning in 3D Scenes: 3D point clouds are unordered and scattered, creating extra difficulty for scene understanding compared with regular 2D image grids.
- Deep Learning in 3D Scenes: Prior methods address point-cloud irregularity with MLPs, dynamic context aggregation, attention, learned ordering, voxelization, or multi-view projections.
- 2D Instance Segmentation: 2D instance segmentation includes top-down detection methods and bottom-up methods that group pixel-level semantic segmentations into object instances.
- 3D Instance Segmentation: 3D instance segmentation similarly includes detection-based box-and-mask methods and segmentation-based methods that group points using semantic labels, embeddings, or affinities.
- 3D Instance Segmentation: PointGroup combines semantic predictions in original and shifted coordinate spaces with ScoreNet for selecting clusters in a differentiable end-to-end framework.
3. Our Method
PointGroup is an end-to-end bottom-up framework that predicts per-point semantics and centroid-directed offsets, groups points in original and shifted spaces, and scores candidate clusters. Its dual-set grouping addresses nearby same-category objects and boundary inaccuracies through complementary coordinate representations.
- Architecture Overview: PointGroup separates instance segmentation into backbone feature extraction, point clustering, and ScoreNet-based proposal scoring.The clustering output is passed to ScoreNet, followed by proposal selection through NMS.
- Backbone Network: The backbone predicts semantic labels and per-point offset vectors that shift points toward their respective instance centroids.The semantic branch uses an MLP and cross-entropy loss, while the offset branch predicts three-dimensional vectors with regression supervision.
- Clustering Algorithm: Dual-Set Point Grouping clusters same-label nearby points separately in original coordinates P and offset-shifted coordinates Q.Shifted coordinates can separate nearby same-category objects, while original coordinates help when boundary offsets are inaccurate.
- ScoreNet: ScoreNet assigns scores to candidate clusters, which are then processed by non-maximum suppression to produce final instance predictions.Candidate clusters comprise the union of clusters found from P and Q.
- Backbone Network: The direction loss constrains predicted offset directions toward instance centroids, addressing inaccurate offset magnitudes for boundary points of large objects.The paper notes that point-to-centroid distances are commonly between 0 and 1m and that precise offsets are difficult for distant boundary points.
- Clustering Algorithm: The clustering algorithm uses a radius-constrained breadth-first search to form same-label connected components and discards stuff classes such as walls.Its inputs include the clustering radius, a point-count threshold, coordinates, and semantic labels.
4. Experiments
PointGroup is evaluated on ScanNet v2 and S3DIS using standard instance-segmentation metrics, ablations of its dual-coordinate clustering and ScoreNet, and runtime analysis. It achieves strong benchmark performance, while experiments show complementary coordinate sets and ScoreNet improve clustering-based instance selection.
- 4.2.1 Benchmark Results: PointGroup achieves the highest reported ScanNet v2 testing performance, with 63.6% AP50 versus 54.9% for the former best solution.It ranks first in 13 of 18 object categories.
- 4.2.2 Ablation Studies: Using both original coordinates P and shifted coordinates Q gives the best clustering performance, combining separation of nearby objects with robustness to offset errors.P-only clustering can merge nearby same-label objects, while Q-only clustering can be inaccurate near large-object boundaries.
- 4.2.2 Ablation Studies: The clustering radius trades off density sensitivity against merging nearby same-class objects, leading to an empirical setting of r = 0.03 meter.Small radii may fail to grow clusters in low-density regions, whereas large radii increase erroneous merging risk.
- 4.2.2 Ablation Studies: ScoreNet improves candidate-instance ranking: AP/AP50/AP25 are 34.8/56.9/71.3% with ScoreNet versus 30.2/51.9/68.9% using averaged semantic probability.The comparison evaluates ScoreNet as a source of quality scores for NMS.
- 4.2.3 Runtime Analysis: Inference time depends on scene point count and complexity, and clustering on shifted coordinates Q usually takes longer than clustering on original coordinates P.Shifted points can have more neighbors, increasing the clustering workload.
- 4.3. Evaluation on S3DIS: PointGroup reaches 57.8% AP50, 61.9% mPrec50, and 62.1% mRec50 on S3DIS Area 5.Its mPrec50 and mRec50 exceed ASIS by 6.6 and 19.7 points, respectively.
- 4.3. Evaluation on S3DIS: On S3DIS six-fold cross validation, PointGroup exceeds SGPN by 9.6 AP50 points and the second-best method by 4 and 21.6 points on mPrec50 and mRec50.The reported comparisons cover all three evaluation metrics.
5. Conclusion
PointGroup targets better 3D point grouping by combining semantic labels, centroid-directed offsets, and clustering over original and shifted coordinates. The authors identify semantic inaccuracy as a remaining issue for future refinement.
- PointGroup focuses on better grouping points by exploring the in-between space and point semantic labels among object instances.
- A two-branch network learns per-point semantic labels and offset vectors that move points toward their respective instance centroids.
- Clustering both original and offset-shifted coordinates combines their complementary strengths to improve point-grouping precision.
- Future work will address semantic inaccuracy affecting instance grouping through progressive refinement and explore weakly or self-supervised techniques.