Source-linked AI summary
Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia
TL;DR
3D instance segmentation is challenged by irregular point clouds, unknown instance counts, and separately trained point grouping. SSTNet addresses this with superpoint aggregation, a semantic superpoint tree, divisive proposal generation, and CliqueNet refinement. It reports results on ScanNet and S3DIS, including 2% higher mAP than the second-best method on the ScanNet (V2) leaderboard at submission time.
Problem
Irregular point clouds, unknown instance counts, and separately trained point grouping make 3D instance segmentation challenging.
Method
SSTNet aggregates point features into superpoints, constructs and traverses a semantic superpoint tree, and uses CliqueNet to prune wrongly grouped superpoints.
Results
2% higher mAP than the second-best method was reported on the ScanNet (V2) leaderboard at submission time, with experiments on ScanNet and S3DIS.
Takeaways & Limitations
The end-to-end superpoint-based design directly proposes and evaluates object instances while using geometric regularity to support consistent, sharp segmentations.
Takeaways & Limitations
The method assumes individual superpoints do not cross object boundaries, although this is not guaranteed.
Abstract
from arXiv · showhide
Instance segmentation in 3D scenes is fundamental in many applications of scene understanding. It is yet challenging due to the compound factors of data irregularity and uncertainty in the numbers of instances. State-of-the-art methods largely rely on a general pipeline that first learns point-wise features discriminative at semantic and instance levels, followed by a separate step of point grouping for proposing object instances. While promising, they have the shortcomings that (1) the second step is not supervised by the main objective of instance segmentation, and (2) their point-wise feature learning and grouping are less effective to deal with data irregularities, possibly resulting in fragmented segmentations. To address these issues, we propose in this work an end-to-end solution of Semantic Superpoint Tree Network (SSTNet) for proposing object instances from scene points. Key in SSTNet is an intermediate, semantic superpoint tree (SST), which is constructed based on the learned semantic features of superpoints, and which will be traversed and split at intermediate tree nodes for proposals of object instances. We also design in SSTNet a refinement module, termed CliqueNet, to prune superpoints that may be wrongly grouped into instance proposals. Experiments on the benchmarks of ScanNet and S3DIS show the efficacy of our proposed method. At the time of submission, SSTNet ranks top on the ScanNet (V2) leaderboard, with 2% higher of mAP than the second best method. The source code in PyTorch is available at https://github.com/Gorilla-Lab-SCUT/SSTNet.
1. Introduction
3D instance segmentation is difficult because point clouds are irregular, instance counts are unknown, and point-wise grouping can produce fragmented results. SSTNet addresses these issues with end-to-end superpoint-tree grouping, refinement, and efficient divisive proposals.
- 3D instance segmentation must assign semantic labels and distinguish points belonging to different object instances.
- Irregular point distributions, unknown instance counts, and inconsistent neighboring predictions make the task challenging.
- Existing methods learn point-wise semantic and instance features before separately clustering points, leaving grouping independent of network training.
- SSTNet aggregates point features into superpoints, constructs a semantic superpoint tree, and traverses and splits it to propose instances.
- 2% higher mAP than the second-best method was reported on the ScanNet (V2) leaderboard at submission time.
- Divisive grouping with tree construction, traversal, and node splitting is reported as an order of magnitude faster than agglomerative grouping.
- CliqueNet treats each proposal branch as a graph clique and prunes superpoints that may be incorrectly grouped.
2. Related Works
Related work emphasizes geometric regularization for irregular point clouds, while SSTNet uses superpoints and hierarchical tree construction to organize instance proposals.
- 3D Semantic Segmentation: Prior 3D semantic-segmentation methods regularize irregular point clouds through projection, voxelization, or local aggregation.
- SSTNet’s overview presents a pipeline from input scenes and superpoints to semantic superpoint trees and proposals after tree traversal and splitting.
3. Overview
SSTNet converts irregular scene points into geometrically homogeneous superpoints, then uses semantic features and a divisive tree pipeline to generate and refine instance proposals.
- Superpoints are geometrically homogeneous local neighborhoods computed by over-segmenting the input point set with graph partitioning.
- Grouping spatially close superpoints that share an object instance turns instance segmentation into a 3D clustering problem.
- The method assumes individual superpoints do not cross object boundaries, trading this guarantee for geometric regularity.
- A backbone predicts point features, semantic scores, and instance-center offsets, which are average-pooled within each superpoint.
- SSTNet constructs a semantic superpoint tree and uses top-down divisive grouping rather than agglomerative grouping for efficient training.
- CliqueNet converts proposal branches into graph cliques and prunes branch nodes before ScoreNet evaluates the generated proposals.
4. Individual Modules of the Proposed Network
SSTNet builds a semantic superpoint tree from pooled point features, traverses it to propose instances, and refines proposals with CliqueNet. Its modules jointly address irregular point data and erroneous superpoint grouping.
- Backbone and semantic scoring: A U-Net-style sparse-convolution backbone learns point-wise features, while MLPs produce semantic scores and offset predictions.Semantic scoring uses cross-entropy with a dice-loss component; offset prediction targets object-instance centers.
- Semantic superpoint tree: Point-wise features are average-pooled within precomputed superpoints to obtain superpoint-level representations.Superpoints provide geometrically homogeneous neighborhoods for subsequent tree construction.
- Semantic superpoint tree: SST construction greedily merges superpoints using Euclidean distance between augmented semantic scores that combine semantic predictions and predicted centers.Merged nodes inherit features, semantic scores, and offsets through size-weighted averaging; the tree depth ranges between logM and M −1.
- Proposal generation: A binary classifier traverses the tree breadth-first, splitting nodes when child superpoints are predicted to belong to different instances and accepting branches otherwise.The classifier is trained with soft instance labels inherited from superpoints through weighted averaging.
- Proposal generation: SSTNet uses divisive grouping after tree construction, rather than the agglomerative grouping common in hierarchical image segmentation.The stated motivation is computational efficiency from nearest-neighbor search during tree construction.
- CliqueNet refinement: CliqueNet converts each proposed branch into an independent graph clique and learns feature interactions to prune superpoints assigned to the wrong instance or background.The clique connects the branch node directly with its leaf superpoints and uses an adjacency matrix to specify node connections.
5. Experiments
Experiments evaluate SSTNet on ScanNet and S3DIS, including ablations of SST construction, proposal generation, and CliqueNet refinement. SSTNet outperforms alternatives and existing methods across the reported benchmark evaluations.
- ScanNet experiments: ScanNet experiments include hidden-test leaderboard submissions and validation-set analyses of SST construction, proposal generation, and proposal refinement.The reported studies use ScanNet (V2), with Tables 1–5 covering benchmark results and ablations.
- Proposal-generation ablations: SST-thresholding performs best at mAP@25, while SSTNet greatly outperforms two alternatives on averaged mAP.The alternatives are a thresholded feature-based strategy and Superpoint Graph.
- CliqueNet refinement: Pruning superpoints from proposed tree branches is effective in high-precision mAP regimes.This result is reported for the CliqueNet refinement ablation.
- ScanNet results: SSTNet outperforms all existing methods on the ScanNet (V2) hidden test set.The model was trained on the ScanNet training set and evaluated through the testing server.
- S3DIS results: SSTNet outperforms all existing methods on S3DIS using mAP, AP@50, mean precision, and mean recall at IoU threshold 0.5.Evaluation follows Area-5 and 6-fold cross-validation protocols.