Source-linked AI summary
Superpoint Transformer for 3D Scene Instance Segmentation
Jiahao Sun, Chunmei Qing, Junpeng Tan, Xiangmin Xu
TL;DR
3D instance segmentation methods are limited by imprecise proposals or semantic predictions and by time-consuming aggregation. SPFormer addresses these issues with superpoint-based query decoding and direct instance prediction, achieving state-of-the-art benchmark results while retaining fast inference.
Problem
Existing 3D instance segmentation pipelines rely on object-detection or semantic-segmentation results and require a time-consuming intermediate aggregation step.
Method
SPFormer is an end-to-end two-stage framework that groups potential point-cloud features into superpoints and directly predicts instances with transformer query vectors and superpoint cross-attention.
Results
54.9% mAP on the ScanNetv2 hidden test set exceeded the previous best result by 4.3%, while SPFormer achieved state-of-the-art results on ScanNetv2 and S3DIS.
Takeaways & Limitations
SPFormer provides a concise end-to-end framework that avoids intermediate aggregation and retains fast inference speed for 3D instance segmentation.
Takeaways & Limitations
Grouping-based approaches can propagate erroneous semantic predictions and require an additional refinement module because aggregation lacks supervision.
Abstract
from arXiv · showhide
Most existing methods realize 3D instance segmentation by extending those models used for 3D object detection or 3D semantic segmentation. However, these non-straightforward methods suffer from two drawbacks: 1) Imprecise bounding boxes or unsatisfactory semantic predictions limit the performance of the overall 3D instance segmentation framework. 2) Existing method requires a time-consuming intermediate step of aggregation. To address these issues, this paper proposes a novel end-to-end 3D instance segmentation method based on Superpoint Transformer, named as SPFormer. It groups potential features from point clouds into superpoints, and directly predicts instances through query vectors without relying on the results of object detection or semantic segmentation. The key step in this framework is a novel query decoder with transformers that can capture the instance information through the superpoint cross-attention mechanism and generate the superpoint masks of the instances. Through bipartite matching based on superpoint masks, SPFormer can implement the network training without the intermediate aggregation step, which accelerates the network. Extensive experiments on ScanNetv2 and S3DIS benchmarks verify that our method is concise yet efficient. Notably, SPFormer exceeds compared state-of-the-art methods by 4.3% on ScanNetv2 hidden test set in terms of mAP and keeps fast inference speed (247ms per frame) simultaneously. Code is available at https://github.com/sunjiahao1999/SPFormer.
1 Introduction
3D instance segmentation methods face accuracy and efficiency limitations, motivating SPFormer’s end-to-end hybrid design using superpoints and query-based instance prediction.
- 3D instance segmentation must detect instances in sparse point clouds and provide a clear mask for each instance.
- Proposal-based methods struggle with 3D bounding-box fitting because boxes have more degrees of freedom and point clouds cover only object surfaces.Low-quality proposals also degrade box-based bipartite matching and model performance.
- Grouping-based methods depend on semantic predictions and require an unsupervised intermediate aggregation step that increases training and inference time.Errors can propagate into later processing, while aggregation requires an additional refinement module.
- SPFormer is an end-to-end two-stage framework that groups bottom-up point-cloud features into superpoints and proposes instances through query vectors.It combines bottom-up grouping with a top-down proposal pipeline.
- Potential superpoint features avoid supervision through semantic and central-distance labels while representing homogeneous neighboring points.The method uses superpoints as a potential mid-level representation of 3D scenes.
- SPFormer achieves state-of-the-art performance on ScanNetv2 and S3DIS while retaining fast inference speed.Its training uses bipartite matching based on superpoint masks without the time-consuming aggregation step.
2 Related Work
Prior 3D instance segmentation work uses proposal-based or grouping-based pipelines, while transformer-based 3D segmentation faces computational overhead; SPFormer connects sparse features to query decoding through superpoints.
- Proposal-based Methods: Proposal-based methods generate region proposals or bounding boxes before predicting instance masks.
- Grouping-based Methods: Grouping-based methods learn point-level representations and aggregate points into instances using offsets, embeddings, or hierarchical procedures.
- 2D Instance Segmentation with Transformer: Transformer models have been applied broadly to 2D classification, detection, and segmentation, motivating their use for 3D instance segmentation.
- 2D Instance Segmentation with Transformer: Naively applying transformers to sparse-convolution outputs introduces high computational overhead because attention complexity grows with the input.
- SPFormer uses superpoints to bridge the sparse-convolution backbone and a transformer query decoder for 3D instance segmentation.
3 Method
SPFormer uses superpoint features and query-based transformer decoding to predict instances end to end, avoiding time-consuming aggregation. Bipartite matching trains predictions from superpoint masks, while iterative shared-head predictions and direct scoring support efficient inference.
- Bottom-up Grouping: SPFormer extracts point-wise features with a sparse 3D U-net and averages features within pre-computed superpoints, reducing subsequent processing to hundreds of elements.The pooling layer produces superpoint features S ∈ R^M×C and downsampled representations for later processing.
- Query Decoder: Learnable query vectors capture instance information through superpoint cross-attention in stacked transformer decoder layers.The decoder exchanges self-attention and cross-attention order, removes position embedding, and uses variable-length superpoint inputs.
- Query Decoder: Predicted masks from earlier decoder layers create attention masks that, with threshold τ = 0.5, constrain later cross-attention to foreground superpoints.The attention masks are formed from former-layer superpoint masks and adaptively restrict attention within the foreground instance.
- Inference: SPFormer directly predicts classifications, IoU-aware scores, and superpoint masks, then sorts instances without non-maximum suppression.Its final score combines classification probability, IoU-aware score, and mask score, supporting fast post-processing.
- Iterative Prediction: Iterative prediction applies a shared prediction head to every decoder-layer output during training but uses only the final head during inference.Layer-wise supervision updates query features progressively while final-layer-only inference avoids redundant proposals and accelerates inference.
- Bipartite Matching and Loss Function: Bipartite matching assigns fixed-number proposals to ground truth using classification and superpoint-mask costs, followed by classification, mask, and score losses.Unassigned proposals receive the “no instance” class; mask loss combines BCE and Dice, while score supervision uses an L2 loss for high-quality proposals.
4 Experiments
Experiments show SPFormer achieves strong accuracy across ScanNetv2 and S3DIS while maintaining fast inference. Ablations identify superpoints, mask matching, transformer design, query capacity, and mask-loss composition as important factors.
- Benchmark Results: 54.9% mAP on the ScanNetv2 hidden test set exceeds the previous best result by 4.3%.SPFormer also achieves the highest AP scores in 8 of 18 categories.
- Benchmark Results: 6.9%, 6.3%, and 4.0% improvements over the second-best ScanNetv2 validation results are reported for mAP, AP50, and AP25, respectively.
- Benchmark Results: SPFormer achieves state-of-the-art AP50 and competitive mPrec/mRec results on S3DIS under Area 5 and 6-fold cross-validation.The authors report these results as confirming the method's generalization ability.
- Runtime Analysis: SPFormer is the fastest compared method on ScanNetv2 validation inference, even when superpoint extraction time is included.Superpoints can be pre-computed during training, which can further reduce model training time.
- Components Analysis: Mask-based bipartite matching outperforms box-based matching by 6.4% on mAP, while the IoU-aware score branch improves mAP/AP50/AP25 by +1.3/1.5/0.4.The analysis attributes the matching difference to the greater degrees of freedom of 3D boxes and usually undetectable object geometric centers.
- Ablation Study: Superpoints substantially improve query decoding over directly feeding backbone outputs, and transformer attention-mask design adds +3.8/2.4/1.3 performance on mAP/AP50/AP25.Shared prediction heads add +1.5/1.8/1.8, while performance saturates at 800 query vectors and the Dice-plus-BCE mask-loss combination performs best.
5 Conclusion
SPFormer is an end-to-end two-stage framework for 3D instance segmentation that combines bottom-up grouping with top-down instance proposal. It achieves state-of-the-art performance on ScanNetv2 and S3DIS while retaining fast inference speed.
- SPFormer is a novel end-to-end two-stage framework for 3D instance segmentation.
- Its hybrid pipeline groups bottom-up potential point-cloud features into superpoints and proposes instances through query vectors.
- SPFormer achieves state-of-the-art results on both ScanNetv2 and S3DIS benchmarks while retaining fast inference speed.