Source-linked AI summary
SoftGroup for 3D Instance Segmentation on Point Clouds
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo
TL;DR
Hard semantic predictions can propagate local classification errors into 3D instance grouping, reducing overlap and creating false positives. SoftGroup instead performs bottom-up grouping with soft semantic scores, then refines proposals top-down. It surpasses prior methods on ScanNet v2 and S3DIS and processes a ScanNet scene in 345ms.
Problem
Hard semantic predictions associate each point with one class, so locally ambiguous errors can produce low-overlap instances and false positives during grouping.
Method
SoftGroup groups points using thresholded soft semantic scores and applies top-down refinement that refines positive proposals and suppresses negative ones.
Results
+6.2% AP50 on the ScanNet hidden test set and +6.8% AP50 on S3DIS Area 5 over the previous strongest method.
Takeaways & Limitations
SoftGroup provides a simple, effective, and fast approach for 3D point-cloud instance segmentation across ScanNet v2 and S3DIS.
Abstract
from arXiv · showhide
Existing state-of-the-art 3D instance segmentation methods perform semantic segmentation followed by grouping. The hard predictions are made when performing semantic segmentation such that each point is associated with a single class. However, the errors stemming from hard decision propagate into grouping that results in (1) low overlaps between the predicted instance with the ground truth and (2) substantial false positives. To address the aforementioned problems, this paper proposes a 3D instance segmentation method referred to as SoftGroup by performing bottom-up soft grouping followed by top-down refinement. SoftGroup allows each point to be associated with multiple classes to mitigate the problems stemming from semantic prediction errors and suppresses false positive instances by learning to categorize them as background. Experimental results on different datasets and multiple evaluation metrics demonstrate the efficacy of SoftGroup. Its performance surpasses the strongest prior method by a significant margin of +6.2% on the ScanNet v2 hidden test set and +6.8% on S3DIS Area 5 in terms of AP_50. SoftGroup is also fast, running at 345ms per scan with a single Titan X on ScanNet v2 dataset. The source code and trained models for both datasets are available at \url{https://github.com/thangvubk/SoftGroup.git}.
1. Introduction
SoftGroup addresses error propagation from hard semantic predictions by grouping points using soft semantic scores and refining proposals top-down. It improves instance segmentation accuracy on ScanNet v2 and S3DIS while remaining fast.
- Motivation: Hard semantic grouping propagates locally ambiguous classification errors, causing low ground-truth overlap and false-positive instances.Points are assigned to one class before grouping, so wrongly labeled object regions can corrupt instance predictions.
- Method: SoftGroup performs grouping on soft semantic scores, allowing points to belong to multiple classes instead of using hard one-hot predictions.A score threshold identifies class memberships, preserving plausible true-class scores for incorrectly predicted object parts.
- Method: Top-down refinement classifies proposals as foreground or background, refining positive samples and suppressing false positives.Proposals are assigned positive or negative status using their maximum IoU with ground truth.
- Results: +6.2% AP50 on the ScanNet hidden test set and +6.8% AP50 on S3DIS Area 5 over the previous state-of-the-art method.Experiments use multiple datasets and evaluation metrics.
- Results: 345ms per scan on ScanNet v2 demonstrates that SoftGroup is fast as well as effective.The reported runtime uses a single Titan X.
2. Related work
Prior 3D point-cloud instance segmentation methods use proposal-based or grouping-based strategies, each with distinct strengths and drawbacks. SoftGroup belongs to the grouping-based family while addressing its dependence on semantic predictions.
- Deep Learning on 3D Point Clouds: Point-cloud deep learning methods commonly use pointwise networks or voxel-based sparse volumetric grids to extract features.Pointwise methods aggregate features from shared MLP outputs, whereas voxel-based methods apply sparse 3D convolutions.
- Instance Segmentation: Proposal-based methods generate object regions before segmentation, while grouping-based methods group per-point predictions into instances.GSPN and 3DSIS exemplify proposal-based methods; SGPN, JSIS3D, MTML, probabilistic clustering, and PointGroup exemplify grouping-based methods.
- SoftGroup: SoftGroup combines bottom-up grouping with top-down refinement to retain grouping efficiency while addressing semantic-error propagation.Its architecture uses a U-Net backbone, semantic and offset branches, soft grouping, and proposal-level refinement.
- Comparison: Proposal-based methods avoid interference between instances but struggle to generate high-quality 3D proposals from surface-only points.Grouping-based methods enable fast whole-scene inference but depend heavily on semantic segmentation accuracy.
3. Method
SoftGroup uses soft semantic scores for bottom-up point grouping, then applies top-down refinement to classify proposals, predict masks, and suppress false positives.
- 3.1. Point-wise Prediction Network: The point-wise prediction network voxelizes coordinate-color points and uses a sparse-convolution U-Net with semantic-score and offset-vector branches.Semantic scores cover Nclass classes, while offsets point toward each instance’s geometric center.
- 3.2. Soft Grouping: Soft grouping thresholds semantic scores so each point may belong to multiple classes, then links nearby shifted points within bandwidth b to form proposals.Class-specific point subsets are grouped separately, and their proposals are united for fast inference.
- 3.2. Soft Grouping: The score threshold trades recall against precision: lowering τ increases recall but also lowers precision, motivating top-down refinement.Hard semantic prediction has 79.1% recall, leaving more than 20% of points uncovered across classes.
- 3.3. Top-Down Refinement: Top-down refinement extracts proposal features and predicts instance classes, masks, and mask scores as final outputs.The classification branch derives the instance category and confidence, while the mask score estimates IoU and multiplies with classification confidence.
- 3.3. Top-Down Refinement: Proposals with IoU above 50% are positive, while the remaining proposals are negatives assigned to a background class for classification.Segmentation and mask scoring train only on positive samples; the network uses combined cross-entropy, binary cross-entropy, and ℓ2 losses end to end.
4. Experiments
SoftGroup is evaluated on ScanNet v2 and S3DIS using standard instance-segmentation metrics, benchmark comparisons, runtime analysis, qualitative results, and component ablations. The method achieves strong accuracy while remaining efficient, with gains linked to soft grouping, top-down refinement, and classification-based instance categorization.
- Experimental Settings: ScanNet v2 contains 1,613 scans and 18 evaluated classes, while S3DIS contains 271 scenes across six areas and 13 evaluated classes.S3DIS is evaluated on Area 5 and 6-fold cross-validation; ScanNet benchmark results use the hidden test split.
- Experimental Settings: AP50 and AP25 use 50% and 25% IoU thresholds, while AP averages scores from 50% to 95% in 5% increments.S3DIS additionally reports mean coverage, weighted coverage, precision, and recall.
- Benchmarking Results: 76.1% AP50 on the ScanNet v2 hidden test set surpasses the previous strongest method by 6.2%, with best class-wise performance in 12 of 18 classes.The model was submitted to the ScanNet server for hidden-test evaluation.
- Benchmarking Results: 51.6/66.1% AP/AP50 on S3DIS Area 5 improves over the second-best method by 8.9/6.8%, respectively.SoftGroup also achieves higher overall performance than existing methods under Area 5 and 6-fold cross-validation.
- Benchmarking Results: On ScanNet v2 validation, SoftGroup improves over the second-best method by 3.2, 3.3, 6.3, and 7.3% for AP50, AP25, box AP50, and box AP25.The evaluation covers both instance segmentation masks and object-detection boxes.
- Runtime Analysis: 345ms per ScanNet scan on Titan X is only 6ms slower than the fastest model, with 152ms, 132ms, and 70ms spent on prediction, soft grouping, and refinement.The runtime comparison uses the same Titan X GPU across methods.
- Component-wise Analysis: Combining soft grouping and top-down refinement reaches 46.0/67.6/78.9% AP/AP50/AP25, improving over the baseline by 6.5/6.5/3.4%.Either component alone provides significant improvement, while the combined model performs best.
- Ablation Study: The best soft-grouping performance uses score threshold τ=0.2, while combining all top-down refinement branches reaches 46.0/67.6/78.9% AP/AP50/AP25.Thresholds that are too high or too low perform worse than the hard-grouping baseline.
5. Conclusion
SoftGroup performs grouping on soft semantic scores and then refines instance proposals top-down. Across ScanNet v2 and S3DIS, it surpasses prior state of the art while processing a ScanNet scene in 345ms.
- 5. Conclusion: SoftGroup groups points using soft semantic scores to address errors from hard grouping on locally ambiguous objects.Its proposals are assigned positive or negative labels before top-down refinement refines positives and suppresses negatives.
- 5. Conclusion: SoftGroup improves over the existing state-of-the-art method by 6.2% on ScanNet v2 and 6.8% on S3DIS Area 5 in AP50.The reported results span two datasets and use hidden ScanNet testing and S3DIS Area 5 evaluation.
- 5. Conclusion: SoftGroup requires 345ms to process a ScanNet scene.The conclusion characterizes the method as fast in addition to accurate.