Source-linked AI summary
Hierarchical Aggregation for 3D Instance Segmentation
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang
TL;DR
Bottom-up clustering for 3D point-cloud instance segmentation often produces over- or under-segmentation because points have weak instance-level information and scenes vary widely. HAIS uses hierarchical point-to-set and set-to-instance aggregation, followed by intra-instance prediction for filtering and scoring. It achieves state-of-the-art results on ScanNet v2 and S3DIS while retaining efficient single-forward inference.
Problem
Bottom-up clustering commonly suffers over- and under-segmentation because point clouds contain variable numbers and sizes of instances, while individual points have weak instance-level features.
Method
HAIS hierarchically aggregates points into preliminary sets, combines sets into complete instances, and applies intra-instance prediction for outlier filtering and mask quality scoring.
Results
HAIS achieves state-of-the-art performance on ScanNet v2 and S3DIS, ranks first on ScanNet v2, and runs at 410 ms per frame without post-processing.
Takeaways & Limitations
HAIS demonstrates effective and generalized 3D instance segmentation with a concise, latency-efficient inference pipeline.
Abstract
from arXiv · showhide
Instance segmentation on point clouds is a fundamental task in 3D scene perception. In this work, we propose a concise clustering-based framework named HAIS, which makes full use of spatial relation of points and point sets. Considering clustering-based methods may result in over-segmentation or under-segmentation, we introduce the hierarchical aggregation to progressively generate instance proposals, i.e., point aggregation for preliminarily clustering points to sets and set aggregation for generating complete instances from sets. Once the complete 3D instances are obtained, a sub-network of intra-instance prediction is adopted for noisy points filtering and mask quality scoring. HAIS is fast (only 410ms per frame) and does not require non-maximum suppression. It ranks 1st on the ScanNet v2 benchmark, achieving the highest 69.9% AP50 and surpassing previous state-of-the-art (SOTA) methods by a large margin. Besides, the SOTA results on the S3DIS dataset validate the good generalization ability. Code will be available at https://github.com/hustvl/HAIS.
1. Introduction
HAIS addresses the over- and under-segmentation difficulties of bottom-up 3D point-cloud instance segmentation with hierarchical aggregation and intra-instance prediction. It achieves strong benchmark performance and efficient inference across ScanNet v2 and S3DIS.
- 3D point-cloud instance segmentation supports scene understanding applications including robotics, augmented and virtual reality, and autonomous driving.
- Direct clustering is difficult because scenes contain many variable-sized instances while individual points provide weak instance-level features.
- HAIS first aggregates points into sets with low bandwidth, then uses dynamic-bandwidth set aggregation to form complete instances and filters noisy points.
- The framework combines hierarchical aggregation with an intra-instance prediction network to produce more fine-grained instance predictions.
- HAIS ranks first on ScanNet v2 and achieves state-of-the-art results on S3DIS, demonstrating generalization across challenging datasets.
- 410 ms per frame is the reported average ScanNet v2 inference time, using a concise single-forward pipeline without post-processing.
2. Related Works
HAIS belongs to the clustering-based 3D instance-segmentation paradigm but differs through a concise pipeline and object-level refinement. Its framework combines point-wise feature learning, point aggregation, set aggregation, and intra-instance prediction.
- Deep Learning on Point Clouds: Voxel-based point-cloud methods convert unordered points into volumetric grids and use 3D sparse convolutions for feature extraction.
- Instance Segmentation Paradigms: Proposal-based methods generate object proposals before predicting masks, whereas clustering-based methods predict point-wise labels and cluster points into instances.
- Clustering-based Instance Segmentation: HAIS differs from earlier clustering methods by using a concise pipeline and adding set aggregation plus intra-instance prediction for object-level refinement.
- HAIS Framework: The framework extracts point-wise features and predictions, forms preliminary instances through point aggregation, expands them with set aggregation, and filters outliers.
3. Method
HAIS builds instances through hierarchical aggregation: point-level predictions form preliminary sets, set-level clustering restores missing parts, and intra-instance prediction refines masks and scores.
- Point-wise Prediction Network: The point-wise prediction network extracts point features and predicts semantic labels plus per-point center shift vectors.Input point clouds are voxelized for sparse-convolution feature extraction, then mapped back to point features.
- Point Aggregation: Center shifts move points toward instance centers, after which same-label foreground points within fixed bandwidth rpoint are connected into preliminary instances.Each connected set becomes a preliminary instance prediction.
- Set Aggregation: Point aggregation produces primary instances and small fragments because inaccurate center shifts prevent all points from an instance being grouped together.Fragments may represent missing parts of primary instances, so hard filtering would be inappropriate.
- Set Aggregation: Set aggregation assigns fragments to nearby same-label primary instances using a dynamic bandwidth rset = max(rsize, rcls).The size-specific term expands with primary-instance size, while the class-specific term reflects the average instance radius for that class.
- Intra-instance Prediction Network: The intra-instance prediction network predicts binary masks and certainty scores to filter noisy points and refine aggregated instances.Masks supervise foreground separation and IoU-based instance certainty; only instances with IoU higher than 0.5 train the mask branch.
- NMS-free Inference: Mask-assisted score prediction improves performance, while HAIS avoids NMS because point aggregation assigns each point to only one instance.The resulting instance predictions do not overlap, allowing certainty scores to rank them directly.
4. Experiments
Experiments evaluate HAIS on ScanNet v2 and S3DIS, using quantitative, qualitative, ablation, and efficiency analyses. HAIS achieves leading accuracy, generalizes across datasets, improves through its components, and supports fast inference.
- Quantitative Evaluation: 69.9% AP50 ranks HAIS first on the ScanNet v2 testing leaderboard, surpassing the previous SOTA by 6.1%.HAIS also performs best in 12 of 18 evaluated classes.
- Quantitative Evaluation: HAIS achieves SOTA results across all reported S3DIS metrics, supporting generalization across substantially different datasets.The datasets differ in category, scene style, and point-cloud density.
- Ablation Study: 2.7% AP, 2.4% AP50, and 0.3% AP25 are gained from intra-instance prediction, while set aggregation adds 1.0% AP, 0.7% AP50, and 0.7% AP25.These ablations are conducted on the ScanNet v2 validation set.
- Ablation Study: Mask-based point filtering and mask-to-ground-truth IoU supervision improve instance scoring by reducing background noise and providing a more suitable certainty signal.The ablation compares mask-based scoring with using whole instance features and original input instances.
- Ablation Study: Filtering low-quality mask-training instances at an IoU threshold of 0.5 benefits instance-level refinement because they contain few foreground points and much background.The paper reports that these samples introduce ambiguity during training.
- Efficiency Evaluation: 410 ms per frame is the reported average ScanNet v2 inference time, with HAIS evaluated against other methods on the same Titan X GPU.The study uses whole-validation-set inference time for fair efficiency comparison.
5. Conclusion
HAIS is a concise bottom-up approach that uses hierarchical aggregation and intra-instance prediction for 3D instance segmentation, with effectiveness, generalization, and practical inference speed demonstrated on ScanNet v2 and S3DIS.
- 5. Conclusion: HAIS generates instance predictions through hierarchical aggregation in two steps and refines them with intra-instance prediction.The framework is described as a concise bottom-up approach for 3D instance segmentation.
- 5. Conclusion: Experiments on ScanNet v2 and S3DIS demonstrate HAIS’s effectiveness and generalization.
- 5. Conclusion: HAIS retains better inference speed than existing methods, supporting its practicability in latency-sensitive scenarios.
A. Detailed Inference Time
HAIS maintains an efficient inference pipeline by avoiding time-consuming clustering and post-processing procedures used by other methods.
- B. Detailed Inference Time: 172 ms, 125 ms, 4 ms, and 109 ms are required for point-wise prediction, point aggregation, set aggregation, and intra-instance prediction, respectively.These components together account for the reported HAIS inference pipeline timing.
- B. Detailed Inference Time: HAIS keeps a more efficient pipeline than methods requiring time-consuming clustering and post-processing procedures.
B. Additional Qualitative Results
Additional ScanNet v2 validation results show that hierarchical aggregation and intra-instance prediction produce fine-grained instance predictions despite inaccurate center-shift vectors and fragmented instances.
- B. Additional Qualitative Results: Inaccurate predicted center-shift vectors can produce a large number of instance fragments in qualitative results.
- B. Additional Qualitative Results: Hierarchical aggregation and intra-instance prediction generate fine-grained instance predictions.
- B. Additional Qualitative Results: Figure 7 compares input point clouds, semantic labels, shifted coordinates, ground-truth instance masks, and predicted instance masks.