Source-linked AI summary
MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation
Chen Liu, Yasutaka Furukawa
TL;DR
3D instance segmentation must group points into objects without a fixed label set while handling large indoor point clouds. MASC combines sparse-convolution semantic and multi-scale affinity prediction with mesh-based clustering, and it outperforms published state-of-the-art methods by a large margin on ScanNet. Its sequential clustering, unclear scale contributions, and occasional co-planar failures remain limitations.
Problem
3D instance segmentation requires accurate instance-based point grouping without a fixed label set, while large indoor scenes challenge methods that cannot process the entire point cloud.
Method
MASC uses submanifold sparse convolution to predict semantic scores and multi-scale neighboring-voxel affinities, then clusters mesh nodes using those affinities and mesh topology.
Results
MASC outperforms published state-of-the-art instance segmentation methods by a large margin on the ScanNet benchmark.
Takeaways & Limitations
Learned local affinities combined with sparse-convolution processing provide a clustering-based approach for 3D instance segmentation on entire indoor scenes.
Takeaways & Limitations
The sequential clustering is slow, the role of individual affinity scales is unclear, and the method sometimes fails to distinguish co-planar objects.
Abstract
from arXiv · showhide
We propose a new approach for 3D instance segmentation based on sparse convolution and point affinity prediction, which indicates the likelihood of two points belonging to the same instance. The proposed network, built upon submanifold sparse convolution [3], processes a voxelized point cloud and predicts semantic scores for each occupied voxel as well as the affinity between neighboring voxels at different scales. A simple yet effective clustering algorithm segments points into instances based on the predicted affinity and the mesh topology. The semantic for each instance is determined by the semantic prediction. Experiments show that our method outperforms the state-of-the-art instance segmentation methods by a large margin on the widely used ScanNet benchmark [2]. We share our code publicly at https://github.com/art-programmer/MASC.
1. Introduction
3D instance segmentation is challenging because it must assign points to objects without a fixed label set while reasoning over large, irregular point clouds. MASC addresses this by processing entire indoor scenes with sparse convolution and clustering points using learned multi-scale affinities.
- MASC targets 3D instance segmentation, which requires accurate instance-based point grouping without a fixed label set.
- Large indoor point clouds challenge methods that process only limited point subsets, preventing global scene reasoning.Prior methods may process only 4,096 points at a time, whereas indoor scenes contain many more points.
- The method adopts submanifold sparse convolution to process an entire indoor point cloud and predicts semantic scores plus affinities between neighboring voxels at multiple scales.
- A clustering algorithm groups points into instances using learned local similarity and the input mesh topology.
2. Methods
MASC voxelizes an entire ScanNet scene and uses a sparse U-Net to predict point semantics and multi-scale neighboring-voxel affinities. It then iteratively merges mesh nodes according to affinity, stopping when clustering stabilizes.
- Network: The sparse U-Net voxelizes the entire scene and predicts point-wise semantics together with affinity scores between active voxels and their six neighbors at multiple scales.The finest scale uses 4,096 × 4,096 × 4,096 voxels, with 2 cm voxelization.
- Clustering: The input mesh is treated as the initial graph, while voxel affinities are averaged to obtain affinities between mesh nodes.
- Clustering: Each node maps to its most similar neighbor when similarity exceeds 0.5, otherwise it maps to itself.
- Clustering: Mapped nodes are merged into clusters, graph edges are updated, and the process repeats until no assignment changes.
- Output: The clustering algorithm merges nodes in parallel, and each final instance receives the semantic label with the maximum point votes.
3. Results
MASC is evaluated on ScanNet using affinity supervision and sparse-mesh augmentation, with additional handling for planar object classes. The authors report a large-margin improvement over published state-of-the-art methods and provide qualitative validation examples.
- 3.2. Quantitative evaluation: Affinity supervision labels neighboring voxels as similar when both contain enough points and their instance distributions match.
- 3.2. Quantitative evaluation: The implementation augments sparse input meshes by sampling five points inside each triangle to improve local-affinity prediction.
- 3.2. Quantitative evaluation: For planar classes such as pictures, curtains, sinks, and bathtubs, connected components from predicted semantics provide additional instances because these objects can be difficult to segment by clustering.
- 3.3. Qualitative evaluation: Qualitative ScanNet results compare each input mesh with the corresponding predicted instance segmentation.
4. Discussion
The method achieves promising results but retains practical and methodological limitations. Its sequential clustering is slow, the contribution of individual affinity scales is unclear, and co-planar objects can remain difficult to distinguish.
- The sequential clustering implementation is slow, although a GPU-parallel and potentially back-propagatable version is possible.
- The role of individual affinity scales remains under-explored despite two scales giving good performance and fast training in practice.
- The current method sometimes fails to distinguish co-planar objects.