Source-linked AI summary
3D Instance Segmentation via Multi-Task Metric Learning
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald
TL;DR
3D instance segmentation seeks to separate individual objects in dense voxelized scenes, but the problem has been less explored than 2D instance segmentation. The paper uses multi-task learning to combine instance-specific feature embeddings with per-voxel directions toward object centers. Experiments on synthetic and real-world data demonstrate viability, with state-of-the-art performance reported on ScanNet.
Problem
3D instance segmentation is less explored than 2D segmentation, with scarce datasets and methods whose extension to 3D is not straightforward.
Method
The method processes voxelized 3D scenes with metric learning for instance-specific embeddings and directional prediction toward each object’s center.
Results
The authors report state-of-the-art performance and first place in average AP50 on the ScanNet 3D instance-segmentation benchmark at submission.
Takeaways & Limitations
Multi-task learning improves the instance-grouping results by adding directional information that supplies more discriminative features.
Takeaways & Limitations
Feature embeddings are affected by the number and locations of other cluster centers, which can be disadvantageous when many instances occupy one scene.
Abstract
from arXiv · showhide
We propose a novel method for instance label segmentation of dense 3D voxel grids. We target volumetric scene representations, which have been acquired with depth sensors or multi-view stereo methods and which have been processed with semantic 3D reconstruction or scene completion methods. The main task is to learn shape information about individual object instances in order to accurately separate them, including connected and incompletely scanned objects. We solve the 3D instance-labeling problem with a multi-task learning strategy. The first goal is to learn an abstract feature embedding, which groups voxels with the same instance label close to each other while separating clusters with different instance labels from each other. The second goal is to learn instance information by densely estimating directional information of the instance's center of mass for each voxel. This is particularly useful to find instance boundaries in the clustering post-processing step, as well as, for scoring the segmentation quality for the first goal. Both synthetic and real-world experiments demonstrate the viability and merits of our approach. In fact, it achieves state-of-the-art performance on the ScanNet 3D instance segmentation benchmark.
1. Introduction
3D instance segmentation remains less explored than its 2D counterpart, while existing methods often entangle instance and semantic labeling. The paper proposes multi-task learning that combines instance-specific embeddings with directions toward object centers.
- 3D instance segmentation has been less explored than 2D instance segmentation, partly because suitable datasets are scarce and 2D methods do not transfer straightforwardly.
- The method focuses primarily on grouping information belonging to each object instance rather than entangling instance labeling with semantic labeling.
- The network learns a feature descriptor for each instance and a direction pointing toward that instance’s center.
- The proposed architecture combines metric learning with directional prediction and is designed for voxel-based 3D scene representations.
- The authors report that multi-task learning improves both tasks and that the approach ranks first in average AP50 on the ScanNet benchmark at submission.
2. Related Work
Prior 2D instance-segmentation approaches include proposal, detection, and metric-learning strategies, but their extension to 3D is challenging. The paper extends metric learning to dense 3D voxel grids with a multi-task approach.
- 2D Instance Segmentation: Many successful 2D instance-segmentation methods require substantial resources, and extending them to 3D is non-trivial.
- 2D Instance Segmentation via Metric Learning: Prior metric-learning approaches group pixels or learn position-sensitive representations, but the cited approaches had not been applied to a 3D setting.
- 2D Instance Segmentation via Metric Learning: The paper builds on DeBrabandere et al. by extending metric learning with a multi-task approach for 3D instance segmentation on dense voxel grids.
- 3D Instance Segmentation: Existing 3D methods include similarity matrices, shape proposals, RGB-D feature fusion, multi-scale affinity scores, and RGB-to-3D label transfer.
- 3D Instance Segmentation: Research on 3D instance segmentation had generally remained sparse despite recent concurrent work and emerging benchmarks.
3. Method Overview
The method treats 3D instance segmentation as grouping and splitting semantically labeled voxels, using feature embeddings and object-center directions to separate instances. A multi-task voxel-grid network jointly learns these representations and post-processing clusters, scores, and suppresses proposals.
- Method Overview: Instance segmentation is handled after semantic labeling because grouping or separating semantically labeled volumes requires holistic scene understanding.The method relies on inter-instance and intra-instance relations rather than jointly solving scene completion, semantic labeling, and instance segmentation.
- Method Overview: The pipeline takes a voxelized 3D space with semantic labels or learned local features and outputs instance masks with semantic labels.Masks can be reprojected to point clouds by assigning each voxel's label to its contained points.
- Method Overview: The network learns an embedding that clusters voxels from the same instance and a 3D vector for each voxel pointing toward its object's physical center.The directional representation supplies local shape-containment information and helps distinguish similar shapes.
- Multi-task Loss Function: A multi-task loss combines feature discrimination with directional accuracy, while the embedding loss pulls same-instance features together, pushes different-instance centers apart, and regularizes activations.The feature-loss weights are γvar = γdist = 1 and γreg = 0.001; δvar bounds within-cluster distances and 2δdist sets the minimum separation between cluster centers.
- Multi-task Loss Function: The directional loss learns local voxel-to-center relationships that are not affected by other clusters.Its desired direction points from each voxel center zi toward the object center zc, and it penalizes angular deviations from the predicted normalized direction.
- Post-processing: Mean-shift clustering generates embedding-based proposals, which are scored using embedding and directional consistency, split with connected components, and filtered by non-maximum suppression.The final scoring also favors regular object sizes and assigns semantic labels by the most frequent label among clustered voxels.
4. Results and Evaluation
Experiments on synthetic data and ScanNet evaluate the method using AP metrics, baselines, qualitative visualizations, and ablations. The multi-task approach generally improves instance grouping and achieves strong ScanNet benchmark performance, while remaining limited by some geometric boundary errors.
- Synthetic evaluation: On the synthetic toy dataset, multi-task learning and feature embedding outperform connected components despite the latter using ground-truth semantic labels.The difference between feature embedding alone and multi-task learning is small in the noise-free setting.
- Synthetic evaluation: The network separates spatially touching objects, including same-class instances, by placing their feature vectors in distinct regions of feature space.The toy scenes test whether the method can distinguish objects with similar shapes, sizes, and semantic labels.
- Qualitative results: Qualitative ScanNet results show that the method usually groups single objects correctly but sometimes splits desks or bleeds furniture labels into neighboring geometry.The geometric approach requires structural changes to recognize some object boundaries and relabel new instances.
- ScanNet ablation: Multi-task training consistently outperforms single-task learning in the ScanNet ablation, except for classes that rarely have multiple instances per scene.The directional loss is reported to add discriminative features useful for grouping instances in feature space.
- ScanNet benchmark: The method outperforms other submissions in ScanNet AP50 and operates directly on 3D data without requiring 2D information.The paper states that this direct 3D operation enables faster processing and requires substantially less information for extracting instance segments.
5. Conclusion
The paper concludes with a two-part approach for voxel-based 3D instance segmentation: metric learning groups voxels by instance, while center-direction estimates score the resulting segments.
- Conclusion: Metric learning places voxels from the same object instance close together while separating voxels from different instances in feature space.This produces feature vectors that support instance grouping in voxel-based scenes.
- Conclusion: Directional estimates of object centers provide information for scoring segmentation results generated by the feature-grouping stage.The two components therefore combine instance separation with evaluation of candidate segmentations.
- Conclusion: The paper presents the method as a solution for 3D instance segmentation of voxel-based scenes.The conclusion restates the target problem without extending the claimed scope beyond voxel-based scene representations.