Source-linked AI summary
OccuSeg: Occupancy-aware 3D Instance Segmentation
Lei Han, Tian Zheng, Lan Xu, Lu Fang
TL;DR
3D instance segmentation must handle irregular 3D data while exploiting metric reconstruction that avoids the ambiguities of projective 2D observations. OccuSeg adds a voxel-occupancy signal to multi-task embedding learning and clustering, achieving state-of-the-art results on three datasets with high efficiency.
Problem
Existing 3D instance-segmentation approaches pay limited attention to the inherent metric, non-occluded, and scale-unambiguous properties of reconstructed 3D models.
Method
OccuSeg jointly learns semantic, spatial, feature-embedding, and occupancy representations, then uses predicted occupancy size to guide graph-based clustering.
Results
State-of-the-art performance is reported on ScanNetV2, S3DIS, and SceneNN while maintaining high efficiency, including a 12.3 mAP gain on ScanNetV2.
Takeaways & Limitations
Occupancy-aware learning and clustering improve overall 3D instance-segmentation quality, help correctly cluster hard samples, and prohibit over-segmented clusters.
Abstract
from arXiv · showhide
3D instance segmentation, with a variety of applications in robotics and augmented reality, is in large demands these days. Unlike 2D images that are projective observations of the environment, 3D models provide metric reconstruction of the scenes without occlusion or scale ambiguity. In this paper, we define "3D occupancy size", as the number of voxels occupied by each instance. It owns advantages of robustness in prediction, on which basis, OccuSeg, an occupancy-aware 3D instance segmentation scheme is proposed. Our multi-task learning produces both occupancy signal and embedding representations, where the training of spatial and feature embeddings varies with their difference in scale-aware. Our clustering scheme benefits from the reliable comparison between the predicted occupancy size and the clustered occupancy size, which encourages hard samples being correctly clustered and avoids over segmentation. The proposed approach achieves state-of-the-art performance on 3 real-world datasets, i.e. ScanNetV2, S3DIS and SceneNN, while maintaining high efficiency.
1. Introduction
OccuSeg uses 3D occupancy—the number of voxels occupied by each instance—to augment embedding-based 3D instance segmentation. Its learning and clustering stages exploit this signal, achieving strong benchmark performance while retaining efficiency.
- 3D instance segmentation recognizes object-associated points and infers their semantic classes for robotics and augmented/virtual reality applications.
- Directly projecting 2D MaskRCNN predictions into 3D performs poorly because irregular 3D data introduces challenges beyond 2D solutions.
- OccuSeg introduces a 3D occupancy signal that represents the number of voxels occupied by each instance.
- The method jointly learns semantic, spatial, feature-embedding, and occupancy representations, then uses occupancy-aware clustering to merge similar super-voxels.Feature and spatial embeddings receive different objectives, while clustering compares embedding distance with occupancy size using an adaptive threshold.
- 12.3 gain in mAP on ScanNetV2 accompanies state-of-the-art performance across ScanNetV2, S3DIS, and SceneNN while maintaining high efficiency.
2. Related Work
Prior 2D and 3D instance-segmentation methods use proposals, embeddings, offsets, affinities, or geometric features, but few explicitly exploit 3D metric reconstruction. OccuSeg is motivated by occupancy because the same 3D instance can produce variable 2D observations.
- 2D instance-segmentation methods are commonly proposal-based or proposal-free, with proposal-free methods clustering per-pixel embeddings.
- Extending 2D approaches to 3D is non-trivial because point clouds are irregularly distributed in physical space.
- Existing 3D methods include pairwise similarity, bounding-box, shape-proposal, voting, and joint 2D–3D feature approaches.
- Occupancy varies greatly across 2D observations of the same 3D instance but can be predicted robustly from the reconstructed 3D model.
3. Methods
OccuSeg combines semantic, spatial, feature, covariance, and occupancy predictions with occupancy-aware graph clustering for 3D instance segmentation. The method estimates instance occupancy from voxel predictions and uses it to guide merging and reduce over-segmentation.
- 3.1. Methods: OccuSeg voxelizes colored 3D scenes and predicts semantic labels, joint feature-spatial embeddings, covariance, and instance occupancy through multi-task learning.The joint objective combines semantic segmentation, embedding learning, and occupancy regression.
- 3.1.1 Embedding Learning: Feature and spatial embeddings are explicitly separated and supervised with different objectives, then regularized through covariance estimation.This design reflects the scale-aware physical interpretation of spatial embeddings and the ambiguous scale of feature embeddings.
- 3.1.1 Embedding Learning: Spatial embedding predicts each voxel’s 3D instance center, while feature embedding uses discriminative learning to separate instances.The spatial term regresses a three-dimensional center offset; feature learning includes variance, distance, and regularization terms.
- 3.1.2 Occupancy Regression: Occupancy regression predicts the positive logarithm of each instance’s voxel count, averages voxel predictions, and evaluates relative prediction error Rc.On over 4000 ScanNetV2 validation instances, more than 68% have relative error below 0.3.
- 3.2. Instance Clustering: Inference groups voxels into super-voxels, averages their representations, and constructs a graph whose edge weights measure super-voxel similarity.Spatial embedding, feature embedding, occupancy, and covariance are aggregated for each super-voxel.
- 3.2. Instance Clustering: The occupancy ratio modifies graph merging weights: ratios above 1 penalize oversized super-voxels, while smaller ratios encourage partial instances to attract more voxels.The occupancy-aware scheme is reported to punish over-segmented instances and encourage merging partial instances.
4. Experiments
Experiments on ScanNetV2, S3DIS, and SceneNN show strong segmentation accuracy, robust real-world behavior, and high processing efficiency. Ablations further indicate that occupancy awareness improves overall segmentation quality.
- Qualitative Evaluation: FlashFusion-based online experiments show robust 3D instance segmentation in real-world scenarios using a network pretrained on ScanNetV2.The real-world setup uses depth and color sequences from a consumer-level RGB-D camera.
- Quantitative Evaluation: ScanNetV2, S3DIS, and SceneNN evaluations show OccuSeg outperforming previous methods across the reported metrics.On ScanNetV2, the method leads all three metrics; on S3DIS it leads mPrec and mRec; on SceneNN it exceeds prior work with only 50 training scans.
- Efficiency: More than 4× faster processing than 3D-BoNet demonstrates OccuSeg’s efficiency on 312 ScanNetV2 validation scans.The comparison reports both full-pipeline and stage-level processing times.
- Ablation Study: The ablation study finds that removing occupancy prediction and fixing r = 1 during clustering reduces overall segmentation quality.The w/o occupancy variant disables occupancy prediction in learning and sets the occupancy ratio to one during clustering.
5. Discussion and Conclusion
OccuSeg combines feature and spatial embeddings with a 3D occupancy signal, using occupancy-guided graph clustering to merge hard samples and prevent over-segmentation. Experiments show improved effectiveness and efficiency, while future work targets partially reconstructed objects and finer-grained segmentation.
- OccuSeg is an occupancy-aware instance segmentation method for 3D scenes.
- The learning stage combines feature embedding, spatial embedding, and a 3D occupancy signal representing inherent 3D object properties.
- The occupancy signal guides graph-based clustering to merge hard samples correctly and prevent over-segmented clusters.
- Extensive experiments demonstrate effectiveness, significant gains over previous methods, and high efficiency.
- Future work will address partially reconstructed objects, sub-object-level segmentation, and further efficiency improvements.
A. Network Architecture Details
OccuSeg uses a UNet-style sparse-convolutional architecture for feature learning, combining submanifold sparse convolution and sparse convolution layers. The architecture uses a 32-dimensional semantic embedding term in the reported experiment.
- OccuSeg adopts a UNet-style network for feature learning.
- The network is mainly built from submanifold sparse convolution and sparse convolution layers.
- The semantic term of the learned embedding has dimension 32 in the experiment.
B. Additional Qualitative Comparisons
Additional ScanNetV2 qualitative comparisons report that OccuSeg handles small objects better and produces less noise than the compared methods. The comparisons include sparse convolutional networks, SGPN, and Lahoud et al.'s multi-task metric learning method.
- OccuSeg is qualitatively compared with sparse convolutional networks, SGPN, and Lahoud et al.'s multi-task metric learning method.
- OccuSeg generally shows better capacity for dealing with small objects than the compared methods.
- OccuSeg generally produces less noise in the qualitative ScanNetV2 results.
- Figure 9 presents additional qualitative comparisons on ScanNetV2, with previous-method results taken from Lahoud et al.