Source-linked AI summary
Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance
Phuc D. A. Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, Khoi Nguyen
TL;DR
Open3DIS addresses open-vocabulary 3D instance segmentation beyond predefined object classes, a capability relevant to identifying and localizing arbitrary objects in 3D environments. It combines class-agnostic 3D proposals with geometrically coherent proposals derived from aggregated multi-frame 2D masks, achieving state-of-the-art results across three datasets, including an approximately 1.5-times improvement over the leading method on ScanNet200.
Problem
Open-vocabulary 3D instance segmentation seeks binary masks for object classes absent from training, overcoming closed-set recognition constraints in conventional 3D segmentation.
Method
Open3DIS combines 3D instance proposals with a 2D-Guided 3D Instance Proposal Module that aggregates multi-frame 2D masks into geometrically coherent 3D proposals.
Results
Open3DIS achieves state-of-the-art OV-3DIS results on ScanNet200, S3DIS, and Replica, with an approximately 1.5-times improvement over the leading method on ScanNet200.
Takeaways & Limitations
Open3DIS extends 3D scene understanding beyond predefined concept sets and produces proposals spanning a wide range of real-world objects.
Takeaways & Limitations
The Class-agnostic 3D Proposal and 2D-Guided-3D Instance Proposal Module operate independently, with synergistic integration left as future work.
Abstract
from arXiv · showhide
We introduce Open3DIS, a novel solution designed to tackle the problem of Open-Vocabulary Instance Segmentation within 3D scenes. Objects within 3D environments exhibit diverse shapes, scales, and colors, making precise instance-level identification a challenging task. Recent advancements in Open-Vocabulary scene understanding have made significant strides in this area by employing class-agnostic 3D instance proposal networks for object localization and learning queryable features for each 3D mask. While these methods produce high-quality instance proposals, they struggle with identifying small-scale and geometrically ambiguous objects. The key idea of our method is a new module that aggregates 2D instance masks across frames and maps them to geometrically coherent point cloud regions as high-quality object proposals addressing the above limitations. These are then combined with 3D class-agnostic instance proposals to include a wide range of objects in the real world. To validate our approach, we conducted experiments on three prominent datasets, including ScanNet200, S3DIS, and Replica, demonstrating significant performance gains in segmenting objects with diverse categories over the state-of-the-art approaches.
1. Introduction
Open3DIS addresses open-vocabulary 3D point-cloud instance segmentation by combining complementary 3D and 2D-guided proposals. It reports state-of-the-art performance across ScanNet200, S3DIS, and Replica.
- OV-3DIS segments 3D point clouds into binary instance masks for classes that may be absent during training.
- Closed-set 3D instance segmentation restricts recognition to predefined training classes, limiting open-vocabulary recognition in applications such as robotics and VR.
- Existing 3D proposal methods struggle with rare objects, while 2D proposals capture only visible object portions and may include background regions.
- Open3DIS combines class-agnostic 3D proposals with geometrically coherent regions extracted under guidance from 2D masks across multiple frames.
- The method aggregates multi-scale CLIP features across views to construct instance-aware point-cloud features for open-vocabulary segmentation.
- ∼1.5 times improvement over the leading method is reported on ScanNet200, alongside state-of-the-art results on ScanNet200, S3DIS, and Replica.
2. Related Work
Related work spans open-vocabulary 2D and 3D recognition, fully supervised 3D instance segmentation, and several OV-3DIS strategies. Open3DIS combines 3D-network masks with 2D-guided geometrically coherent proposals.
- Open-vocabulary 2D recognition covers object detection, semantic segmentation, and instance segmentation, often using visual-text models such as CLIP.
- Fully supervised 3D instance segmentation methods include box-based, cluster-based, and dynamic-convolution-based techniques for training classes.
- Open-vocabulary 3D semantic segmentation and detection provide semantic understanding but do not precisely distinguish objects with 3D instance masks.
- OV-3DIS approaches include semantic-segmentation-based, text-description and proposal-contrastive-learning-based, and 2D-powered methods.
- Open3DIS combines 3D masks from a 3DIS network with proposals formed by grouping geometrically coherent superpoints guided by 2D instance masks.
3. Method
Open3DIS combines class-agnostic 3D proposals with a 2D-guided module that aggregates multi-view masks into geometrically coherent 3D regions. It also extracts pointwise features for open-vocabulary instance classification.
- Overall architecture: The method combines proposals from a 3D instance network with 2D-guided 3D proposals to produce binary object-instance masks.The 3D network supplies class-agnostic masks, while the 2D-guided module addresses incomplete or ambiguous 3D observations.
- Superpoints: Superpoints group points into geometrically homogeneous regions, improving processing efficiency and supporting well-formed candidate instances.They are used as the basic units for merging regions derived from projected 2D masks.
- Per-frame proposal construction: For each 2D mask, the method initializes a region from the best-overlapping superpoint and expands it using neighboring superpoints meeting overlap and cosine-similarity thresholds.The expansion continues until no eligible overlapping or neighboring superpoints remain.
- Cross-frame aggregation: Regions from different frames are matched using feature similarity and IoU, then merged hierarchically with agglomerative clustering to form more complete 3D proposals.The hierarchical strategy merges region sets across non-consecutive frames in successive passes.
- 3D instance network: The 3D instance network predicts binary masks directly from point clouds, while omitting semantic-labeling heads for the open-vocabulary setting.The network uses established 3D instance-segmentation backbones and outputs K2 binary masks.
- Pointwise feature extraction: Proposal features are extracted from projected views with multiscale crops, averaged across views, and accumulated into point-cloud features for open-vocabulary classification.The resulting features support instance-based tasks such as comparison with text prompts in CLIP space.
4. Experiments
Open3DIS is evaluated on ScanNet200, Replica, and S3DIS using standard AP metrics and ablations of its proposal, feature, merging, viewpoint, and segmenter choices. It achieves strong open-vocabulary segmentation results across datasets and supports text-driven queries for unseen categories and properties.
- Experimental setup: Experiments cover ScanNet200, Replica, and S3DIS, with AP evaluated at IoU thresholds of 50% and 25%, plus mAP from 50% to 95%.ScanNet200 also reports APhead, APcom, and APtail.
- Comparison to prior work: Open3DIS surpasses prior methods on ScanNet200 by +5.2 AP over OVIR-3D and +2.8 AP over OpenMask3D.The method also exceeds fully supervised approaches on APtail in the reported ScanNet200 comparison.
- Comparison to prior work: 23.7 AP, 29.4 AP50, and 32.8 AP25 are achieved on ScanNet200 after combining 2D-guided and class-agnostic 3D proposals.This represents a 1.5x AP enhancement compared to prior methods.
- Comparison to prior work: On Replica, Open3DIS exceeds OpenMask3D and OVIR-3D by +5.0 and +7.0 AP, respectively, while using only 2D proposals for the zero-shot setting.The class-agnostic 3D proposal network was omitted because of semantic overlap between Replica and ScanNet200 categories.
- Comparison to prior work: On S3DIS, Open3DIS significantly outperforms existing methods in AP50 for novel categories, achieving more than double their scores.The model is trained on base classes and tested on both base and novel classes.
- Qualitative results: Qualitative results show that arbitrary text prompts can retrieve and segment instances from categories absent from the labels, including prompts describing functionality and other properties.Each scene displays the instance with the highest similarity to the query embedding.
- Ablation study: Pointwise Feature Extraction achieves the best ablation result at 23.7 AP, compared with 22.2 AP for top-λ-view mask features and 17.5 AP for direct feature fusion.The 2D-Guided-3D module reaches 18.2 AP; removing cosine-similarity filtering lowers AP by 2.3, while Euclidean filtering reaches 16.0 AP.
- Ablation study: Agglomerative clustering with hierarchical merging performs best; Hungarian matching drops AP by ∼2.0 and sequential merging by ∼1.0.The viewpoint ablation finds λ=5 best, while one view gives 21.2 AP and 10 or more views perform worse.
5. Discussion
Open3DIS aggregates proposals from point cloud-based instance segmenters and 2D image-based networks in a geometrically coherent manner. Its current proposal modules operate independently, leaving synergistic integration as future work.
- Open3DIS aggregates proposals from point cloud-based instance segmenters and 2D image-based networks in a geometrically coherent manner.
- The Class-agnostic 3D Proposal and 2D-Guided-3D Instance Proposal Module operate independently, and their outputs are combined into the final 3D proposal set.
- A better-integrating strategy in which the modules enhance each other synergistically remains an identified future direction.
6. Implementation Details
Open3DIS combines class-agnostic 3D proposals with 2D open-vocabulary segmentation and projects 2D masks into coherent 3D regions. Its implementation uses camera projection, superpoint-based processing, and hierarchical merging across frames.
- 6.1. Class-agnostic 3D Segmenter: The class-agnostic 3D proposal network adopts ISBNet to produce binary instance masks from colored point clouds.Its 3D UNet backbone uses sparse convolutions, instance-specific kernels, bounding boxes, and box-aware dynamic convolution.
- 6.2. Open-Vocabulary 2D Segmenter: Four 2D open-vocabulary instance segmenters—Grounded-SAM2, DETIC, SEEM, and ODISE—generate image-level instance masks.Grounded-SAM2 uses Grounding DINO and SAM; the other systems use their specified pretrained visual and mask-generation components.
- 6.6. 3D Object Proposal Formation: The proposal-formation algorithm recursively merges per-frame point-cloud regions using a cost matrix and agglomerative clustering.Hierarchical traversal combines regions from multiple frames into augmented 3D proposals.
- 6.5. Point cloud - Image Projection: Known camera intrinsics and extrinsics project 3D points into each RGB-D frame, after which out-of-bounds and occluded points are filtered.The projection uses Πt = Kt · [R|c]t, with depth consistency checked against the recorded depth image.
7. Additional Analysis
Additional analyses examine depth filtering, RGB-D frame subsampling, and class-agnostic proposal quality. The results identify a best depth threshold, show degradation when frames are subsampled too aggressively, and evaluate proposals on ScanNet200 and ScanNet++.
- Depth threshold: τdepth = 0.1 gives the best performance in the depth-threshold ablation.The study reports this setting as the overall best among the evaluated thresholds.
- RGB-D image subsampling: Subsampling RGB-D images by factors of 20 or 40 slightly reduces performance to 17.1 AP, while reducing the number of images too much yields worse results.The default subsampling factor is 10, and total validation-set inference runtime is also reported.
- Class-agnostic evaluation: Open3DIS with ISBNet produces superior class-agnostic 3D proposals on ScanNet200 compared with existing methods.The ScanNet++ evaluation uses 100 subsampled RGB-D frames per scene for computational efficiency.
- Evaluation protocol: The class-agnostic evaluation uses all 2D-G-3DIP masks for high recall and the top 100 ISBNet masks selected by confidence for 3D proposals.Open-vocabulary evaluation then postprocesses masks by selecting approximately 300–600 proposals.
8. Qualitative Results
Qualitative results show that Open3DIS forms cleaner, finer, and more complete 3D instance regions, especially for small, rare, occluded, or geometrically ambiguous objects. Results are presented across ScanNet200, Replica, S3DIS, ARKitScenes, and multiple comparison figures.
- 2D-G-3DIP: 2D masks are mapped to geometrically coherent 3D regions by assigning masks to superpoints and merging them with averaged 3D features.This addresses overlapping 2D masks and imperfect point-to-pixel correspondence that impair point-based lifting.
- 2D-G-3DIP: Superpoint-wise masks provide finer-grained and more precise instance representations under occlusions and depth imperfections.The method weights groups of points and combines 3D deep features with geometric IoU calculations.
- ScanNet200: ScanNet200 visualizations show comprehensive masks covering significantly small objects, although some such objects are absent from the dataset annotations.The method combines 2D and 3D segmenters, while omitted ground-truth objects appear as black points.
- ScanNet200: Open3DIS produces finer masks for complex and ambiguous geometries than the compared methods in ScanNet200 visualizations.The qualitative comparison contrasts its masks with noisy or suboptimal point-based lifting and rare-class behavior from other approaches.
- ScanNet200: 19.0 AP versus 12.6 AP for OpenMask3D on ScanNet200 demonstrates Open3DIS’s stronger performance in the reported comparison.The comparison uses OpenMask3D trained on ScanNet20 and reports Average Precision on ScanNet200.
- Replica and S3DIS: Qualitative results include predictions on Replica and S3DIS, with each row showing RGB point clouds, instance ground truth, and Open3DIS predictions.The corresponding results are presented in Figure 11.
- Open-vocabulary scene exploration: Open3DIS demonstrates text-queryable scene exploration on ARKitScenes and ScanNet200 using attributes such as material, color, affordances, and usage.The exploration visualization presents text queries, original point clouds, and colored 3D instance proposals.
- Proposal construction: The 2D-G-3DIP module generates per-frame 3D proposals from 2D masks and superpoints, then hierarchically merges them across views.The resulting augmented proposals are illustrated in the module and qualitative-construction figures.