PaperlayerView exact revision on arXiv

Source-linked AI summary

SAI3D: Segment Any Instance in 3D Scenes

Yingda Yin, Yuzheng Liu, Yang Xiao, Daniel Cohen-Or, Jingwei Huang, Baoquan Chen

arXiv:2312.11557v2cs.CV

TL;DR

3D instance segmentation has been limited by reliance on annotated datasets, while open-set methods can miss distinctions between objects of the same category. SAI3D combines geometric primitives with multi-view SAM masks to generate zero-shot 3D instances. It outperforms prior unsupervised methods and surpasses fully supervised Mask3D on ScanNet++.

  • Problem

    Supervised 3D instance segmentation depends on precise annotations and is limited to a narrow range of categories, while open-set methods may not separate same-category objects or work without specific prompts.

    Source: §1 · p. 1

  • Method

    SAI3D builds affinities between geometric primitives from SAM masks and progressively merges primitives using multi-view evidence to produce 3D instances without 3D annotation training.

    Source: §1 · p. 2

  • Results

    SAI3D significantly outperforms prior unsupervised methods in class-agnostic segmentation and surpasses fully supervised Mask3D models on ScanNet++.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results support using geometric primitives and multi-view consistency for high-quality instance segmentation in complex 3D scenes without 3D labeled data.

    Source: §5 · p. 8

    Fine-grained results depend on accurate 2D masks and reliable 2D-3D alignment; errors in either can make affinities unreliable.

    Source: §5 · p. 8

Abstract

from arXiv · show

Advancements in 3D instance segmentation have traditionally been tethered to the availability of annotated datasets, limiting their application to a narrow spectrum of object categories. Recent efforts have sought to harness vision-language models like CLIP for open-set semantic reasoning, yet these methods struggle to distinguish between objects of the same categories and rely on specific prompts that are not universally applicable. In this paper, we introduce SAI3D, a novel zero-shot 3D instance segmentation approach that synergistically leverages geometric priors and semantic cues derived from Segment Anything Model (SAM). Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations that are consistent with the multi-view SAM masks. Moreover, we design a hierarchical region-growing algorithm with a dynamic thresholding mechanism, which largely improves the robustness of finegrained 3D scene parsing.Empirical evaluations on ScanNet, Matterport3D and the more challenging ScanNet++ datasets demonstrate the superiority of our approach. Notably, SAI3D outperforms existing open-vocabulary baselines and even surpasses fully-supervised methods in class-agnostic segmentation on ScanNet++. Our project page is at https://yd-yin.github.io/SAI3D.

1. Introduction

Supervised 3D instance segmentation depends on precise annotations and remains tied to a narrow range of categories, while open-set approaches can miss distinctions between same-category objects. SAI3D combines geometric priors with SAM masks for zero-shot 3D instance segmentation and reports strong class-agnostic results, including on ScanNet++.

1. Introduction

  • Supervised methods require precise 3D annotations and are confined to a narrow range of categories in datasets such as ScanNet or KITTI.The authors connect this restriction to reduced applicability in open-world settings such as embodied agents and autonomous driving.Source: §1 · p. 1
  • Open-set methods can produce heatmaps without distinguishing same-category objects and may depend on prompts unavailable for objects in a scene.Earlier SAM-based approaches use manual prompts or iterative local-frame merging, which may overlook global consensus.Source: §1 · pp. 1–2
  • SAI3D uses SAM-generated 2D masks to build pairwise affinities among geometric primitives, then progressively merges them with votes aggregated across valid views.The resulting 3D instance segmentation is designed to be consistent across views and requires no training or finetuning with 3D annotations.Source: §1 · p. 2
  • SAI3D significantly outperforms SAM3D and other open-vocabulary baselines in class-agnostic segmentation, and surpasses fully supervised Mask3D on ScanNet++.The paper evaluates on ScanNetV2, ScanNet200, Matterport3D, and ScanNet++; it describes ScanNet++ as a more realistic and challenging benchmark.Source: §1 · p. 2
  • The paper presents SAI3D as combining geometric priors with semantic-aware image segmentation and aggregating 2D masks into coherent, cross-view-consistent 3D segments.The authors state that the generated masks are more accurate than previous approaches and identify them as an opportunity for unsupervised 3D learning.Source: §1 · p. 2

2. Related Work

Prior 3D instance segmentation methods rely on annotated data, while open-vocabulary methods often cannot separate objects sharing a category. Related work explores language-aligned 3D semantics and 2D foundation masks, but SAI3D combines geometric priors with multi-view SAM masks for fine-grained instances.

Closed-vocabulary 3D segmentation

  • Closed-vocabulary instance segmentation identifies distinct objects within a semantic category, but methods such as Mask3D require extensive 3D annotations and may not generalize to unseen categories.Collecting 3D annotations is costly and can be impossible for privacy reasons.Source: §2 · p. 2

Open-vocabulary 2D image segmentation

  • SAM [20] supports zero-shot 2D segmentation with user-provided or automatic prompts, and SAI3D aggregates its multi-view masks into 3D segments using primitive-based region growing.Other open-vocabulary image segmentation approaches described here learn semantic-aware pixel-wise embeddings from image-language models.Source: §2 · p. 2

Open-vocabulary 3D segmentation

  • CLIP-based 3D approaches enable text-prompt querying but cannot distinguish instances of the same category; OpenMask3D and OpenIns3D [16] use Mask3D for class-agnostic proposals.The authors report that these proposal methods fail on complex scenes with fine-grained objects, unlike their performance on indoor scenes similar to ScanNet.Source: §2 · pp. 2–3
  • SAI3D’s related-work distinction is its integration of 3D geometric priors with semantic cues from multi-view SAM masks for fine-grained 3D instance segmentation.The paper contrasts this combination with concurrent approaches that aggregate 2D foundation-model masks into 3D representations.Source: §2 · p. 3

3. Method

SAI3D builds a scene graph from geometry-based 3D primitives and SAM masks, then progressively merges primitives into 3D instances. Its merging uses multi-level affinity criteria and thresholds that change as regions grow.

3.1. Scene Graph Construction

  • Normal-based graph cuts over-segment the point cloud into geometric superpoints, which serve as scene-graph nodes connected by affinity-weighted edges.Primitive-level affinities are described as more reliable than point-level affinities, improving robustness.Source: §3.1 · p. 3

3.2. Primitive Affinity

  • Each primitive’s projected 2D mask-label histogram is compared with another’s by cosine similarity to estimate single-view affinity.Affinity across views is combined using a weighted sum, with weights based on primitive visibility; images where either primitive is invisible are invalid.Source: §3.2 · pp. 3–4
  • Projected primitives are assessed against multi-view SAM masks, while occluded or out-of-frame points are excluded from affinity calculations.Accurate masks can confirm object unity, whereas incorrect masks introduce noise into affinity assessment.Source: Figure 3 · p. 4

3.3. Primitive Merging

  • SAI3D progressively merges neighboring 3D primitives with high affinity scores into final 3D instance masks.The framework uses a multi-level criterion and stages with thresholds that descend from strict to relaxed as regions grow.Source: §3.3 · pp. 4–5
  • Progressive growing varies the affinity threshold from high to low, beginning with strict merges and relaxing the criterion as regions grow.This design addresses the sensitivity of region growing to a fixed threshold and aims to prevent early incorrect merges from accumulating.Source: §3.3 · p. 5

3.4. Open-vocabulary 3D Object Search

  • For open-vocabulary object search, SAI3D back-projects OVSeg masks and assigns target labels to 3D instances with more than 50% overlap.Source: §3.4 · p. 5

4. Experiment

The experiments compare SAI3D with open- and closed-vocabulary baselines across datasets, then examine region-growing design choices and sensitivity to the number of 2D views.

4.1. Experiment Setup

  • The evaluation spans ScanNet++, ScanNetV2, ScanNet200, and Matterport3D, using class-agnostic and semantic instance-segmentation setups.The benchmarks include open-vocabulary methods and supervised closed-vocabulary baselines.Source: §4.1 · p. 5, §4 · p. 5

4.2. Results

  • On ScanNet++, SAI3D outperforms prior works across all evaluation metrics, including a closed-vocabulary baseline trained on ScanNetV2.The comparison is for class-agnostic instance segmentation.Source: §4.2 · pp. 5–6, Table 1 · p. 5
  • Visual comparisons show cleaner segments for small objects in clutter, while prompt-based querying retrieves rare objects such as toilet rolls and shower gel.In the cabinet example, SAI3D segments stored items separately where other methods group them as one instance.Source: §4.2 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7
  • On ScanNetV2, SAI3D significantly outperforms open-vocabulary baselines, while Mask3D loses performance when trained on a different dataset.The authors interpret Mask3D’s drop as indicating overfitting to training data and weaker generalization to open scenes.Source: §4.2 · p. 6, Table 2 · p. 6
  • On ScanNet200, SAI3D exceeds open-vocabulary baselines but trails OpenMask3D, with weaker Head and stronger Tail-label performance.The authors associate the Tail-label result with stronger zero-shot generalization to less common labels.Source: §4.2 · p. 6, Table 3 · p. 6
  • On Matterport3D, SAI3D consistently outperforms the other baselines in both class-agnostic and semantic instance segmentation.OpenMask3D’s performance drops when evaluated there after training on ScanNet200.Source: §4.2 · p. 8, Table 4 · p. 8

4.3. Ablation and Analysis

  • Combining 3D superpoints, multi-level merging, and progressive growing improves robustness; together, the designs yield approximately 50% better performance than the basic algorithm.Progressive thresholds mitigate the under- and over-segmentation associated with fixed thresholds.Source: Figure 5 · p. 6, §4.3 · p. 8, Table 5 · p. 8
  • AP score larger than 20 with only 1% of views exceeds SAM3D’s best performance, and SAI3D improves as more views are provided.SAM3D fails to aggregate useful information when the view proportion exceeds 5%.Source: §4.3 · p. 8, Figure 8 · p. 8

5. Conclusion

SAI3D uses geometric primitives and progressive region growing to produce zero-shot 3D instance masks. It outperforms prior unsupervised methods and surpasses supervised Mask3D on ScanNet++, while relying on accurate 2D masks and 2D–3D alignment and scaling linearly with image count.

5. Conclusion

  • SAI3D progressively merges 3D superpoints into object masks using 2D and 3D segmentation techniques and a region-growing algorithm.The method is presented as an efficient zero-shot alternative to supervised approaches confined to specific datasets.Source: §5 · p. 8
  • SAI3D significantly outperforms prior unsupervised methods on ScanNet, ScanNet++ and Matterport3D, and surpasses supervised Mask3D on ScanNet++.The approach uses no 3D labeled data.Source: §5 · p. 8
  • Incorrect 2D masks or camera poses can produce unreliable affinities and affect fine segmentation results.The method relies on accurate 2D segmentation and reliable 2D–3D alignment.Source: §5 · p. 8
  • Processing time scales linearly with the number of images, making the method difficult to apply in large-scale scenes.The authors identify avoiding iteration over all images as a future direction.Source: §5 · p. 8

A. Implementation Details

Implementation computes region-to-superpoint affinity from pairwise affinities, then grows regions through multiple threshold stages. Image sampling, label handling, evaluation sets, and 2D foundation models vary across datasets.

A. Implementation Details

  • Region-to-superpoint affinity is a weighted average of the affinities between a superpoint and the superpoints in the region.The weight factor depends on graph distance and the number of points in the compared superpoint.Source: A. Implementation Details · p. 11
  • The multi-stage region-growing thresholds descend from 0.9 to 0.7 for ScanNet++ and from 0.9 to 0.5 for ScanNet.Contributions from superpoints at graph distance greater than 2 are eliminated, and γ is set to 0.5.Source: A. Implementation Details · p. 11
  • The method samples 5% of ScanNet++ views, 20% of ScanNetV2 and ScanNet200 views, and all Matterport3D images.Matterport3D is described as sparsely scanned.Source: A. Implementation Details · p. 11
  • ScanNetV2 and ScanNet200 omit wall and floor instances, while Matterport3D uses its 160 most frequent classes and omits wall, floor, and ceiling.ScanNet++ uses all provided instance-segmentation ground-truth labels, with iPhone images as input.Source: A. Implementation Details · p. 11
  • Evaluation uses validation sets for ScanNetV2, ScanNet200, and ScanNet++, and the Matterport3D test set; class-agnostic performance is measured by mask AP.The implementation uses Semantic-SAM for ScanNetV2, ScanNet200, and Matterport3D, and SAM-HQ for ScanNet++.Source: A. Implementation Details · p. 11

B. Additional Qualitative Results

The additional qualitative results show 3D instance segmentation examples from ScanNet++ and ScanNetV2. The ScanNet examples include cases where the method produces finer masks than the ground-truth annotations.

B. Additional Qualitative Results

Related papers

Loading 2312.11557v2…