PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenMask3D: Open-Vocabulary 3D Instance Segmentation

Ayça Takmaz, Elisabetta Fedele, Robert W. Sumner, Marc Pollefeys, Federico Tombari, Francis Engelmann

arXiv:2306.13631v2cs.CV

TL;DR

Existing 3D instance segmentation methods are limited to closed vocabularies and cannot separate multiple instances for novel text queries. OpenMask3D addresses this with zero-shot class-agnostic masks and multi-view CLIP feature aggregation, outperforming other open-vocabulary approaches on the reported metrics and recognizing queried object properties.

  • Problem

    Closed-set 3D instance segmentation recognizes only categories predefined during training, while existing open-vocabulary point features cannot separate multiple object instances.

  • Method

    OpenMask3D uses predicted class-agnostic 3D instance masks to select visible frames and aggregate multi-view, multi-scale CLIP image features into per-mask representations.

  • Results

    OpenMask3D performs better than other open-vocabulary approaches on 6 out of 6 metrics and qualitatively segments queried objects and properties such as colors, textures, context, and affordances.

  • Takeaways & Limitations

    The method enables zero-shot identification of multiple 3D object instances using arbitrary text queries, including queries about semantics, geometry, affordances, materials, and situational context.

  • Takeaways & Limitations

    Performance remains constrained by 3D mask proposal quality, camera-frustum visibility, and evaluation methods that do not fully reveal open-vocabulary capability.

Abstract

from arXiv · show

We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.

1 Introduction

OpenMask3D introduces zero-shot open-vocabulary 3D instance segmentation to overcome the closed-set limits of existing methods. It combines class-agnostic instance masks with CLIP-based mask features for querying objects beyond predefined training categories.

  • Closed-set 3D instance segmentation predicts object masks and categories, but its categories remain tied to training datasets.
  • Open-vocabulary methods address novel-object recognition by supporting categories absent from the training set.This capability is relevant to robotics, augmented reality, scene understanding, and 3D visual search.
  • Existing open-vocabulary 3D scene-understanding methods learn queryable per-point features but cannot separate multiple object instances.
  • OpenMask3D uses a two-stage pipeline with class-agnostic mask proposals and multi-view aggregation of CLIP features for each mask.
  • The paper introduces open-vocabulary 3D instance segmentation and evaluates design choices for zero-shot recognition of text-queryed object instances.

2 Related work

Related work progresses from closed-vocabulary 3D segmentation and multimodal foundation models toward open-vocabulary 2D and 3D scene understanding. These approaches provide language-queryable features, but the cited 3D methods generally represent scenes at the point level.

  • 3D semantic segmentation labels scene points, whereas 3D instance segmentation additionally distinguishes multiple objects sharing a category.
  • Closed-vocabulary approaches face limitations because natural-language descriptions can contain numerous nouns beyond their predefined object categories.
  • Multimodal foundation models use large-scale image-text pretraining to learn representations that support zero-shot transfer to downstream recognition tasks.
  • Open-vocabulary 2D segmentation extends image-level embeddings to pixel-level embeddings, adding localization but relying strongly on accurate 2D masks.
  • Open-vocabulary 3D scene-understanding methods transfer image features from posed views into task-agnostic point-wise scene representations.

3 Method

OpenMask3D predicts class-agnostic 3D instance masks and computes queryable per-mask CLIP representations from selected RGB-D views. Its mask-feature module uses visibility-based frame selection, SAM-derived multi-scale crops, and multi-view feature aggregation.

  • Overview: The method takes posed RGB-D images, a reconstructed point cloud, and known camera parameters as input.
  • Class agnostic mask proposals: OpenMask3D first predicts binary, class-agnostic instance masks over point-cloud points using a mask proposal head.The proposal component leverages a frozen transformer-based mask module from a pre-trained 3D instance segmentation model.
  • Overview: Unlike point-feature-oriented 3D methods, OpenMask3D organizes representations around instance masks to identify separate objects.
  • Frame selection: For each mask, the method selects the top-k views with maximal visibility before extracting image features.Visibility is computed from projected mask points while accounting for camera field of view and occlusion using depth measurements.
  • 2D mask computation and multi-scale crops: SAM produces a selected 2D object mask from sampled projected points, which is expanded into L = 3 multi-level crops for contextual CLIP features.The highest-confidence SAM mask is retained after repeated sampled-point trials.
  • Mask-feature computation: CLIP encodes the collected k ⋅ L crops, whose features are average-pooled across crops and views into one representation per instance mask.

4 Experiments

OpenMask3D is evaluated on closed-vocabulary 3D instance segmentation, generalization, ablations, and qualitative open-vocabulary queries. It outperforms other open-vocabulary approaches, especially on tail categories, while mask quality and scene-context limitations remain important boundaries.

  • Experimental setting: Experiments use ScanNet200 and Replica, with ScanNet200 evaluated across head, common, and tail category subsets.ScanNet200 validation uses 312 scenes and 200 annotated categories.
  • Experimental setting: Average precision is measured at 50% and 25% mask-overlap thresholds and across the [0.5:0.95:0.05] overlap range.Each predicted mask receives a confidence score of 1.0 in these experiments.
  • Experimental setting: OpenMask3D is compared with fully supervised Mask3D and adapted OpenScene variants using per-instance aggregation of OpenScene’s per-point features.OpenScene does not directly provide instance masks, so its features are averaged within masks generated by OpenMask3D’s mask module.
  • Results and analysis: OpenMask3D performs better than open-vocabulary approaches on all six reported metrics, with especially strong gains on tail categories.Mask3D remains stronger overall, particularly on head and common categories, while the gap is less apparent for tail categories.
  • Results and analysis: On Replica, OpenMask3D outperforms other open-vocabulary counterparts when using mask proposals trained on ScanNet200.This evaluates generalization to scenes from another dataset.
  • Results and analysis: Multi-scale cropping and SAM-based 2D mask segmentation both positively affect performance in the ablation study.Additional experiments analyze the number of views used in top-k view selection.
  • Results and analysis: Oracle-mask experiments show that 3D mask quality is important, while qualitative results demonstrate zero-shot segmentation of novel objects and properties such as colors, textures, context, and affordances.The instance-based representation also produces crisp instance boundaries for query-driven object identification.
  • Limitations: Per-mask image features lack global understanding beyond the camera frustum, and closed-vocabulary evaluation does not fully reveal open-vocabulary capability.The authors identify both issues as limitations and note that improved mask proposals remain future work.

5 Conclusion

OpenMask3D identifies 3D object instances from arbitrary text queries by combining class-agnostic mask proposals with per-mask visual features. Its pipeline selects visible views, refines projected masks with SAM, and aggregates multi-scale CLIP features.

  • OpenMask3D identifies object instances in 3D scenes using arbitrary text queries, beyond closed-vocabulary category prediction.
  • The method begins with class-agnostic instance mask proposals that are not associated with class labels.
  • For each proposal, OpenMask3D selects views with high instance visibility before extracting visual features.
  • The selected 2D masks produce multi-scale image crops whose CLIP features are aggregated into per-mask representations for querying.
  • SAM converts projected 3D mask points into class-agnostic 2D masks to improve cropping robustness against noisy or outlier projections.

A.2.4 Why do we need to run SAM for multiple rounds?

A small random point set can produce unreliable SAM masks when points are outliers or spatially concentrated. Multiple confidence-scored sampling rounds make it possible to select a better mask.

  • The sampling procedure runs 10 rounds, drawing 5 projected-mask points per round and retaining the mask with the highest SAM confidence score.
  • Five randomly sampled points can be unreliable when concentrated in a small region, causing SAM to omit parts such as chair legs.
  • The highest-confidence mask in the illustrated example accurately segments the chair.

B.1 Ablation Studies

Additional ablations examine how view selection and multi-scale cropping settings affect OpenMask3D. The studies vary top-k views on ScanNet200 and cropping levels and expansion ratios on Replica.

  • Additional results analyze varying numbers of selected views on ScanNet200 and multi-scale cropping hyperparameters on Replica.
  • The top-k frame-selection ablation is conducted on the ScanNet200 validation set.
  • The multi-scale cropping ablation on Replica varies the number of levels and the crop expansion ratio.

B.2 Evaluation on Replica without RGB-D images

OpenMask3D requires images because it relies on image-based visual-language models combined with text. When only a 3D scan is available, rendering images remains a possible route.

  • OpenMask3D requires images as input because its visual-language models operate on images together with text.
  • The approach prioritizes recognizing uncommon or long-tail objects over generalization across modalities.
  • When only a 3D scan is available, rendering images could still make image-based processing possible.

C Details on baseline experiments

The baseline experiments adapt OpenScene’s point-based features into per-mask representations for comparison with OpenMask3D, while also evaluating RGB-D image sources on Replica.

  • Replica evaluation: Table 8 compares OpenMask3D on Replica using original RGB-D images versus RGB-D images rendered from the point cloud.
  • Baseline adaptation: OpenScene features are aggregated within OpenMask3D’s class-agnostic instance masks to associate each mask with a per-mask feature.This adaptation enables comparison between OpenScene’s point-based representation and OpenMask3D’s mask-based representation.
  • Baseline adaptation: The comparison primarily uses OpenScene models with OpenSeg features because they share OpenMask3D’s 768-dimensional ViT-L/14 CLIP architecture.The experiments also include the OpenScene 2D fusion model with LSeg features.
  • Baseline adaptation: OpenScene provides three model variants: 2D fusion, 3D distill, and 2D/3D ensemble.

Qualitative baseline experiments

The qualitative baseline experiments compare point-based OpenScene features with OpenMask3D’s per-mask features, showing that mask-level similarities produce clearer instance boundaries.

  • Qualitative comparison: The experiments qualitatively compare OpenMask3D’s mask-based open-world representation with OpenScene’s point-based representation.
  • Similarity representation: For OpenScene, query similarity is computed between the text embedding and each point feature, whereas OpenMask3D compares the query with each per-mask feature.
  • Qualitative comparison: OpenScene heatmaps provide reasonable similarity patterns but make object-instance information difficult to extract.
  • Qualitative comparison: OpenMask3D’s per-mask similarities produce crisp instance boundaries suited to applications requiring object-instance information.

D Qualitative results

Additional qualitative results show OpenMask3D segmenting novel object categories and responding to free-form queries about properties beyond object semantics.

  • Novel categories: OpenMask3D successfully segments object categories absent from the ScanNet200 label set.The results are presented as evidence of zero-shot learning for queried objects not represented in common segmentation datasets.
  • Object properties: The method can segment queried object properties involving affordances, color, geometry, material type, and object state.
  • Object properties: These qualitative results indicate that OpenMask3D preserves information about object properties and goes beyond object semantics.
  • Novel categories: Figure 15 visualizes the instance with the highest similarity score for each open-vocabulary query.
Loading 2306.13631v2…