PaperlayerView exact revision on arXiv

Source-linked AI summary

Details Matter for Indoor Open-vocabulary 3D Instance Segmentation

Sanghun Jung, Jingjing Zheng, Ke Zhang, Nan Qiao, Albert Y. C. Chen, Lu Xia, Chi Liu, Yuyin Sun, Xiao Zeng, Hsiang-Wei Huang, Byron Boots, Min Sun, Cheng-Hao Kuo

arXiv:2507.23134v1cs.CV

TL;DR

OV-3DIS must identify queried objects, including novel classes, in 3D scenes where closed-vocabulary approaches are limited. The paper combines complementary proposal-generation and classification refinements, achieving state-of-the-art AP and AR across ScanNet200 and S3DIS.

  • Problem

    OV-3DIS requires 3D instance masks for open-vocabulary queries, but closed-vocabulary models often miss novel or rare classes.

    Source: §1 · p. 1

  • Method

    The method combines image- and point-cloud-based proposals with tracking, iterative merging/removal, Alpha-CLIP representations, and SMS filtering.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

  • Results

    The method achieves state-of-the-art performance on ScanNet200 and S3DIS across AP and AR metrics, surpassing previous state-of-the-art methods.

    Source: §1 · p. 2, §4.2 · p. 7, §4.4 · p. 8

  • Takeaways & Limitations

    Combining complementary proposal-generation and classification concepts yields a precise and accurate OV-3DIS system across the evaluated benchmarks.

    Source: §1 · p. 1, §5 · p. 8

Abstract

from arXiv · show

Unlike closed-vocabulary 3D instance segmentation that is often trained end-to-end, open-vocabulary 3D instance segmentation (OV-3DIS) often leverages vision-language models (VLMs) to generate 3D instance proposals and classify them. While various concepts have been proposed from existing research, we observe that these individual concepts are not mutually exclusive but complementary. In this paper, we propose a new state-of-the-art solution for OV-3DIS by carefully designing a recipe to combine the concepts together and refining them to address key challenges. Our solution follows the two-stage scheme: 3D proposal generation and instance classification. We employ robust 3D tracking-based proposal aggregation to generate 3D proposals and remove overlapped or partial proposals by iterative merging/removal. For the classification stage, we replace the standard CLIP model with Alpha-CLIP, which incorporates object masks as an alpha channel to reduce background noise and obtain object-centric representation. Additionally, we introduce the standardized maximum similarity (SMS) score to normalize text-to-proposal similarity, effectively filtering out false positives and boosting precision. Our framework achieves state-of-the-art performance on ScanNet200 and S3DIS across all AP and AR metrics, even surpassing an end-to-end closed-vocabulary method.

1. Introduction

OV-3DIS uses a two-stage pipeline that generates class-agnostic 3D proposals and classifies them with open-vocabulary models. This paper combines complementary proposal-generation concepts and refines classification to reduce false positives.

  • The paper combines complementary OV-3DIS concepts in a two-stage pipeline covering 3D proposal generation and instance classification.The approach generates proposals from images and point clouds before classifying them with CLIP-based models.Source: §1 · p. 1
  • Overlap removal in each frame suppresses masks covering multiple objects before 2D predictions are lifted into 3D.Predictions are sorted by size, and overlapped regions are removed from larger masks.Source: §1 · pp. 1–2
  • Frame-wise superpoint IoU tracking aggregates lifted predictions robustly across frames while reducing memory and computation compared with point-level IoU.Frame-wise comparisons are reported as more robust to wrong 2D predictions and noisy projections than tracklet-wise comparisons.Source: §1 · p. 2
  • Iterative merging, refinement, and removal suppress overlapped or partial 3D proposals after aggregation.Merged proposals are refined using multi-view consensus, and partial masks included in other proposals are removed.Source: §1 · pp. 1–2
  • Alpha-CLIP provides object-centric representations, while standardized maximum similarity filtering removes low-confidence proposals and reduces false positives.Alpha-channel masks reduce contamination from co-visible objects and irregular shapes during classification.Source: §1 · p. 2
  • The contributions include refined tracking-based proposal generation, iterative false-positive suppression, Alpha-CLIP classification, and SMS filtering.These design choices are presented as the basis for improvements on ScanNet200 and S3DIS across AP and AR metrics.Source: §1 · p. 2

2. Related Work

Related work spans closed-vocabulary 3D segmentation, open-vocabulary 2D grounding, and open-vocabulary 3D instance segmentation. The paper builds on two-stage and image-grounded approaches to address novel-class detection in 3D scenes.

2.1. Closed-vocabulary 3D Instance Segmentation

  • Closed-vocabulary 3D methods use boxes, point embeddings, graphs, clustering, transformers, or dynamic convolutions to predict instances from point clouds.Mask3D and ISBNet are used as the paper’s point-cloud instance segmentation networks.Source: §2.1 · p. 2

2.2. Open-vocabulary 2D Grounding

  • Closed-vocabulary models struggle to generalize to novel classes, motivating open-vocabulary 2D grounding with vision-language representations.Grounded SAM combines Grounding DINO for open-vocabulary boxes with SAM for instance masks.Source: §2.2 · p. 2

2.3. Open-vocabulary 3D Instance Segmentation

  • Image-based 3D proposal generation lifts grounded 2D regions into point clouds and aggregates them across frames to recover complete masks.The cited pipeline includes frame-wise grounding, 3D lifting, and temporal aggregation before classification.Source: Figure 2 · p. 3, §2.3 · p. 3
  • OV-3DIS methods either align point embeddings to CLIP or use two-stage class-agnostic proposal generation followed by instance classification.Point-cloud proposals can miss rare tail classes, so image-based proposals and combined modalities are used as alternatives.Source: §2.3 · p. 3

3. Method

The method combines image- and point-cloud-based proposal generation with refined tracking, merging, removal, Alpha-CLIP classification, and SMS filtering for OV-3DIS.

3. Method

  • The framework concatenates class-agnostic proposals from image and point-cloud methods, applies NMS, and classifies them with Alpha-CLIP and SMS filtering.Image proposals use four stages: grounding, lifting, tracking-based aggregation, and iterative merging/removal.Source: §3 · p. 3, §3.2 · p. 5

3.1. Image-based Proposal Generation

  • 2D overlap removal suppresses masks covering multiple objects before lifting image predictions into 3D superpoints using visibility and instance-support thresholds.The method accepts partial proposals because separating multi-instance masks is more difficult than merging partial proposals later.Source: §3.1 · pp. 3–4
  • Tracking associates observations with existing tracklets when co-visible-superpoint sIOU exceeds a threshold; otherwise, it initializes a new tracklet.Each tracklet corresponds to one aggregated 3D instance proposal.Source: §3.1 · p. 4, Figure 3 · p. 4
  • Multi-view consensus refinement removes infrequently visible superpoints, producing tighter boundaries and reducing geometric noise in tracked proposals.A superpoint is retained only when its consensus rate exceeds the predefined refinement threshold.Source: §3.1 · p. 4, Figure 4 · p. 4
  • Iterative merging combines highly overlapping partial proposals and refines each merge, while inclusion-based removal deletes smaller proposals contained within larger ones.Merging repeats until no further qualifying overlaps remain; containment filtering is applied once.Source: §3.1 · p. 5, Figure 5 · p. 5

3.2. Open-Vocabulary Instance Classification

  • Alpha-CLIP uses object masks to reduce co-visible-object contamination, while SMS standardizes maximum proposal-query similarity before threshold-based filtering.Visual features aggregate multiscale views weighted by proposal visibility, and low-SMS proposals are removed.Source: §3.2 · p. 5
  • On ScanNet200, the method achieves state-of-the-art performance across Top-1 and Top-K protocols and 2D-only, 3D-only, and combined proposal settings.The table evaluates image-based, point-cloud-based, and combined proposals under both classification protocols.Source: Table 1 · p. 6

4. Experiments

The method is evaluated across ScanNet200, S3DIS, and Replica using multiple proposal settings, qualitative comparisons, and ablations. It consistently improves benchmark performance, while experiments identify contributions from proposal refinement, tracklet matching, Alpha-CLIP, and SMS filtering.

4.1. Experimental Setup

  • The evaluation covers ScanNet200, S3DIS, and Replica, using mAP and mAR at IoU thresholds of 25%, 50%, and 50%–95%, with both Top-1 and Top-K protocols where applicable.Replica additionally tests generalization using a ScanNet200-trained 3D instance segmentation model.Source: §4.1 · p. 6, Table 3 · p. 7

4.2. Quantitative Results

  • Our method outperforms baselines across 2D-only, 3D-only, and 2D+3D settings on ScanNet200 and S3DIS, with gains reported across AP and AR metrics.On ScanNet200, it exceeds previous methods by 8.8%, 7.2%, and 9.0% in the reported comparisons; on S3DIS, combining proposal sources improves recall by 12.9% and 9.5%.Source: §4.2 · pp. 6–7

4.3. Qualitative Results

  • Qualitative ScanNet200 results show that our method detects instances missed by Open3DIS and OpenYOLO3D, matching its higher mAR of 61.4% versus 43.3% and 47.7%.The framework also retrieves instances from functional descriptions and object attributes.Source: §4.3 · p. 7, Figure 7 · p. 7

4.4. Ablation Study

  • Combining image-based and point-cloud-based proposals achieves the highest class-agnostic recalls, including over fully supervised approaches.The 2D-only method also surpasses the previous image-based state of the art by 3.6%.Source: §4.4 · p. 8, Table 4 · p. 8
  • Frame-wise sIOU tracklet matching improves AP50 and AP25 over tracklet-wise matching by avoiding the influence of wrong predictions in aggregated masks.The comparison is reported on a ScanNet200 validation subset.Source: §4.4 · p. 8, Table 5 · p. 8
  • Iterative merging and removal improves AP25 by 5.0%, while overlap removal and iterative refinement provide additional gains across AP metrics.Overlap removal helps separate masks spanning multiple instances or partial masks before later consolidation.Source: §4.4 · p. 8, Table 6 · p. 8
  • Alpha-CLIP raises mAP from 27.5 to 30.5, and SMS filtering further improves AP by removing unconfident instances.The ablation evaluates object-centric representation and proposal filtering on ScanNet200.Source: Table 7 · p. 8

5. Conclusion

The paper combines complementary proposal-generation and classification concepts into a precise OV-3DIS pipeline. It reports strong precision and recall across datasets but remains computationally intensive and does not improve performance on small objects.

  • The framework combines robust tracking, overlap removal, iterative merging and refinement, Alpha-CLIP, and SMS filtering to improve proposal and classification quality.These components target overlapped, partial, or unconfident 3D proposals.Source: §5 · p. 8
  • The method achieves state-of-the-art precision and recall across datasets but is computationally intensive because it uses heavy 2D foundation models.The paper identifies this computational cost as a limitation.Source: §5 · p. 8
  • Performance on small objects remains similar to existing approaches because iterative merging and removal is more effective for medium or large objects.Improving small-object performance is left as future work.Source: §5 · p. 8

A. Algorithm on 3D Proposal Merge and Refinement

The merge-and-refinement algorithm repeatedly consolidates similar 3D proposals and removes low-visibility content until no mergeable pairs remain. It maintains synchronized proposal and tracklet lists throughout this process.

B. More Implementation Details

Implementation varies proposal filtering and feature processing by dataset while using Alpha-CLIP and SAM for instance classification. Classification dominates inference time, accounting for approximately 76% of Replica computation.

  • Alpha-CLIP uses SAM-derived instance queries, with bounding boxes for ScanNet200 and Replica and subsampled points for S3DIS.Subsampled points reduce noise from S3DIS stuff classes.Source: B. More Implementation Details · p. 12
  • SMS thresholds are tuned by experiment, while performance remains stable over the range τ_SMS ∈ [−1.0, 1.0].The method selects the top 20 visible images for ScanNet200 and Replica and top 40 for S3DIS.Source: B. More Implementation Details · p. 12
  • Replica inference takes approximately 597 seconds per scene, with instance classification contributing around 456 seconds, or 76% of total cost.Measurements use a single NVIDIA RTX 4090 GPU.Source: B. More Implementation Details · p. 12

C. Analysis of Computational Cost

The computation-time analysis examines how scene and pipeline factors relate to runtime, identifying 2D grounding as the dominant cost.

D. Additional Experiments

This section reports additional quantitative and qualitative results that were not included in the main paper.

D.1. Quantative Results

Additional experiments evaluate ScanNet++, S3DIS with stuff classes, SMS filtering, hyper-parameters, and Alpha-CLIP-based classification. Results generally remain competitive or improve over baselines, with dataset- and class-specific limitations.

Class-agnostic Evaluation Results on ScanNet++

  • On ScanNet++, the method achieves on-par AP with Open3DIS + SAM and improves AP50 by 1.7% and AP25 by 3.7% under Grounded SAM.Source: §D.1 · p. 13

Ablation Study on SMS filtering

  • The SMS score filters partial, incorrect, and noisy proposals that lack complete or meaningful object representations.Source: Figure 9 · p. 13

S3DIS Results Including “stuff” Classes

  • Including stuff classes, the method consistently outperforms baselines in 2D-only and 2D+3D groups and achieves SoTA results across all AR metrics.Source: §D.1 · p. 13
  • In the 3D-only S3DIS group, the method falls slightly behind OpenYOLO3D on mAP50 and mAP25 because of weaker stuff-class performance.Source: §D.1 · p. 13

Ablation Study on SMS filtering

  • SMS filtering consistently outperforms the no-filtering baseline, while thresholds within [-1, 1] vary by at most 0.4% in AP.Source: §D.1 · p. 14, Figure 10 · p. 14

Impact of Hyper-parameters

  • The proposal-generation algorithm is sensitive to τ_img and τ_inst, whereas it remains less sensitive to other hyper-parameters.Source: Table 10 · p. 14

Alpha CLIP vs CLIP

  • Alpha-CLIP consistently improves performance across datasets and metrics, and combining it with SMS filtering achieves SoTA performance.Source: §D.1 · p. 14

D.2. Qualitative Results

Qualitative evaluations show accurate instance retrieval across datasets and text-query types, while also exposing missed detections, misclassifications, and incomplete masks in specific settings.

Qualitative Results with Dataset-Provided Text Queries

Qualitative Results with the New Text Queries

  • With new text queries, the method retrieves instances using functional descriptions and attributes such as color, brand name, and other features.Source: §D.2 · p. 14, Figure 14 · p. 18

Related papers

Loading 2507.23134v1…