PaperlayerView exact revision on arXiv

Source-linked AI summary

OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data

Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boularias, Kostas Bekris

arXiv:2311.02873v1cs.CVcs.RO

TL;DR

Open-vocabulary 3D instance retrieval is constrained by limited 3D data and by existing approaches that struggle to provide arbitrary-query instance separation. OVIR-3D fuses text-aligned 2D region proposals into 3D instances, and the paper reports better performance than baselines without 3D instance annotations or additional training.

  • Problem

    Limited large 3D datasets and existing approaches’ reliance on heatmap thresholding or extra instance annotations constrain open-vocabulary 3D instance retrieval.

    Source: §1 · p. 1

  • Method

    OVIR-3D fuses text-aligned 2D region proposals and features into 3D, then ranks instance segments by similarity to a text query.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    OVIR-3D achieves much better performance than other baselines without 3D instance annotation, additional training, or manual heatmap thresholding during inference.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The method can also be used to generate 3D instance pseudo-labels for self-supervised learning.

    Source: §8 · p. 9

    The method can fail to merge very large instances, miss tiny objects removed as noise, and depends on the 2D proposal model not consistently missing objects or producing poor segmentation.

    Source: §8 · p. 9

Abstract

from arXiv · show

This work presents OVIR-3D, a straightforward yet effective method for open-vocabulary 3D object instance retrieval without using any 3D data for training. Given a language query, the proposed method is able to return a ranked set of 3D object instance segments based on the feature similarity of the instance and the text query. This is achieved by a multi-view fusion of text-aligned 2D region proposals into 3D space, where the 2D region proposal network could leverage 2D datasets, which are more accessible and typically larger than 3D datasets. The proposed fusion process is efficient as it can be performed in real-time for most indoor 3D scenes and does not require additional training in 3D space. Experiments on public datasets and a real robot show the effectiveness of the method and its potential for applications in robot navigation and manipulation.

1 Introduction

OVIR-3D addresses open-vocabulary 3D instance retrieval without additional 3D training, a setting motivated by limited 3D data and existing methods’ difficulty with arbitrary queries or instance separation. It fuses text-aligned 2D region proposals into 3D to rank object instances for a language query.

1 Introduction

  • Existing 3D open-vocabulary methods face limited 3D training data, while heatmap thresholding can hinder instance separation and additional instance annotations reduce openness.The introduction identifies these constraints as barriers to robotic manipulation and navigation.Source: §1 · p. 1
  • For 3D scans reconstructed from RGB-D video, the system retrieves instances ranked by semantic similarity to text queries such as bed or lamp, and can query unannotated instances such as cushions on the sofa.Source: Figure 1 · p. 2

2 Related Work

The related work spans open-vocabulary 2D detection and segmentation, 3D reconstruction and semantic mapping, and open-vocabulary 3D scene understanding. OVIR-3D builds on 2D text-aligned proposals while targeting instance-level 3D querying without manual 3D annotation.

2.1 2D Open-Vocabulary Detection and Segmentation

  • Detic provides pixel-level instance segmentation and text-aligned features, and can be queried with a large vocabulary without sacrificing much speed.The paper adopts it as the 2D backbone detector; related methods include LSeg, OpenSeg, GroupViT, ViLD, and OWL-ViT.Source: §2.1 · pp. 2–3

2.2 3D Reconstruction and Closed-Vocabulary Semantic Mapping

  • Prior 3D mapping methods use varied reconstruction representations, but often assume mutually exclusive instances or use category labels for data association.OVIR-3D instead associates open-vocabulary 2D instance detections and fuses them into a point cloud while retaining open-vocabulary querying.Source: §2.2 · p. 3

2.3 3D Open-Vocabulary Scene Understanding

  • Many open-vocabulary 3D methods fuse pretrained 2D features, but heatmap-based methods do not provide instance-level segmentation for interaction with specific objects.PLA also targets instance segmentation, though the paper says its demonstrated performance is limited to certain furniture-scale objects and other categories remain unclear.Source: §2.3 · p. 3

3 Problem Formulation

The paper formulates open-vocabulary 3D instance retrieval as information retrieval over a reconstructed point-cloud scan. Given a text query, the goal is to return a ranked set of 3D instance masks ordered by semantic similarity.

3 Problem Formulation

  • The input scan is reconstructed from an RGB-D video, given camera intrinsics and the camera poses for its frames.The scan is represented by N points.Source: §3 · p. 3
  • For a text query and desired retrieval count K, the output is a set of K binary masks over the 3D scan.Source: §3 · p. 3
  • Retrieved instance masks are ranked by semantic similarity between each 3D instance and the text query, with the most similar instance first.Source: §3 · p. 3

4 Method

OVIR-3D builds 3D instances by projecting text-aligned 2D proposals into a reconstructed point cloud, associating and integrating them over frames, then filtering and ranking instances against a text query. Its method combines 2D region proposals, 3D fusion and cleanup, and representative-feature matching at inference.

4 Method

  • Text-aligned 2D proposals are projected into the 3D scan, matched to existing instances or added as new ones, and later filtered, merged, and ranked against a query.The pipeline uses camera intrinsics and poses for projection, then matches using feature similarity and 3D overlap.Source: §4 · pp. 3–4

4.1 Text-aligned 2D Region Proposal

  • Detic generates pixel-wise 2D region masks and text-aligned features by querying an extensive vocabulary, avoiding direct 3D proposal learning.The method uses the 21k ImageNet21k categories; region labels are dropped because they are noisy for the large vocabulary.Source: §4.1 · p. 4

4.2 2D-to-3D Instance Fusion

  • A projected region matches a 3D instance when feature similarity exceeds 0.75 and 3D overlap exceeds 0.25; matched regions are integrated, including multiple proposals per instance.The instance feature used for matching is the average of its associated 2D features.Source: §4.2 · p. 5

4.3 Periodic 3D Instance Filtering and Merging

  • Every 300 frames, point filtering removes points with detection rate below 0.2, then removes instances smaller than the median corresponding projected segment; similar, overlapping instances can be merged.Merging also accepts an instance mostly contained in another when recall is at least 0.25 and their features are sufficiently similar.Source: §4.3 · p. 5

4.4 Post-processing

  • DBSCAN separates disconnected point clusters within an instance, while segments likely to be noise are filtered using a 10 cm distance parameter.An instance with multiple disconnected clusters is split into multiple instances.Source: §4.4 · p. 5

4.5 Inference

  • At inference, K-Means clustering supplies K representative features per instance, which are scored by their largest cosine similarity to the text-query feature.The method uses CLIP to convert the query into a feature vector.Source: §4.5 · p. 5

5 Experiments

Experiments evaluate retrieval on room-scale ScanNet200 and tabletop YCB-Video, comparing the method with OpenScene and adapted mapping baselines. The method leads in instance-retrieval mAP, with near-instant query inference and fusion at about 30 FPS on most ScanNet200 scans.

5.1 Datasets

  • ScanNet200 provides 312 validation scans across 200 object categories, while YCB-Video provides 12 tabletop validation videos.YCB-Video scans are reconstructed with KinectFusion, and instance labels are generated from object meshes and annotated 6DoF poses.Source: §5.1 · pp. 5–6

5.2 Metrics

  • Retrieval is evaluated with category-averaged mAP at IoU thresholds 0.25, 0.5, and 0.50–0.95.Each annotated category is treated as an independent query, regardless of the number of test categories.Source: §5.2 · p. 6

5.3 Baselines

  • OpenScene is the main comparison; Fusion++ and PanopticFusion are adapted with given reconstructions and poses, Detic detections, and instance-feature matching.OpenScene heatmaps are thresholded and clustered into 3D instances, with the best overall threshold reported.Source: §5.3 · p. 6

5.4 Results

  • The method outperforms all baselines by a large margin in instance-retrieval mAP on ScanNet200 and YCB-Video.Table 1 reports results on both datasets; qualitative examples show imperfect masks but often detect instances beyond the testing categories.Source: Table 1 · p. 6, §5.4 · p. 6, Figure 3 · p. 7
  • Masks are imperfect, but the method often detects instances beyond the testing categories in ScanNet200 and YCB-Video scenes.The first three figure rows show ScanNet200 scenes and the last two show YCB-Video scenes.Source: Figure 3 · p. 7

5.5 Running Time

  • Text-query inference takes about 20 ms, and fusion runs at about 30 FPS for most ScanNet200 scans on an NVIDIA RTX 3090.Fusion runtime is O(MN), where N is detections per frame and M is fused 3D instances.Source: §5.5 · p. 7

6 Ablation Studies

The ablations test input vocabularies, feature and mask alternatives, feature-ensemble strategies, and periodic filtering settings. They report benefits from extensive vocabularies, SAM masks, clustered features, and the T = 300 and θvis = 0.2 settings.

6.1 Input queries to the 2D region proposal method

  • An extensive query vocabulary helps generate regions of interest, and removing ScanNet200 categories from ImageNet21K only slightly drops retrieval performance.The proposal network can still find most regions using similar categories in the vocabulary.Source: §6.1 · p. 8, Table 3 · p. 8

6.2 Instance features and 2D masks

  • SAM masks improve performance over Detic masks, while cropped-image CLIP features yield slightly inferior results to Detic features.The ablation fused instances every three frames, primarily because SAM was relatively slow.Source: §6.2 · p. 8, Table 4 · p. 8

6.3 Feature ensemble strategies

  • Feature clustering outperforms simple averaging, while using the feature from the largest associated 2D region yields the poorest results.The three tested strategies represent each instance using an average, K-Means centers, or its largest associated region.Source: §6.3 · p. 8, Table 5 · p. 8

6.4 Time intervals and visibility threshold for periodic instance filtering and merging

  • T = 300 and θvis = 0.2 yield the best results among the tested periodic filtering intervals and visibility thresholds.The results are shown in Table 6 for intervals and Table 7 for visibility thresholds.Source: §6.4 · p. 9, Table 6 · p. 9, Table 7 · p. 9

7 Robotics Experiments

The robot experiments test whether language queries can guide grasping of specific object parts after a short RGB-D scanning phase. OVIR-3D detected more target parts and achieved a higher grasp success rate than OpenScene in these setups.

7 Robotics Experiments

  • OVIR-3D achieved a 90% grasp success rate, versus 30% for OpenScene, across experiments targeting bottle caps and pitcher handles.OVIR-3D detected 5/5 graspable bottle caps and 4/5 handles; OpenScene detected 0/5 caps and 3/5 handles.Source: §7 · p. 9

8 Conclusion and Limitations

OVIR-3D fuses text-aligned 2D regions and features in 3D to retrieve instances without 3D instance annotations, additional training, or manual heatmap thresholding at inference. Its limitations include difficulty merging very large instances, missing tiny objects, and reliance on the 2D proposal model.

8 Conclusion and Limitations

  • OVIR-3D fuses off-the-shelf open-vocabulary 2D regions and text-aligned features in 3D, achieving better performance than baselines without 3D instance annotations or additional training.The method also avoids manual heatmap thresholding during inference and may support 3D instance pseudo-label generation for self-supervised learning.Source: §8 · p. 9
  • The method may fail to merge segments of very large instances, miss tiny objects removed as noise, or inherit consistent misses and poor segmentation from its 2D proposal model.The authors suggest integrating the method with a 3D learning-based method using scarcer but cleaner 3D annotations.Source: §8 · p. 9

Related papers

Loading 2311.02873v1…