PaperlayerView exact revision on arXiv

Source-linked AI summary

OVI-MAP:Open-Vocabulary Instance-Semantic Mapping

Zilong Deng, Federico Tombari, Marc Pollefeys, Johanna Wald, Daniel Barath

arXiv:2603.26541v1cs.CV

TL;DR

Open-vocabulary 3D mapping must combine robust instance tracking, real-time processing, and flexible open-set reasoning despite the limits of closed-set and dense semantic-fusion pipelines. OVI-MAP separates class-agnostic instance reconstruction from semantic inference, using selective VLM views to build stable instance embeddings. It achieves state-of-the-art instance mapping and open-set semantic segmentation while maintaining online processing rates.

  • Problem

    Existing 3D mapping pipelines rely on closed-set labels or dense semantic fusion, causing computational overhead and instability under open-set, multi-view online operation.

    Source: §1 · p. 1

  • Method

    OVI-MAP incrementally builds a class-agnostic TSDF-based 3D instance map, then aggregates VLM features from informative views selected by object-centric coverage.

    Source: §1 · pp. 1–2, §3 · p. 3, §3.2 · p. 5

  • Results

    OVI-MAP achieves state-of-the-art instance mapping and open-set semantic segmentation while maintaining online processing rates, with less than half the VLM queries of pixel-counting selection on average.

    Source: §1 · p. 2, §4.3 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    Decoupling geometry-based instance tracking from selective semantic inference supports stable zero-shot semantic reasoning under online constraints.

    Source: §3 · p. 3, §3.2 · p. 5, §5 · p. 8

    Performance remains dependent on 2D segmentation quality, limiting results for small or visually complex objects.

    Source: §5 · p. 8

Abstract

from arXiv · show

Incremental open-vocabulary 3D instance-semantic mapping is essential for autonomous agents operating in complex everyday environments. However, it remains challenging due to the need for robust instance segmentation, real-time processing, and flexible open-set reasoning. Existing methods often rely on the closed-set assumption or dense per-pixel language fusion, which limits scalability and temporal consistency. We introduce OVI-MAP that decouples instance reconstruction from semantic inference. We propose to build a class-agnostic 3D instance map that is incrementally constructed from RGB-D input, while semantic features are extracted only from a small set of automatically selected views using vision-language models. This design enables stable instance tracking and zero-shot semantic labeling throughout online exploration. Our system operates in real time and outperforms state-of-the-art open-vocabulary mapping baselines on standard benchmarks.

1. Introduction

OVI-MAP addresses the scalability and consistency limits of open-vocabulary 3D mapping by separating class-agnostic instance reconstruction from semantic inference. It incrementally builds a TSDF-based instance map and extracts VLM features only from selected views for efficient open-set recognition.

1. Introduction

  • Closed-set pipelines struggle with open-set mapping because continuous VLM features are costly to store densely, while semantic-free grouping and per-pixel fusion are unstable.These challenges are compounded by occlusions, viewpoint changes, background noise, inconsistent 2D segmentation, and the cost of running segmentation models per frame.Source: §1 · p. 1
  • OVI-MAP decouples instance mapping from open-set semantic recognition, reconstructing class-agnostic instances from multi-view evidence without requiring semantic labels.The resulting instances are incrementally lifted into a global TSDF-based representation for consistent tracking across frames.Source: §1 · pp. 1–2
  • Object-centric view selection queries VLMs only when a new viewpoint provides non-redundant 3D surface coverage, reducing redundant feature extraction.Selected views are used for efficient semantic aggregation rather than dense per-voxel embedding storage.Source: §1 · p. 2, Figure 2 · p. 3

2. Related Work

  • Prior 2D open-vocabulary models provide promising image-level features but do not natively provide long-term, consistent 3D segmentation across views.OVI-MAP instead focuses on incremental 3D scenes and separates instance mapping from semantic recognition.Source: §2 · p. 2
  • Existing 3D VLM methods often use dense semantic fields or global optimization, limiting their suitability for online incremental mapping.OVI-MAP computes semantics per instance through selective view aggregation to target scalable real-time operation.Source: §2 · p. 2
  • Prior open-set instance-semantic systems commonly assume available instance masks, operate offline, use closed-set segmentation, or incur redundant semantic fusion.OVI-MAP reconstructs instances online in a class-agnostic manner and adds object-centric view selection for efficient VLM aggregation.Source: §2 · pp. 2–3

3. Proposed Method

OVI-MAP decouples class-agnostic 3D instance reconstruction from open-vocabulary semantic inference. It incrementally stabilizes instances geometrically, then extracts VLM features only from informative, non-redundant viewpoints.

3. Proposed Method

  • OVI-MAP incrementally constructs a class-agnostic 3D instance map and assigns each instance a zero-shot semantic embedding through selective multi-view extraction.The design separates instance formation from semantic inference: instances use geometric and region-consistent evidence, while semantics are assigned from informative views.Source: §3 · p. 3

3.1. Class-Agnostic Instance Map Reconstruction

  • Class-agnostic entity masks are refined with depth discontinuities, lifted into global 3D point clouds, and associated with existing instances using spatial voxel voting.Geometric refinement splits masks at boundaries that appearance may miss, reducing over-merging between adjacent objects with similar texture or color.Source: §3.1 · pp. 3–4
  • Spatial voting assigns a lifted segment to the most frequent voxel instance when overlap exceeds θ_assoc; otherwise, OVI-MAP creates a new instance.The association uses spatial consistency rather than semantic similarity, avoiding dependence on a closed label set or hierarchical class priors.Source: §3.1 · p. 4
  • Weighted TSDF fusion and per-voxel label-support accumulation produce a temporally stable class-agnostic map that improves with additional viewpoints.Voxel labels are updated from accumulated support, while spatially close super-points can be merged to handle over-segmentation and multi-view fusion.Source: §3.1 · pp. 3–4

3.2. View-Adaptive Semantic Feature Aggregation

  • For each instance, OVI-MAP maintains a spherical view-coverage map and selects a frame for semantic extraction only when its novel-view ratio exceeds θ_novel.Viewing directions are converted to spherical coordinates, and newly occupied bins identify geometric or appearance information not previously observed.Source: §3.2 · p. 4
  • Object-centric coverage favors diverse novel viewpoints over redundant large front-facing views, then aggregates VLM features from cropped and background-masked object views with visibility-aware weighting.The resulting embedding is intended to remain stable and view-invariant while reducing noise, redundancy, memory use, and VLM calls.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

4. Experiments

Experiments show that OVI-MAP supports accurate online instance-semantic mapping across Replica and ScanNet while reducing redundant VLM queries through object-centric view selection. Ablations further identify visibility-aware fusion and instance-segmentation quality as important determinants of semantic performance.

4.1. Instance Segmentation Results

  • OVI-MAP matches or exceeds offline instance-segmentation accuracy while consistently outperforming prior online systems, especially at AP75.Mask3D performs best on ScanNet but was trained on that dataset.Source: Table 2 · p. 5, §4.1 · p. 5

4.2. Semantic Segmentation Results

  • OVI-MAP produces spatially coherent maps with sharp instance boundaries, consistent semantics, and zero-shot retrieval of concrete and abstract text queries.Figure 5 uses cosine similarity to highlight queried 3D instances.Source: Figure 4 · p. 6, Figure 5 · p. 7, §4.2 · p. 7
  • OVI-MAP achieves the best open-vocabulary semantic-instance accuracy among online systems and remains comparable to offline methods under real-time constraints.On both Replica and ScanNet, it improves instance AP over OVO-SLAM, particularly at AP50 and AP25.Source: Table 3 · p. 6, §4.2 · pp. 6–7

4.3. Ablation Studies

  • Object-centric view coverage achieves comparable accuracy to pixel-counting while using 47.0% as many VLM queries on average.It selects frames contributing novel object-surface regions rather than redundant observations.Source: Table 4 · p. 7, §4.3 · p. 7
  • As more views are processed, view coverage and pixel-counting improve semantic accuracy, while view coverage uses fewer queries and provides incremental feedback unavailable to post-hoc methods.The view-coverage strategy achieves slightly better accuracy than pixel-counting in the incremental evaluation.Source: §4.3 · pp. 7–8, Figure 6 · p. 8
  • CropFormer improves over SAM2 by +5.7 AP50 and +6.8 semantic mIoU, while ground-truth masks provide an upper bound.The results associate cleaner, more consistent instance boundaries with more discriminative VLM embeddings.Source: §4.3 · p. 8, Table 5 · p. 8
  • Visibility-aware weighted fusion performs best across metrics, whereas clustering adds no benefit and can degrade performance through redundant-feature emphasis.Averaging remains a strong baseline because multi-view aggregation reduces noise in individual predictions.Source: §4.3 · p. 8, Table 6 · p. 8

5. Conclusions

OVI-MAP enables incremental open-vocabulary semantic mapping by combining class-agnostic instance reconstruction with language-guided feature aggregation. Its view-coverage strategy reduces VLM queries while maintaining semantic consistency, although performance remains constrained by segmentation quality and weak vision-language alignment.

5. Conclusions

  • OVI-MAP performs incremental open-vocabulary mapping efficiently by combining class-agnostic instance reconstruction with language-guided feature aggregation.The system is designed for online constraints and maintains semantic consistency through object-centric view coverage.Source: §5 · p. 8
  • Object-centric view coverage substantially reduces VLM queries while preserving semantic consistency during online exploration.This component is presented as central to real-time semantic reasoning.Source: §5 · p. 8
  • Performance is limited by 2D segmentation quality, background-biased masked crops, and weak visual-textual alignment in current VLMs.These limitations particularly affect small or visually complex objects and can cause ambiguous labels.Source: §5 · p. 8

6. Additional Details of the Method

The method merges spatially overlapping super-points into stable instances, then ray-casts the TSDF map to obtain globally aligned 2D instance masks for view selection and feature extraction.

6.1. Super-Point Merging

  • Super-points whose spatial overlap exceeds θmerge are merged into one instance with the same label.Overlap counts point clouds containing sufficient voxel support for both super-points, using θassoc as the per-super-point threshold.Source: §6.1 · p. 12

6.2. TSDF Map Projection via Ray-Casting

  • After TSDF fusion and label stabilization, the system obtains a global TSDF-based instance map with stabilized labels across frames.Source: §6.2 · p. 12
  • Ray-casting each camera pixel through TSDF voxels, combined with depth input, produces a globally aligned 2D instance mask that accounts for occlusion and multi-view consistency.Source: §6.2 · p. 12

7. Ablation Study on the VLM Backbone

The ablation compares CLIP with several SigLIP backbones and finds that view selection remains effective across models, while larger SigLIP variants improve accuracy at moderate parameter cost.

7. Ablation Study on the VLM Backbone

  • View selection remains effective across CLIP and multiple SigLIP backbones, indicating that the strategy is not tied to one VLM.Table 7 compares ViT-L, siglip-large, siglip-so400m, siglip2-large, and siglip2-so400m, along with parameter counts.Source: §7 · p. 12, Table 7 · p. 12
  • Larger SigLIP variants consistently improve mIoU and AP while adding parameters moderately.Source: §7 · p. 12
  • siglip-large-patch16-384 is selected as the method’s backbone because it offers the best trade-off between accuracy and computational cost.Source: §7 · p. 12

8. Runtime

The runtime design uses a multithreaded pipeline to parallelize mapping and semantic components, while processing expensive segmentation and semantic steps only on selected frames.

8. Runtime

  • Segmentation, 2D-3D association, view selection, and semantic feature extraction run in parallel on dedicated CPU and GPU threads.2D-3D association includes 3D lifting, super-point updates, and global instance-map generation by ray-casting.Source: §8 · p. 12
  • Instance and semantic segmentation run only every n frames rather than every incoming frame, substantially reducing overall latency.Source: §8 · p. 12

9. Visualizations of Instance Maps

Visual comparisons show that OVI-MAP maintains coherent instance boundaries during incremental mapping and provides dense coverage relative to the compared methods.

9. Visualizations of Instance Maps

  • OVI-MAP maintains consistent instance boundaries and achieves dense scene coverage, whereas offline methods leave large unlabeled regions despite full-scene access.Source: §9 · p. 12, Figure 7 · p. 13
  • Figure 7 compares OVI-MAP with online and offline approaches on Replica and ScanNet, using instance-ID colors and gray regions for unobserved or unlabeled areas.Source: §9 · p. 12, Figure 7 · p. 13

10. Heat Maps

The heat maps visualize how closely scene instances match queried semantic labels, showing stronger matches in red and weaker matches in blue. These visualizations support object localization and semantic-map updates.

  • Cosine similarities between instance and query-label semantic features are normalized and mapped to a color scale.Source: §10 · p. 13
  • Red indicates higher similarity to the queried label, blue indicates lower similarity, and black marks unobserved areas.Source: Figure 8 · p. 14
  • Matching objects are highlighted more strongly than irrelevant objects, demonstrating the method’s scene-recognition effectiveness.Source: §10 · pp. 13–14
  • The heat maps can help locate queried objects and update the semantic map in those regions.Source: §10 · p. 14

11. Datasets

Experiments use the Replica and ScanNet datasets, both established benchmarks for 3D scene understanding. Evaluation covers eight Replica scenes and eighteen ScanNet scenes.

  • Experiments are conducted on Replica and ScanNet, widely used benchmarks for 3D scene understanding.Source: §11 · p. 14
  • Replica evaluation uses eight scenes: office0–office4 and room0–room2.Source: §11 · p. 14
  • ScanNet evaluation uses eighteen scenes selected similarly to ConceptFusion [19].Source: §11 · p. 14

Related papers

Loading 2603.26541v1…