PaperlayerView exact revision on arXiv

Source-linked AI summary

AOV-GS: Active Open-Vocabulary Semantic Mapping with 3D Gaussian Splatting

Rodion Shkokov, Tatiana Zemskova, Dmitry Yudin

arXiv:2610.29374v1cs.CVcs.RO

TL;DR

Robots need maps that support both navigation and open-vocabulary semantic queries, but geometry-driven exploration leaves semantic utility to the collected trajectory. AOV-GS uses frozen SAM and CLIP to guide coarse view selection and fits a language field from collected embeddings. It improves average language-field mIoU over geometry-only exploration on Replica and Matterport3D while preserving comparable novel-view reconstruction quality.

  • Problem

    Robots need maps for navigation and open-vocabulary semantic queries, while geometry-driven exploration treats semantic utility as a by-product of the collected trajectory.

    Source: §1 · pp. 1–2

  • Method

    AOV-GS re-ranks geometrically strong coarse-exploration views using CLIP novelty from SAM masks, then fits an offline language field from collected embeddings.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    34.6% on Replica (+1.3 pp) and 23.1% on Matterport3D (+1.8 pp) are AOV-GS’s average language-field mIoU results over geometry-only exploration, with comparable novel-view reconstruction quality.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    Across Replica and Matterport3D, AOV-GS maps improve average open-vocabulary language-field mIoU over geometry-only exploration while preserving comparable novel-view reconstruction quality.

    Source: §5 · p. 14

    CorrCLIP is not uniformly beneficial: merging regions or suppressing small and overlapping masks can remove useful object evidence, so its effect depends on the scene.

    Source: §4.5 · p. 12

Abstract

from arXiv · show

Robots deployed in human-centered environments need maps that support both geometric navigation and open-vocabulary semantic queries beyond a fixed label set. A key challenge is enriching such maps with semantic information without relying on manually annotated segmentation masks or closed-set semantic categories. Existing geometry-driven active mapping maximizes coverage but treats semantics as a passive by-product of the collected trajectory, whereas active semantic mapping typically depends on supervised semantic segmentation. We present AOV-GS, an active 3D Gaussian mapping system that guides exploration using frozen foundation models. During coarse exploration, AOV-GS re-ranks geometrically promising candidate views according to CLIP mask-embedding novelty computed from SAM instance segmentation, biasing exploration toward semantically informative regions. Frozen SAM and CLIP provide instance masks and language-aligned embeddings throughout mapping, while a lightweight scene-specific language field is fitted offline directly to the collected CLIP embeddings rather than manually annotated masks, enabling open-vocabulary text-query segmentation on novel views. In a controlled comparison using the same SAM-CLIP perception stack, AOV-GS improves average language-field mIoU over geometry-only active planning by 1.3 percentage points on Replica (34.6% on eight scenes) and by 1.8 points on Matterport3D (23.1% on five scenes). With CorrCLIP, AOV-GS reaches 35.6% on Replica, 2.0 points above the matched Geometry-based + CorrCLIP baseline. Novel-view PSNR remains comparable to the geometry-only baseline on both datasets while depth L1 improves on Replica. Code is available at https://github.com/OptimusFaber/AOV-GS.

1 Introduction

AOV-GS addresses the challenge of building maps that support both navigation and open-vocabulary semantic queries by making semantic information part of exploration. It uses frozen foundation models to guide coarse view selection and reports average semantic-segmentation gains over geometry-only exploration.

Motivation

  • Robots in human-centered environments need maps for navigation and task-level semantic queries, while exploration determines geometric completeness, object visibility, and retrieval evidence.The paper asks whether frozen foundation models can guide exploration to improve open-vocabulary segmentation while preserving competitive RGB-D reconstruction quality.Source: §1 · pp. 1–2

Research gap

  • Geometry-aware active planners improve coverage, while open-vocabulary Gaussian methods typically process fixed trajectories.ActiveSGM [4] guides exploration with semantics but relies on per-scene OneFormer fine-tuning; passive fusion does not affect online acquisition.Source: §1 · p. 2

Approach

  • During coarse exploration, AOV-GS re-ranks the top-K geometric candidates using CLIP embedding novelty from SAM instance masks, then returns to geometry-only planning during refinement.Candidate RGB views are rendered from the current Gaussian map; executed views supply posed RGB-D observations, using ground-truth poses as oracle SLAM input in the experiments.Source: §1 · p. 2

Contributions

  • AOV-GS combines online semantic exploration with offline language-field construction from collected embeddings, without per-scene segmenter fine-tuning.The evaluation covers two benchmarks.Source: §1 · p. 2

Results

  • 34.6% on Replica (+1.3 pp) and 23.1% on MP3D (+1.8 pp) are AOV-GS’s average results over geometry-only exploration under the matched SAM–CLIP stack.Novel-view reconstruction quality remains comparable to geometry-only exploration.Source: §1 · p. 2

2 Related Work

Prior active Gaussian mapping selects views for geometric coverage, while open-vocabulary semantic mapping often processes fixed trajectories. AOV-GS uses semantic novelty to influence exploration, with frozen SAM and CLIP rather than closed-set semantic supervision.

Active and semantic Gaussian mapping

  • Online 3DGS systems such as ActiveGS, ActiveSplat, and ActiveGAMER optimize geometry, while semantic Gaussian mapping [26] [1] typically attaches language-aligned features to 3D primitives on fixed trajectories.AOV-GS follows LangSplatV2 for its language-field head but uses semantic novelty to choose views during exploration.Source: §2 · p. 3

Related active semantic methods

  • ActiveSGM [4] couples geometric and semantic exploration terms but requires per-scene OneFormer fine-tuning; passive fusion improves semantics on recorded paths without steering acquisition.SegmATRon maximizes image segmentation quality through agent actions rather than building a persistent semantic map.Source: §2 · p. 3

Other semantic mapping

  • AOV-GS trades closed-set semantic supervision for open-vocabulary scene representations based on frozen SAM and CLIP.LEG-SLAM [28] and M3DMap integrate language-aligned or object-aware semantics into 3D reconstruction.Source: §2 · p. 3

3 Method

AOV-GS combines Gaussian mapping, staged active next-best-view planning, parallel SAM–CLIP keyframe embeddings, and post-exploration language-field training and querying.

3.1 Gaussian Mapping Backbone

  • SplaTAM reconstructs the scene as isotropic 3D Gaussians from posed RGB-D frames, with voxel-based candidate sampling and an RRT-based local planner.The experiments use ground-truth camera poses as oracle SLAM input.Source: §3.1 · p. 3

3.2 Coarse Exploration Criterion

  • Geometric information gain counts unseen silhouette pixels and is normalized across candidates; candidate and robot translations define travel distance.Source: §3.2 · p. 4
  • Stage 0 re-ranks the top-K geometric candidates using SAM–CLIP novelty against accepted-keyframe embeddings, while stage 1+ returns to geometry-only planning.Candidate embeddings are transient; the bank contains embeddings from accepted keyframes, and novelty can shift the argmax when geometric scores are close.Source: §3.2 · pp. 3–5

3.3 Parallel CLIP Mask Embedding Extraction

  • Frozen SAM ViT-H and CLIP ViT-B/16 produce 512-D instance-mask embeddings at accepted keyframes; optional CorrCLIP post-processing runs asynchronously without blocking the planner.Source: §3.3 · p. 5

3.4 Language Field Construction and Query

  • After exploration, stored mask embeddings train a sparse LangSplatV2 field on Gaussians; CLIP text prompts are matched against multi-scale rendered similarity maps to segment novel views.Source: §3.4 · p. 6

3.5 Implementation Details

  • Default online settings use top-K=8 semantic candidates on an RTX 4070 Ti SUPER, with language-field training at render downscale 0.5 and query threshold τq=0.55.The query settings use open-dictionary negatives and multi-scale decoding, held fixed across compared open-vocabulary methods.Source: §3.5 · pp. 6–7

4 Experiments

Experiments compare open-vocabulary segmentation and reconstruction across Replica and Matterport3D using shared perception pipelines and held-out novel views. AOV-GS improves language-field mIoU over geometry-based planning on both benchmarks, with scene-dependent effects and reconstruction results reported alongside computational costs and ablations.

4.1 Experimental Setup

  • All methods use ground-truth Habitat camera poses; semantic evaluation uses held-out views, with Replica limited to 2000 steps and MP3D to 5000.The primary metric is language-field mIoU over held-out category–view pairs; passive comparisons are reported only on Replica.Source: §4.1 · pp. 7–8

4.2 Open-Vocabulary Semantic Segmentation on Replica; 4.5 Evaluation on Matterport3D

  • AOV-GS reaches 34.6% average language-field mIoU on Replica (+1.3 pp over Geometry-based) and 23.1% on MP3D (+1.8 pp).With CorrCLIP, it reaches 35.6% on Replica versus 33.6% for Geometry-based + CorrCLIP.Source: §4.2 · p. 8, Table 1 · p. 8, §4.5 · p. 12

4.2 Open-Vocabulary Semantic Segmentation on Replica

  • Replica gains vary by scene: reliable observations from large, well-segmented objects help, while poor lighting, overexposure, and clutter limit semantic feature quality.On office2, CorrCLIP recovers 8.6 pp; on room2, chair IoU falls from 76.3% to 32.8% while floor rises from 7.0% to 57.6%.Source: §4.2 · pp. 8–9

4.3 RGB-D Reconstruction on Replica

  • On Replica, AOV-GS matches geometry-based PSNR (31.19 vs. 31.47 dB) and improves depth L1 (1.18 vs. 1.26 cm).Across five seeds on office0, semantic mIoU is 33.6±1.1% for AOV-GS versus 28.8±1.4% for Geometry-based.Source: §4.3 · p. 11, Table 3 · p. 12, Table 4 · p. 12

4.4 Computational Cost

  • Stage-0 semantic scoring takes about 8 seconds per office0 view; increasing top-K raises runtime more than memory, while later refinement adds no semantic overhead.A complete office0 run takes 52:32, 61:23, and 70:41 for K=4, 8, and 16, respectively.Source: §4.4 · p. 11

4.5 Evaluation on Matterport3D

  • On MP3D, CorrCLIP yields 24.0% average mIoU for AOV-GS versus 21.8% for Geometry-based + CorrCLIP, while novel-view RGB rendering improves on PSNR, SSIM, and LPIPS.Depth L1 is slightly higher for AOV-GS: 2.83 versus 2.80 cm.Source: §4.5 · pp. 12–13, Table 5 · p. 13

4.6 Ablation Studies

  • On office0, top-K=8 improves mIoU over K=4 (34.4% vs. 33.2%), while render downscale 0.5 gives the best reported mIoU at 34.4%.Full resolution overfits CLIP feature noise (26.2%); downscale 0.25 reaches 32.0% with lower offline memory, and K=16 does not improve quality.Source: §4.6 · pp. 13–14, Table 7 · p. 14

5 Conclusion

AOV-GS uses frozen SAM and CLIP to guide exploration with semantic novelty and trains a scene-specific language field on collected embeddings. Its maps improve average open-vocabulary mIoU over geometry-only exploration across two benchmarks, while reconstruction quality remains comparable; gains vary by scene and the simulator study uses ground-truth poses.

5 Conclusion

  • Across Replica and Matterport3D, AOV-GS improves average open-vocabulary language-field mIoU over geometry-only exploration while preserving comparable novel-view reconstruction quality.Replica gains vary by scene, while a five-seed office0 experiment shows a consistent advantage over geometry-only exploration.Source: §5 · p. 14
  • AOV-GS guides exploration with CLIP mask-embedding novelty from frozen SAM and CLIP, then trains a scene-specific LangSplatV2 language field on collected embeddings for text-query segmentation on novel views.Source: §5 · p. 14
  • Trajectories optimized for a closed-set OneFormer pipeline do not transfer to a SAM–CLIP language field, whereas semantic novelty co-designed with that representation improves averages on both benchmarks.Source: §5 · pp. 14–15
  • The evaluation uses ground-truth camera poses, and robustness to SLAM drift and real sensor noise is outside the scope of this simulator study.Source: §5 · p. 15

Related papers

Loading 2610.29374v1…