PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Target-Aware Refinement of 3D Gaussian Language Map

Sibaek Lee, Kyeongsu Kang, Seongbo Ha, Yewon Jeon, Joonyeol Choi, Daegeol Ko, Jaehyuk Hur, Taemin Choi, Minsu Kim, Hyungjoo Ahn, Wonsok Yoo, Hyeonwoo Yu

arXiv:2610.21619v1cs.ROcs.CV

TL;DR

Distilled 3D language maps support arbitrary text queries, but often miss user-specified targets needed for reliable localization. The paper uses training-free, target-aware feature refinement with closed-form injection strengths and a centered localization read-out. It reports improved segmentation and goal localization across benchmarks, with plug-in transfer and a real-world robot demonstration.

  • Problem

    Distilled language maps can miss user-specified targets because context mixed into features makes target retrieval spatially diffuse, while open-vocabulary queryability must be retained.

    Source: A. Problem Setup · p. 2, §I · p. 2

  • Method

    The method blends detected target-region features toward target text anchors using the smallest winning coefficients in closed form, then applies a centered read-out for localization.

    Source: §I · p. 2

  • Results

    Across three benchmarks, refinement improves segmentation and localization; on Replica, SAM 2+SigLIP 2 mIoU rises from 0.207 to 0.436.

    Source: B. Open-vocabulary segmentation · p. 5, §V · p. 7

  • Takeaways & Limitations

    The refinement transfers as a plug-in to field-based and object-level maps and supports a real-world legged-robot demonstration navigating to natural-language goals.

    Source: §I · p. 2, §V · p. 7

Abstract

from arXiv · show

Distilling vision-language features into 3D Gaussian Splatting yields a language map that can be queried with arbitrary text. In deployment, however, a small set of user-specified targets must be reliably located, which current distilled maps often miss, while open-vocabulary queryability for other words must remain intact. We propose a training-free, target-aware refinement of an already-distilled language map. Guided by a text-promptable segmenter, the visual embedding inside each detected target region is blended with the target text embedding, a convex combination whose coefficient sets the amount of injected prior, while non-target regions are left untouched. Injecting too much overwrites appearance cues and general queryability, so we seek, for each target, the smallest coefficient that still lets it win, and we obtain every coefficient in closed form, leaving no hyperparameters to tune. We further identify a failure mode of injection in language-goal localization. Text embeddings share a common component, and injecting them makes the standard cosine read-out query-agnostic, which a centering step in the read-out removes. Across three benchmarks, the refinement consistently improves open-vocabulary segmentation, roughly doubling the base mIoU on Replica, and localizes language goals more reliably than the base map. It also transfers as a plug-in to off-the-shelf field-based and object-level language maps and supports a real-world demonstration in which a legged robot navigates to natural-language goals. Project page and code: https://lab-of-ai-and-robotics.github.io/target-aware-langmap/

I. INTRODUCTION

Open-vocabulary language maps support arbitrary text queries, but their target retrieval can be spatially diffuse and imprecise for navigation. The paper refines already-distilled maps with minimal target-specific injection and a centered read-out, reporting gains in segmentation and localization.

I. INTRODUCTION

  • Distilled open-vocabulary maps can produce diffuse relevancy whose peak is not precise enough for navigation, and errors can already appear in 2D features.The authors report that post-hoc sharpening is insufficient because top-ranked retrieval can be wrong before lifting to 3D.Source: §I · pp. 1–2
  • The training-free refinement blends target-region visual embeddings toward target text anchors, selecting the smallest winning coefficient in closed form while leaving non-target regions untouched.This minimal injection is intended to preserve the appearance cues and general queryability of the map.Source: §I · p. 2
  • A centered read-out removes the query-agnostic common component introduced by injected text embeddings, enabling more reliable language-goal localization.The refinement also transfers as a plug-in to dense feature fields and object-level maps.Source: §I · p. 2
  • The approach reports consistent gains in open-vocabulary segmentation and localization across three benchmarks, plug-in transfer, and a legged-robot demonstration.The paper describes a real-world demonstration in which a legged robot navigates to natural-language goals.Source: §I · p. 2

II. RELATED WORK

Prior open-vocabulary maps distill vision-language features into 3D, while language-goal systems commonly retrieve locations using raw cosine similarity. This work refines an existing map so target features win retrieval while preserving open-vocabulary features and producing navigation goals.

A. Open-vocabulary 3D language maps

  • Existing open-vocabulary 3D maps focus on building the map; the authors position training-free refinement of an already-built language field as complementary to them.The related work describes methods over point clouds, implicit fields, and 3D Gaussian Splatting.Source: A. Open-vocabulary 3D language maps · p. 2

B. Language-goal localization and navigation

  • Language-goal systems typically rank stored features by raw cosine similarity, which breaks after text embeddings are injected; centering restores and surpasses base retrieval.The cited task is turning a query word into a navigation goal.Source: B. Language-goal localization and… · p. 2

III. METHOD

  • The method refines per-view features of an existing 3D Gaussian language map, lifts them into 3D, and uses a centered read-out to return goal locations.The overview identifies target localization, target-text blending, and lifting onto frozen geometry as pipeline stages.Source: §III · p. 2, Figure 2 · p. 3

A. Problem Setup

  • Densely distilled features can rank surrounding context above a target; refinement seeks the smallest target-specific blend that makes the target win while leaving non-target features unchanged.Each target has its own coefficient, with the closed-form flip point and capture-curve knee setting the injection strength without a tuned hyperparameter.Source: A. Problem Setup · pp. 2–3

B. Target-Aware Feature Injection

The feature-injection stage builds per-view target-aware features from an open-vocabulary base, then blends detected target regions toward their text anchors.

B. Target-Aware Feature Injection

  • SAM 2 automatic masks and SigLIP 2 embeddings form the per-view open-vocabulary base, while SAM 3 localizes each target and the paired text encoder supplies its anchor.Regions outside target masks retain their visual embeddings; overlapping targets are resolved smaller-mask first.Source: B. Target-Aware Feature Injection · p. 3
  • Within each target region, the visual embedding is blended with its target text anchor as a convex combination.Non-target regions keep their visual embeddings.Source: B. Target-Aware Feature Injection · p. 3
  • The coefficient α ranges from pure visual embedding at 0 to a hard text paste at 1, and its value is selected next in closed form.The coefficient controls how much target-text prior is injected.Source: B. Target-Aware Feature Injection · p. 3

C. Closed-Form Minimal Injection

The method computes each target’s minimal winning injection strength per view, then selects a per-target coefficient from an area-weighted capture curve. The knee limits overwrite but may leave the hardest views unrecovered.

C. Closed-Form Minimal Injection

  • A_j is the smallest blend coefficient at which the target region beats every competing text anchor in a view.It is 0 when the region already wins and 1 when it never does; the calculation uses only a handful of dot products and adds no constants to tune.Source: C. Closed-Form Minimal Injection · p. 3
  • Because views flip at different strengths, the method aggregates their area-weighted capture into a non-decreasing curve and selects the coefficient at its knee.The knee is the point of maximum height above the chord joining the curve’s endpoints.Source: C. Closed-Form Minimal Injection · p. 3
  • The knee assumes no exchange rate between capture gain and injection cost, and may leave stubborn views unrecovered in exchange for minimal overwrite.After multi-view averaging, the captured majority typically suffices for map-level retrieval.Source: C. Closed-Form Minimal Injection · pp. 3–4
  • In Replica room0, capture curves assign larger coefficients to targets whose views flip late, such as switch, than to targets the base almost retrieves, such as plate.Each curve is a target’s capture curve, and its marked dot is the selected knee.Source: Figure 3 · p. 4
  • The selected coefficient stays small and adaptive, averaging around 0.1 to 0.2 and reaching exactly 0 for targets the base already retrieves.Per-view injection is performed before multi-view averaging, preserving adaptivity while allowing instance-level appearance to survive; hard pasting collapses it.Source: C. Closed-Form Minimal Injection · p. 4

D. Centered Localization Read-out

Injection adds a shared component to refined features, making raw-cosine rankings nearly query-agnostic. Centering the query removes this bias for goal localization while leaving the map and segmentation argmax unchanged.

D. Centered Localization Read-out

  • Injected features acquire a shared component because text anchors lie in a narrow cone, causing different queries to rank Gaussians in nearly the same order under raw cosine.The shared component dominates the query-specific score response.Source: D. Centered Localization Read-out · p. 4
  • Centering subtracts the mean anchor from the query, equivalently removing each Gaussian’s mean similarity over the vocabulary and leaving its query-specific response.The centered score is the query response minus the common mode.Source: D. Centered Localization Read-out · p. 4
  • Centering does not change the segmentation argmax because the subtracted term is shared across class anchors, but it matters for ranking Gaussians per query.The feed-forward step leaves the map unchanged, barely changes the base map, and uses a generic vocabulary that can be shared across scenes.Source: D. Centered Localization Read-out · p. 4
  • Goal extraction retains the top-p% Gaussians by centered score, clusters their positions, and ranks clusters by positive-score similarity mass.The top-K clusters yield similarity-weighted centroids as goal candidates; extraction constants are fixed across scenes and datasets.Source: D. Centered Localization Read-out · p. 4

A. Experimental Setup

The evaluation uses per-scene maps from Replica, ScanNet, and HM3D-Semantics to assess rendered-view open-vocabulary segmentation and language-goal localization. The implementation uses a common refinement pipeline with dataset-specific geometry settings and fixed read-out constants.

1) Datasets and metrics:

  • The study evaluates 8 Replica, 10 ScanNet, and 4 multi-room HM3D-Semantics scenes using rendered-view segmentation and language-goal localization protocols.Segmentation uses each scene’s annotated categories and reports mIoU and mAcc; HM3D-Semantics retains categories covering at least 0.5% of labeled pixels.Source: 1) Datasets and metrics · p. 4

2) Implementation details:

  • The pipeline builds a SAM 2 and SigLIP 2 base, applies the refinement, and lifts per-view features with a feature-splatting solver.Replica and ScanNet use RGB-only geometry, while HM3D-Semantics uses depth initialization; its language stages run on every fifth frame.Source: 2) Implementation details · pp. 4–5
  • Table I reports dataset means for zero-shot rendered-view open-vocabulary segmentation using mIoU and mAcc, with bold marking the best value per column.Source: Table I · p. 5
  • Read-out constants are fixed across scenes and datasets, with a 4% kept fraction for centered read-out and 0.5% for raw read-out.Both operate at their own operating point, using a 0.3 m cluster radius and K=5.Source: 2) Implementation details · p. 5

3) Baselines:

The refinement addresses context-mixed features that can rank a target incorrectly, injecting target text selectively while preserving open-vocabulary features. It improves segmentation on the base and transfers to off-the-shelf maps.

3) Baselines:

  • The baselines include field-based LangSplat and Feature-3DGS, with +Ours changing their per-view features using anchors from each baseline’s paired text encoder.Object-level HOV-SG and ConceptGraphs are also evaluated with refinement applied to their per-frame region features.Source: 3) Baselines · p. 5

B. Open-vocabulary segmentation

  • Entropy-based postprocessing degraded mIoU on two language maps because target regions could still rank the correct word too low for the region to win its query.The authors therefore inject target text directly and selectively into the 2D features.Source: B. Open-vocabulary segmentation · p. 5
  • The refinement raises the SAM 2+SigLIP 2 base from 0.207 to 0.436 mIoU on Replica.Most gains recover classes the base misses, while already-retrieved classes are preserved; classes not detected by SAM 3 remain limited by segmenter recall.Source: B. Open-vocabulary segmentation · p. 5
  • Table II reports injection-strength ablations on all three benchmarks and gives mean α* in its last row for Replica, ScanNet, and HM3D.The supplied caption does not state the ablation values.Source: Table II · p. 5
  • On LangSplat, refinement raises mIoU by 79% to 123% across three datasets; Feature-3DGS gains 23% on Replica, 34% on HM3D-Sem, and 4.0 points on ScanNet.Feature-3DGS improves on all twelve HM3D-Sem scenes, and injection strength correlates −0.80 with base mIoU across scenes.Source: B. Open-vocabulary segmentation · p. 5

C. Why minimal injection

Minimal adaptive injection retains appearance and held-out performance better than hard or fixed injection. For language-goal localization, centering corrects injection’s query-agnostic bias, and the refined map improves aggregate success over the base.

C. Why minimal injection

  • The hard paste achieves the highest closed-set mIoU, but adaptive injection reaches 0.436 on Replica, within 8% of hard paste and above fixed α values of comparable mean strength.The text cautions that closed-set mIoU rewards pasting the scored word, even if appearance cues are lost.Source: C. Why minimal injection · p. 5
  • On held-out classes, fixed α degrades accuracy by up to 26% on Replica, 16% on ScanNet, and 28% on HM3D-Sem at α=1; adaptive injection limits losses to 8%, 8%, and 14%, respectively.The held-out evaluation reports mAcc, with classes split evenly between injected targets and held-out classes.Source: Table III · p. 6, C. Why minimal injection · p. 6
  • For room0 cushions, hard paste gives the per-instance features mean pairwise cosine 0.999, while adaptive injection preserves appearance diversity at 0.883 versus the base’s 0.873.The adaptive map ranks the two blue cushions top-2, unlike hard paste, which ranks neither first.Source: C. Why minimal injection · pp. 5–6

D. Language-goal localization

  • 64.0 vs 50.3 at top-1: the centered refined map beats the base on Replica and aggregate localization, and exceeds the centering-matched aggregate base by 10.3 points.On ScanNet, it trails by at most one query at each K; at a 2 m radius, top-1 is 76.0 vs 62.9.Source: Table IV · p. 6, D. Language-goal localization · p. 7
  • The standard cosine read-out collapses after refinement: ScanNet top-1 falls from 72.9 to 30.0, as cross-query correlation rises from 0.28 to 0.98.Centering removes the shared common mode and is decisive for the refined map, while helping the base only modestly.Source: D. Language-goal localization · pp. 6–7, Table IV · p. 6, Figure 6 · p. 7
  • Object-level HOV-SG and ConceptGraphs are strong on ScanNet furniture queries but struggle with small cluttered Replica targets; refinement raises their aggregate top-1 to 52.0 and 46.3, respectively.Replica does not improve because editing existing regions cannot create missing small-object instances; the dense field representation provides those instances.Source: Table V · p. 7, D. Language-goal localization · p. 7
  • On HM3D-Sem, centered top-1 success rises from 24.5% on the base to 32.7% on the refined map; without centering it falls to 10.2%.Fixed α=1 reaches 38.8 with centering, but the paper notes that this setting degrades held-out performance and instance appearance.Source: Table III · p. 6, C. Why minimal injection · p. 6, D. Language-goal localization · p. 7
  • The refined map localizes the goal that guides navigation, and the agent reaches the queried object along its planned path.The figure presents first-person view, a 3DGS render, and a top-down map from left to right.Source: Figure 5 · p. 6, D. Language-goal localization · p. 7

E. Real-world demonstration

A real-world deployment uses the refined language map to convert natural-language requests into robot navigation goals. The robot succeeds more often with the refined map than with the base map.

E. Real-world demonstration

  • The robot reaches natural-language goals with 78% success using the refined map, versus 22% with the base map and raw-similarity read-out.A language-model front end selects a target word, the centered read-out supplies its 3D goal, and the robot plans with A* on a LiDAR-map occupancy grid.Source: E. Real-world demonstration · p. 7

V. CONCLUSIONS

The paper presents a training-free target-aware refinement that preserves open-vocabulary querying while improving target retrieval. It reports gains in segmentation and localization across benchmarks and a real-world robot demonstration.

V. CONCLUSIONS

  • The refinement obtains each target’s injection strength in closed form and uses centered read-out to remove injection’s query-agnostic common mode.The per-view-feature approach is presented as a plug-in for existing language-map pipelines; online mapping and sparsely observed targets remain future directions.Source: §V · p. 7
Loading 2610.21619v1…