OASIS: Online Anchor-centric Scene Inference System for Open-Vocabulary Retrieval and Navigation in Dynamic Environments

1 Introduction

• Dense semantic maps can add clutter and computational and memory costs, while lightweight maps may discard context needed to distinguish similar objects.Repeated-observation relocalization can also miss shifted targets during rapid motion or with a limited camera view.Source: §1 · pp. 1–2
• OASIS organizes salient descriptions and representative frames around stable semantic anchors, using LLM filtering and on-demand VLM verification for fine-grained retrieval.This anchor-centric strategy avoids processing redundant context globally while retaining visual attributes.Source: §1 · p. 2, Figure 1 · p. 2
• A lightweight buffer of transient navigation observations lets OASIS query and re-verify candidate views when a target is missing from its initial anchor.The system uses these observations to relocalize moved objects.Source: §1 · p. 2
• OASIS unifies online mapping, open-vocabulary retrieval, and navigation, with experiments in simulation and real-world static and dynamic settings demonstrating retrieval and relocation performance.Its multimodal scene graph integrates semantic, visual, and geometric representations and supports hierarchical retrieval and view-driven navigation.Source: §1 · p. 2

2.1 Open-Vocabulary 3D Scene Understanding

• Dense open-vocabulary 3D maps retain pixel-aligned features for fine-grained queries but cost more computation and memory; lightweight maps are efficient but can lose surrounding context [6] [18] [28] [9] [15].OASIS adds implicit storage to lightweight representations to retain contextual details without dense fields.Source: §2.1 · p. 2

2.2 Retrieval and Reasoning on Scene Graphs

• CLIP matching is efficient for simple identification, LLM methods support reasoning from textual abstractions, and VLM methods retain visual fidelity but can be costly and slow [6] [19] [26] [27] [7] [13] [28] [31] [18] [35].OASIS uses VLM-driven reasoning on implicit storage to avoid full-image processing overhead.Source: §2.2 · p. 2

2.3 Object Navigation

• Dynamic navigation methods often update maps through fine-grained modeling and geometric matching, but transient observations make small objects difficult to update and displaced targets hard to infer [9] [27].OASIS uses visual details observed along the path to infer a displaced object's location, potentially beyond the immediate trajectory.Source: §2.3 · p. 3

3.1 Overview

• OASIS fuses stable anchors, representative frames, structural relations, and navigable poses into a multimodal graph for retrieval, verification, navigation, and relocation.Online semantic mapping and episodic-memory curation run concurrently; the graph serves as a multimodal index rather than an exhaustive set of static relationship labels.Source: §3.1 · p. 3, §3.3 · p. 4

3.2 Episodic Memory Curation

• LLM-selected anchor categories emphasize volumetric prominence, stationarity, and associative utility, while frame scoring balances global scene richness against local anchor quality.Spatial-visual clusters retain a highest-scoring representative frame; the adaptive weighting emphasizes local fidelity for frames dominated by large anchors and global context for smaller or more balanced anchors.Source: §3.2 · p. 3

3.3 Anchor-Centric Scene Graph Construction

• Stable instances persisting across consecutive frames become anchors, which are associated with representative views and linked to supported objects and nearby co-occurrences.Geometric edges represent objects supported by anchors; visual edges connect representative frames, with co-occurring stable instances abstracted as nearby.Source: §3.3 · p. 4

3.4 Hierarchical Object Retrieval

• Retrieval first uses an LLM to rank likely anchors, then a VLM to verify targets across shared views and CLIP to resolve the matching 3D instance.If candidate-anchor views fail, a global CLIP search checks remaining scene instances.Source: §3.4 · p. 5

3.5 Navigation and Relocation

• The robot navigates to a selected stored viewpoint and verifies the target; if it has moved, a transient buffer supports visual matching before semantic anchor-based relocation.During navigation, the buffer retains the highest-confidence frame for each unmatched transient track; CLIP selects up to five crop candidates for VLM verification, while semantic relocation checks another likely anchor if visual search finds no match.Source: §3.5 · pp. 5–6

4.1 Overview

• The evaluation asks whether OASIS handles complex retrieval, relocates moved objects, and benefits from its components, while preserving real-time online efficiency.Implementation uses GPT-5.2 for LLM planning and Gemini 3 Pro for VLM verification.Source: §4.1 · p. 6

4.2.1 Experimental Setup.

• Evaluation spans OpenLex3D fine-grained object retrieval, HOV-SG functional-region retrieval, Sr3D+ spatial relations, and HM3D navigation.The navigation protocol tests Static, In-Anchor Dynamic, and Cross-Anchor Dynamic settings; dynamic success requires finding the target within three attempts.Source: §4.2.1 · p. 6, Table 2 · p. 6, §4.3.1 · p. 7

4.2.2 Results and Analysis.

• 6.58 mAP makes OASIS the best Fine-grained Object Retrieval method; it excels on Easy and View-Independent spatial-relation splits but trails CORE-3D on Hard queries.Hard queries often span distant anchors, where anchor-centered representative frames are less effective; functional regions are inferred from nearby objects and background-layout verification.Source: §4.2.2 · p. 6, Table 2 · p. 6, §4.2.1 · p. 6

4.3.2 Results and Analysis.

• 85.9% average Success Rate (SR) in Static navigation, 81.5% for In-Anchor moves, and 77.4% for Cross-Anchor moves; OASIS outperforms all baselines across settings.In Cross-Anchor navigation, OASIS reaches 77.4% SR versus DualMap’s 60.3%; transient observations support visual relocalization of displaced targets.Source: §4.3.2 · p. 7, Table 4 · p. 7, §4.3.1 · p. 7

4.4.1 Memory Curation Strategy.

• 91.13% coverage comes from Adaptive frame selection, while Hybrid clustering achieves over 89% coverage across selection methods with moderate cluster counts.Visual-Only clustering yields fewer clusters but lower coverage because global CLIP features neglect spatial structure; Spatial-Only cannot distinguish semantically different nearby frames.Source: §4.4.1 · p. 7, Table 5 · p. 7

4.4.2 Hierarchical Retrieval Pipeline.

• The full hierarchical retrieval pipeline achieves the best accuracy–efficiency trade-off, while removing either stage weakens performance.Without LLM Proposal, global matching and unpruned anchor verification increase latency; without VLM Verification, CLIP loses fine-grained ambiguity and spatial-relation precision.Source: §4.4.2 · pp. 7–8, Table 6 · p. 8

4.4.3 Dynamic Navigation and Relocation.

• 77.4% SR with 1.6 average attempts is the full framework’s best Cross-Anchor result, compared with 17.0% SR for Random Anchor and 56.6% without the visual buffer.Without semantic reasoning, performance is 73.6% SR and random fallback is inefficient.Source: §4.4.3 · p. 8, Table 7 · p. 8

4.5 Real-World Experiments

• Across 60 real-world trials, OASIS outperforms DualMap in success rate and SPL while requiring fewer relocation attempts; four static and seven dynamic trials fail.Failures include attribute confusions, missed small objects, local-anchor limits on distant-room searches, and one missed transient detection.Source: §4.5 · p. 8, Table 8 · p. 8, Figure 4 · p. 8

• OASIS unifies online mapping, open-vocabulary retrieval, and navigation in dynamic environments.Source: §5 · p. 8
• Its multimodal memory and hierarchical LLM–VLM pipeline balance semantic expressiveness with online efficiency.Transient observations collected during navigation support moved-object relocalization.Source: §5 · p. 8
• Simulation and real-world experiments report improvements over baselines in retrieval accuracy, navigation success, and relocalization robustness.Source: §5 · p. 8

A Supplementary Overview

• The supplement adds details on anchor-category prompting, frame scoring, memory and runtime overhead, retrieval settings, pipeline latency, and workflow prompts.Source: A Supplementary Overview · p. 10

B Anchor Definition Prompt

• An LLM selects anchor categories from DualMap’s predefined common indoor categories rather than relying on manual selection.Source: B Anchor Definition Prompt · p. 10
• The prompt favors large, stationary cognitive landmarks and excludes background surfaces, small movable furniture, and categories outside the input list.The resulting examples include beds, tables, sofas, and refrigerators.Source: B Anchor Definition Prompt · p. 10

C Detailed Formulation of Frame Scoring

• The formulation describes scale-aware frame scoring, combining global scene richness with local anchor quality.Source: C Detailed Formulation of Frame Scoring · p. 10

C.1 Scale-Aware Adaptive Weights

• Anchor-area entropy measures balance: higher entropy means multiple anchors contribute more evenly.It is computed from the normalized distribution of anchor pixel areas.Source: §C.1 · p. 10
• The anchor coverage ratio captures the absolute visual footprint of anchors as a fraction of image area.Source: §C.1 · p. 10
• Adaptive weights use coverage and area balance; λ balances those cues, and when M≤1 the weighting uses coverage alone.The local weight is the complement of the global weight.Source: §C.1 · p. 10

C.2 Global and Local Terms

• The raw global score combines clipped anchor count and object count, then is normalized.Source: §C.2 · p. 10
• Local anchor quality combines edge completeness with clipped interaction count.Source: §C.2 · p. 10
• Local values are aggregated using both the maximum and mean across anchors, then normalized to [0, 1].The stated maximum possible anchor value supports this normalization.Source: §C.2 · p. 10

C.3 Hyperparameters and Sensitivity Analysis

• Table S1 lists the frame-scoring hyperparameters used as the defaults in the experiments.Source: §C.3 · p. 10, Table S1 · p. 11
• The one-factor-at-a-time analysis varies λ, α, β, and γ while fixing other parameters at their defaults, using scene instance coverage.Source: §C.3 · p. 10
• Coverage curves vary only slightly across λ, α, β, and γ, and the defaults remain within strong-performing regions.The adopted settings are λ=0.7, α=0.7, β=0.5, and γ=0.7.Source: §C.3 · p. 11, Figure S1 · p. 11

D Mapping Overhead Analysis

• OASIS adds only a marginal increase in memory and time overhead over DualMap while retaining practical real-time performance and far greater efficiency than dense methods.Measurements cover online mapping on one NVIDIA RTX 4090 GPU and exclude retrieval-time LLM/VLM calls.Source: D Mapping Overhead Analysis · p. 11, Table S2 · p. 11
• The comparison reports average memory, peak memory, and time per RGB-D frame for ConceptGraphs, HOV-SG, DualMap, and OASIS.TPF denotes average processing time per RGB-D frame.Source: D Mapping Overhead Analysis · p. 11, Table S2 · p. 11
• The additional overhead comes from a parallel episodic memory curation module built on top of DualMap.Source: D Mapping Overhead Analysis · p. 11

E Default Top-𝐾Settings

• These Top-K values remain fixed throughout experiments unless otherwise specified and were selected from preliminary development experience.The stated trade-off is among robustness, memory usage, and inference latency.Source: E Default Top-𝐾Settings · p. 11
• The LLM anchor proposal stage keeps the top-3 candidate anchors for verification, balancing search-space reduction against retrieval robustness and VLM-call latency.Too few anchors may discard plausible target regions; too many increase VLM calls and latency.Source: E Default Top-𝐾Settings · p. 11
• Each anchor retains at most 5 representative frames ranked by frame score to preserve diverse, high-quality views without excessive redundancy.A small set is described as sufficient for common viewpoint variations around stable indoor anchors.Source: E Default Top-𝐾Settings · p. 11
• During dynamic relocalization, OASIS keeps the top-5 TVB frames by CLIP similarity for VLM verification, limiting verification cost while retaining candidate diversity.Source: E Default Top-𝐾Settings · p. 11

F.1 Prompt Example Overview

• The workflow proceeds from LLM anchor proposal to fine-grained VLM verification, disappearance detection, TVB candidate search, and final same-instance verification at the relocated position.The final pass checks whether the current view matches the retrieved candidate and confirms the target as the same instance.Source: §F.1 · pp. 11–12
• The LLM ranks candidate anchors from the query and structured metadata, while the VLM checks representative anchor frames and returns verification status and target attributes.The LLM prompt constrains output format and anchor-ID validity for deterministic downstream parsing.Source: §F.1 · pp. 11–12
• The authors characterize cloud-based multimodal reasoning latency as a deployment artifact and anticipate onboard migration as edge models improve.Source: §F.1 · p. 12
• If the target is absent after navigation and camera rotation, the system compares candidate TVB images with the target reference image to identify a relocated candidate.Source: §F.1 · p. 12

F.2 Latency Analysis

• Cloud API inference and network I/O dominate end-to-end latency across the five stages, whereas local graph traversal, node filtering, and TVB updates execute in milliseconds.The measured interactions use GPT-5.2 and Gemini-3-Pro.Source: Table S3 · p. 12, §F.2 · p. 12
• API delays occur during semantic reasoning and verification, while the local navigation stack continues motion and processes passive TVB observations without heavy VLM calls.Source: §F.2 · p. 12

G Detailed Prompt Example

• The detailed prompt example covers five deployment stages—anchor proposal, anchor verification, post-navigation existence checking, TVB re-identification, and same-instance verification—extending the paper's three retrieval stages.Source: G Detailed Prompt Example · p. 13, Figure S2 · p. 13, Figure S3 · p. 14