PaperlayerView exact revision on arXiv

Source-linked AI summary

$S^3$-Nav: Structured Spatial-Semantic Decision Making for Language-Guided Robot Navigation

Chaemin Lee, Hae Min Cho

arXiv:2610.19431v2cs.RO

TL;DR

Language-guided navigation under partial observability must reason over spatial-semantic evidence accumulated across observations, especially for spatial-relation goals. S3-Nav maintains persistent object–relation memory and exposes a capacity-controlled, goal-conditioned policy view, with independent validation of provisional decisions. In the evaluated Qwen setting, K14 maintained or improved navigation success relative to Full while reducing token consumption.

  • Problem

    Language-guided navigation must use evidence accumulated across partial observations, and relational goals require distinguishing a plausible target from one that satisfies the requested relation.

    Source: §I · p. 1

  • Method

    S3-Nav preserves object–relation memory while selecting a capacity-controlled, goal-conditioned policy view and independently validating provisional decisions against memory and geometric evidence.

    Source: §I · pp. 1–2

  • Results

    For the evaluated Qwen policy, K14 maintained or improved navigation success relative to Full while reducing token consumption; on HSSD it reduced Qwen Tokens/Ep. by 22.0% relative to Full.

    Source: §I · p. 10, §VI · p. 11

  • Takeaways & Limitations

    Capacity-controlled evidence exposure can reduce language-model token consumption while preserving accumulated memory for subsequent grounding and verification.

    Source: §I · p. 10, §VI · p. 11

    Evaluation is limited to simulation, a single Qwen model, and a small set of navigation tasks, leaving broader generalization and real-world transfer for future work.

    Source: §V · p. 11

Abstract

from arXiv · show

Language-guided robot navigation under partial observability requires robots to reason over spatial-semantic evidence accumulated across observations, particularly when goals involve spatial relations between objects. This paper presents $S^3$-Nav, a structured spatial-semantic decision-making framework that integrates online mapping, persistent object--relation memory, and goal-conditioned navigation. Using RGB-D observations and SLAM, $S^3$-Nav maintains a persistent object--relation memory containing object records and confirmed relation records alongside an evolving navigation context containing exploration frontiers, without requiring a pre-existing map. A goal-conditioned evidence interface selects task-relevant evidence from this structured state and constructs a capacity-controlled decision view as part of the high-level policy input for exploration, target approach, and stopping decisions. The persistent memory remains available for subsequent reasoning and verification even when some of its records are excluded from the decision view. To ensure grounded execution, provisional policy decisions are independently validated against persistent object--relation memory and current geometric evidence before physical execution or task termination. Evaluations across multiple environments and object- and relation-goal tasks show improved closed-loop navigation reliability and reduced language-model token consumption with capacity-controlled evidence exposure under the evaluated inference configuration.

I. INTRODUCTION

Language-guided navigation must preserve evidence across partial observations while distinguishing likely targets from objects that satisfy a requested relation. S3-Nav combines persistent object–relation memory with a capacity-controlled, goal-conditioned policy view and independent execution validation.

I. INTRODUCTION

  • Relational instructions require preserving evidence across observations and distinguishing target or approach evidence from evidence sufficient to authorize task completion.A detected chair alone does not establish that it satisfies “next to a plant”; the robot also needs geometric evidence for an appropriate stopping position.Source: §I · p. 1
  • S3-Nav maintains a structured state from RGB-D observations and SLAM, separating persistent object–relation memory from evolving navigation context.A goal-conditioned interface selects and ranks relevant records for a capacity-controlled policy view without pruning persistent memory.Source: §I · p. 1
  • An independent execution-validation mechanism applies distinct evidence requirements to target approach and task completion.Relational stopping requires confirmed goal-matching relations and recent target-associated range evidence, regardless of decision-view capacity.Source: §I · p. 2
  • The evaluation examines high-level policy selection, decision-view capacity, persistent memory, and frontier records across HSSD and InternScenes.It measures closed-loop navigation performance and reasoning cost.Source: §I · p. 2

A. Spatial-Semantic Mapping and Memory

  • Persistent object records retain map-frame positions across observations, allowing spatial predicates to be evaluated for pairs seen at different times.The accumulated evidence may remain useful even after objects leave the current camera view.Source: §I · p. 1, A. Spatial-Semantic Mapping and Memory · p. 2

B. Object-Goal and Language-Guided Navigation

Prior navigation approaches support semantic target selection and grounding, but relational instructions require verifying that a target satisfies the requested predicate. S3-Nav connects structured evidence, high-level policy decisions, and independently validated execution.

B. Object-Goal and Language-Guided Navigation

  • S3-Nav distinguishes plausible targets from relation-goal matches by allowing object records and transient cues to guide navigation while requiring confirmed relations for satisfaction.The related-work discussion contrasts this requirement with approaches centered on semantic target selection and grounding.Source: B. Object-Goal and Language-Guided… · p. 2

C. Grounded Language-Model Reasoning

  • High-level policy outputs remain provisional because the execution layer independently resolves grounded references and determines whether termination can be accepted.Separating planning from execution alone does not specify when task completion should be accepted.Source: C. Grounded Language-Model Reasoning · p. 2

III. METHOD

  • The architecture accumulates spatial-semantic evidence, selects goal-relevant information for policy decisions, and independently validates proposed navigation intents.This separation preserves accumulated evidence for subsequent reasoning while distinguishing navigation guidance from evidence authorizing completion.Source: §III · p. 2

A. Structured Spatial-Semantic Decision Architecture

  • Online RGB-D perception updates pose, partial occupancy mapping, and semantic observations used to construct the structured decision state.The state combines persistent object–relation memory with current navigation context, including exploration frontiers; the evidence interface supplies policy input for a proposed intent.Source: A. Structured Spatial-Semantic… · p. 2
  • Execution validation evaluates proposed intents using structured state and current geometric evidence, independently of which records the capacity-controlled view exposes.The policy input also includes the language instruction, navigation phase, and relation-goal progress summary.Source: A. Structured Spatial-Semantic… · pp. 2–3
  • The policy’s outputs include an executable motion command and an accepted terminal event, with no terminal event when termination is not accepted.The closed-loop procedure formalizes the sequence of state update, policy invocation, and validation.Source: A. Structured Spatial-Semantic… · p. 3

B. Online Localization, Mapping, and Frontier Construction

S3-Nav builds geometric context through online localization and occupancy mapping without requiring a pre-existing map. It uses frontier candidates to represent potential exploration destinations within its closed-loop procedure.

B. Online Localization, Mapping, and Frontier Construction

  • Online localization and occupancy mapping provide robot pose and an evolving partial map as a common spatial reference for object grounding and navigation.Boundaries between observed free space and unexplored regions yield frontier candidates for exploration, not semantic task-completion targets.Source: B. Online Localization, Mapping, and… · p. 3
  • The map summary tracks explored-space ratio, frontier count, and accumulated path length, while the full occupancy map remains available to the navigation stack.The full map is not included in the policy input.Source: B. Online Localization, Mapping, and… · p. 3
  • At each decision step, the procedure updates perception, localization, mapping, frontier candidates, persistent memory, and map summary before constructing the structured state.It then constructs a candidate sequence and capacity-limited decision view before policy invocation and execution validation.Source: B. Online Localization, Mapping, and… · p. 3
  • The closed-loop procedure executes the validated motion command and ends the loop when an accepted terminal event is returned.The algorithm description identifies the overall procedure as S3-Nav’s closed-loop decision process.Source: B. Online Localization, Mapping, and… · p. 3

C. Structured Spatial-Semantic Decision State

The structured state separates persistent spatial-semantic evidence from changing navigation context. Object and confirmed relation records provide distinct kinds of evidence for search, approach, and relation-goal satisfaction.

C. Structured Spatial-Semantic Decision State

  • The goal schema specifies the target category, requested predicate, reference category, approach-anchor category, and whether relational verification is required.The state also separates persistent memory from the robot’s evolving pose, frontiers, and map progress.Source: C. Structured Spatial-Semantic… · pp. 3–4
  • Persistent object records retain identity, category, map-frame position, confidence, and observation history, allowing objects to remain approach candidates after leaving view.Grounded positions come from valid aligned RGB-D depth, and observations without reliable depth are not admitted as object anchors.Source: C. Structured Spatial-Semantic… · p. 4
  • Confirmed relation records link persistent object identities through a spatial predicate that has passed geometric verification.Object co-occurrence alone cannot confirm a relation; stale relations can expire while their constituent object records remain available.Source: C. Structured Spatial-Semantic… · p. 4
  • For “find a chair next to a plant,” chair and plant records can guide search or approach, but only a geometrically verified pair supplies a confirmed goal relation.The confirmed relation retains the chair as the approach and stopping anchor.Source: C. Structured Spatial-Semantic… · p. 4

D. Decision View Construction and High-Level Policy

The interface ranks goal-relevant object, relation, and frontier evidence into a capacity-controlled view for high-level navigation decisions. Policy outputs refer to structured records, while transient relation cues support approach without being treated as confirmed evidence for completion.

D. Decision View Construction and High-Level Policy

  • Transient relation cues let the policy approach possible object relationships while remaining distinct from confirmed relations and insufficient to establish task completion.For relation-goals, cues retain source-object references so execution can resolve a physical anchor without promoting the cue into persistent confirmed-relation memory.Source: D. Decision View Construction and… · p. 4
  • Candidate ranking prioritizes goal-schema records, relevant object and relation evidence, then frontiers and remaining context, using fixed heuristic parameters rather than learned or optimized scores.The staged limits allow up to two goal-schema records, up to ten goal-related records, and up to five frontier records before filling remaining positions.Source: D. Decision View Construction and… · p. 4
  • For each structured state, finite-capacity views retain an order-preserving prefix of the ranked candidates while leaving persistent memory intact.The capacity setting determines how many records are visible to the policy, not whether underlying evidence is discarded.Source: D. Decision View Construction and… · p. 4
  • The policy input includes navigation phase and relation-search summary to distinguish exploration, target approach, and provisional stopping candidates without authorizing termination.The relation-search summary tracks confirmed matches and previously inspected unconfirmed targets.Source: D. Decision View Construction and… · pp. 4–5
  • The interface connects ranked object, relation, and frontier records to policy outputs through structured-record references that support execution validation.The figure depicts a capacity-controlled decision view, with the selected references linking provisional decisions to validation.Source: Figure 3 · p. 5, D. Decision View Construction and… · p. 5

E. Decision Validation and Execution Authority

The execution layer independently resolves provisional policy references against structured state and current geometric and safety evidence. Task termination requires both an eligible semantic target and recent target-associated range evidence.

E. Decision Validation and Execution Authority

  • Termination requires a goal-compatible stopping reference and recent target-associated range evidence; a remembered target position alone cannot establish that the robot is currently within stopping range.For relation-goals, the persistent memory must contain a confirmed matching relation supporting the physical target; the evaluated range is 0.5–0.9 m.Source: E. Decision Validation and Execution… · pp. 5–6
  • The execution layer resolves references for exploration or approach against current geometry and applies safety constraints before motion; safety overrides do not make tentative evidence admissible for stopping.Depth-based checks may hold, reduce, or steer motion, or trigger bounded recovery.Source: E. Decision Validation and Execution… · pp. 5–6
  • A relation-goal stopping reference is semantically eligible only when persistent relation memory contains a confirmed goal-matching relation supporting its resolved physical target.Semantic eligibility does not independently authorize termination.Source: E. Decision Validation and Execution… · p. 5
  • Terminal range uses target-associated depth observations refreshed no earlier than the immediately preceding memory update, prioritizing the tenth-percentile horizontal distance of valid target-mask points.For approach, remembered map-grounded geometry provides a fallback when a valid terminal-evaluation range is unavailable.Source: E. Decision Validation and Execution… · pp. 5–6
  • The same terminal condition can authorize termination during deterministic standoff control after an approach proposal, which logs distinguish from a policy-requested stop.A policy-generated stop is accepted only when the terminal gate is satisfied.Source: E. Decision Validation and Execution… · p. 6

IV. EXPERIMENTS

The experiments evaluate object- and relation-goal navigation in household environments using a shared mapping, memory, and execution stack. The capacity comparison measures record limits under the implemented inference interface, not record count in isolation.

IV. EXPERIMENTS

  • Primary experiments use indoor HSSD environments in Isaac Sim, with three additional InternScenes tasks examining whether policy and capacity patterns hold across environments.Both evaluations use an RGB-D Ranger Mini and the same perception, mapping, persistent memory, and deterministic execution procedures unless stated otherwise.Source: A. Experimental Setup · p. 6
  • The language-model policy uses Qwen3-4B-Thinking-2507 with a 6,000-token request limit, and long prompts may trigger a concise-input adaptation or context overflow.Full and K14 can differ in whether the prompt adaptation is triggered.Source: A. Experimental Setup · p. 6
  • Capacity comparisons assess the implemented inference interface rather than isolating record count, and K14 was selected before sensitivity analysis without being assumed optimal.The comparison includes Heuristic Full, Heuristic K14, Qwen Full, and Qwen K14 under estimated localization; the localization comparison changes Qwen K14 to simulator ground-truth odometry.Source: A. Experimental Setup · p. 6

B. Tasks and Evaluation Protocol

The protocol tests object- and relation-goal tasks across initial-visibility conditions and repeated runs. Its primary metric checks ground-truth semantic, relational, and stopping-range correctness, alongside measures that separately characterize relation satisfaction, range error, and reasoning demand.

B. Tasks and Evaluation Protocol

  • The task set covers object-goals and relation-goals, with the target serving as the physical approach and stopping anchor for relation-goals.Five HSSD scenarios and three InternScenes tasks are run five times per primary condition, with matched initial poses, instructions, and designated instances across corresponding runs.Source: B. Tasks and Evaluation Protocol · p. 6
  • Seen relation-goal cases require both target and reference objects to be initially observable; paired HSSD cases distinguish immediately accessible evidence from exploration and memory needs.Initial visibility is defined by camera view, sensing range, and unobstructed line of sight, independently of detection success.Source: B. Tasks and Evaluation Protocol · pp. 6–7

C. Evaluation Metrics

  • GT-SSR is the principal task-level metric because controller agreement on stopping evidence does not necessarily establish ground-truth correctness.It evaluates ground-truth target identity, relation when applicable, and geometry; both SSR and GT-SSR include unsuccessful attempts and timeouts, with invalid ground-truth terminal geometry counted as failure for GT-SSR.Source: C. Evaluation Metrics · p. 7
  • RSR isolates predicate satisfaction among relation-goal episodes with valid relation verdicts, so a high score need not mean successful navigation or correct standoff.SNE and GT-SNE measure absolute standoff error only when measurements are valid; trajectory length, duration, policy calls, token use, and latency measure additional navigation and reasoning outcomes.Source: C. Evaluation Metrics · p. 7
  • Surface-range MAE and RMSE compare sensor-derived target ranges with ground-truth ranges at eligible observations along condition-specific trajectories.Sensor range uses the tenth percentile of valid target-mask depth-point distances; ground-truth range is the minimum horizontal distance to the target bounding box.Source: C. Evaluation Metrics · p. 7

D. Comparison with Heuristic Policy

Under the evaluated conditions, Qwen K14 achieved higher GT-SSR than Qwen Full on initially unseen tasks, while the effect of capacity varied by policy and task. In the two tested relation-goal tasks, capacity and GT-SSR had a nonmonotonic relationship, and the results do not establish that smaller views are always preferable.

D. Comparison with Heuristic Policy

  • Qwen K14 achieved higher GT-SSR than Qwen Full on all initially unseen tasks, with the largest improvements on HSSD S4 and S5 and InternScenes I3; initially seen tasks were unchanged.The heuristic showed smaller or absent improvements on unseen tasks and a gain on an initially seen task.Source: D. Comparison with Heuristic Policy · p. 7, Table II · p. 8
  • Qwen selects a navigation mode and record reference from policy input, while the heuristic selects the same fields through fixed rules over the selected evidence.Both policies share evidence construction and execution validation, so the comparison concerns their decision rules; the heuristic is an internal alternative, not a previously published navigation baseline.Source: D. Comparison with Heuristic Policy · p. 7

E. Record-Capacity Sensitivity

  • The Full–K14 comparison does not establish that smaller views are always preferable: finite prefixes may omit useful candidates as well as irrelevant records.The sensitivity study tested capacities 3, 7, 14, and Full on HSSD S5 and InternScenes I3, using the same retained memory, ranking, and execution validation.Source: E. Record-Capacity Sensitivity · pp. 7–8
  • Across the two tested relation-goal tasks, GT-SSR varied nonmonotonically with visible record count; K14 alone achieved GT-SSR of 1.0 on both.Smaller prefixes could exclude useful alternatives, while Full could include less relevant evidence; Fig. 4 reports the capacity sensitivity for HSSD S5 and InternScenes I3.Source: E. Record-Capacity Sensitivity · p. 8, Figure 4 · p. 9
  • Eight Qwen Full episodes terminated after context-length overflow, while K14 had no such failures, preventing attribution of Full–K14 differences solely to record selection or improved reasoning.Length-triggered prompt adaptation also complicates attribution.Source: E. Record-Capacity Sensitivity · p. 9

F. Localization and Surface-Range Diagnostics

The diagnostics distinguish pose accuracy from target-associated geometric evidence, while the qualitative sequence and ablation show how accumulated relation evidence guides navigation and supports completion. In the evaluated unseen relation-goal task, cross-step memory is especially important, though the component comparison is limited in scope.

F. Localization and Surface-Range Diagnostics

  • Improved pose accuracy does not consistently correspond to better navigation success or lower target-surface range error, and the trajectory-specific comparisons do not isolate causality.Pose accuracy and target-associated geometric measurements capture different aspects of navigation reliability.Source: F. Localization and Surface-Range… · p. 9

G. Qualitative Navigation Decision Analysis

  • After geometric verification confirms the chair–plant relation, the policy shifts from frontier-guided exploration to target-directed approach; completion additionally requires recent target-associated range evidence.The sequence depicts goal interpretation, exploration, target approach, and terminal validation.Source: G. Qualitative Navigation Decision… · p. 9

H. Component Ablation

  • The ablation compares cross-step object–relation persistence and explicit frontier records using Qwen K14 under estimated localization in one unseen HSSD relation-goal scenario with a distractor.Without explicit frontier records, map and exploration summaries remain; all conditions retain the same navigation and terminal-validation procedures.Source: H. Component Ablation · p. 9
  • Cross-step memory is particularly important in the evaluated unseen relation-goal task: confirmed relation evidence can disappear before the robot reaches an admissible stopping position.An inspected trace shows that one-step relation confirmation is insufficient when the robot still needs to approach the target.Source: H. Component Ablation · p. 9
  • Removing explicit frontier records yields a smaller GT-SSR decline than removing persistent memory, but this single-task comparison does not establish a general component ranking or isolate the execution layer.The execution layer is unchanged across the ablation conditions.Source: H. Component Ablation · p. 9

I. Runtime Analysis

Under the implemented inference interface, K14 reduces Qwen token consumption relative to Full, but the reduction does not improve simulated throughput as measured by RTF. The conclusion reports maintained or improved navigation success for the evaluated Qwen policy, while broader generalization remains untested beyond simulation, one model, and a small task set.

I. Runtime Analysis

  • Fig. 5 traces a relation-goal task through goal interpretation, frontier-guided exploration, relation confirmation, target approach, and terminal validation.Its panels show decisions, RGB-D observations, 3D reconstructions, and partial occupancy maps, with the executed trajectory in the final map.Source: Figure 5 · p. 10
  • Table V compares reasoning cost and runtime across policy, decision-view capacity, and pose-source conditions in HSSD and InternScenes.HSSD has 25 episodes per condition except 24 valid heuristic K14 runtime records; InternScenes has 15 per condition.Source: §I · p. 9, Table V · p. 10
  • Policy Latency includes the high-level policy routine, whereas E2E Decision Latency also includes perception, mapping, memory updates, and deterministic execution; GT Odometry rows are excluded from the Full–K14 capacity comparison.Tokens/Ep. includes prompt and generated completion tokens, while Qwen calls and tokens exclude cached goal-schema parsing.Source: §I · p. 10
  • Under estimated localization on HSSD, K14 reduces Qwen Tokens/Ep. by 22.0% relative to Full, while mean Policy Latency and E2E Decision Latency decrease by only 3.2% and 2.7%, respectively.InternScenes shows the same qualitative separation between token consumption and per-decision latency; the token reduction does not translate into improved simulated throughput measured by RTF.Source: §I · p. 10, Table V · p. 10
  • Token consumption also depends on trajectory decision counts and length-triggered prompt adaptation, so the observations do not isolate prefix truncation as the sole efficiency source.The heuristic is an alternative decision rule using the same structured evidence, not a competing external navigation baseline.Source: §I · p. 11

V. DISCUSSION AND LIMITATIONS

  • The evaluation is limited to simulation, a single Qwen model, and a small set of navigation tasks, leaving broader generalization and real-world transfer for future work.The component ablation covers one unseen relation-goal scenario, and the execution layer's individual contribution is not isolated because it is shared across conditions.Source: §V · p. 11

VI. CONCLUSION

  • Across HSSD and InternScenes, K14 maintained or improved navigation success relative to Full for the evaluated Qwen policy while reducing language-model token consumption.The conclusion also reports that losing previously accumulated relational evidence can prevent successful task completion.Source: §VI · p. 11

Related papers

Loading 2610.19431v2…