PaperlayerView exact revision on arXiv

Source-linked AI summary

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Haoxiang Shi, Zaijing Li, Muhe Ding, Xiang Deng, Yaowei Wang, Liqiang Nie

arXiv:2609.39915v1cs.CVcs.RO

TL;DR

Long-horizon vision-language navigation must keep local actions consistent with route instructions while managing growing interaction histories. NavHarness links adaptive goals, observed-outcome verification, and progress-aware multimodal memory; it reports results on two VLN-CE benchmarks and in real-world routes.

  • Problem

    In long-horizon navigation, locally plausible actions may not remain consistent with the intended route, while accumulated interaction history increases the input required for later decisions.

    Source: §1 · p. 2

  • Method

    NavHarness uses adaptive local goals and goal-specific verification, with verified completion defining boundaries for multimodal history compression that preserves information needed for subsequent navigation.

    Source: §1 · pp. 2–3

  • Results

    79.0% SR on R2R-CE and 82.5% SR on RxR-CE compare with same-backbone Codex CLI results of 74.0% and 76.3%, respectively; real-world evaluation reports 83.3% SR and 1.51 m navigation error.

    Source: §1 · p. 3

  • Takeaways & Limitations

    NavHarness organizes navigation around observed task progress, using it to guide the next objective and identify completed history that can be summarized while retaining the global instruction.

    Source: §5 · p. 9

    Remote closed-source models incur inference costs and communication latency, while a single RGB camera limits visual coverage and can leave obstacles outside the field of view unobserved.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. Then the Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3\% success and 1.51\,m navigation error across eight challenging routes evaluated three times each.

1 INTRODUCTION

NavHarness addresses the difficulty of keeping local actions aligned with route instructions while managing growing interaction histories. It combines adaptive goals and observed-outcome verification with progress-aware multimodal memory, and reports results on VLN-CE and in real-world navigation.

Motivation

  • Locally plausible actions may diverge from the intended route over long horizons, while accumulating interaction history raises inference overhead and indiscriminate truncation can discard useful information.The lost information may include landmarks, corrections, or unfinished objectives needed for navigation and recovery.Source: §1 · p. 2

Approach

  • The Goal Agent derives local goals from the instruction, current observation, and execution history, while the Verify Agent checks observed outcomes against goal-specific completion conditions.Verification feedback guides continued execution, goal advancement, or goal revision for recovery.Source: §1 · pp. 2–3
  • Verified goal completion defines memory-compression boundaries, allowing completed interactions to be summarized while retaining context and unfinished task state for subsequent navigation.The memory mechanism preserves route progress, relevant landmarks, corrections, and evidence for the active goal.Source: §1 · p. 3

Results

  • 79.0% SR on R2R-CE and 82.5% SR on RxR-CE exceed same-backbone Codex CLI results of 74.0% and 76.3%; real-world evaluation reaches 83.3% SR and 1.51 m navigation error.The real-world results cover eight challenging routes, each evaluated three times.Source: §1 · p. 3
  • The authors report that NavHarness outperforms navigation foundation models trained on large-scale datasets in zero-shot evaluation on VLN-CE and in the real world.The introduction also reports evaluation on R2R-CE and RxR-CE and comparison of framework variants across three backbones.Source: §1 · p. 3

2 RELATED WORK

Related work covers navigation policies and agents, progress assessment and recovery, and memory and context management. NavHarness combines adaptive goals and verification with goal-completion-based compression of completed interactions.

2.1 FROM NAVIGATION POLICIES TO AGENTS.

  • Navigation policies learn instruction grounding and action selection, while language-model agents reason over observations and feedback and use tool calls to interact with the environment.NavHarness uses adaptive goals for execution and verified completion for goal advancement and memory compression.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 PROGRESS ASSESSMENT AND RECOVERY.

  • Prior work uses progress estimates, backtracking, state reasoning, or waypoint-based recovery; NavHarness makes each adaptive goal’s completion condition explicit through a verification question.The resulting judgment guides goal continuation, revision, or advancement and identifies completed interaction for compression.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3 MEMORY AND CONTEXT MANAGEMENT.

  • Navigation systems use temporal context, structured memories, and language summaries, while NavHarness ties compression to verified completion and preserves current observations and unresolved route information.Completed interactions are summarized while information needed for ongoing navigation remains available.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 NAVHARNESS

NavHarness organizes instruction-following around adaptive local goals with observable completion conditions, linking action execution, verification, and memory compression.

3.1 TASK FORMULATION AND FRAMEWORK OVERVIEW

  • NavHarness links local goals, action execution, outcome verification, and history compression; verified completion sets the boundary for compressing interaction history.The framework retains active-goal evidence and unfinished task state for subsequent decisions.Source: §3.1 · p. 4, §3.5 · pp. 5–6

3.2 GOAL AGENT: ADAPTIVE LOCAL GOALS

  • The Goal Agent sets one local outcome and a question specifying supporting visual evidence or spatial relations; it revises goals when the scene no longer fits, without completing unresolved stages.The original instruction constrains revisions, and unfinished state carries forward.Source: §3.2 · pp. 4–5

3.3 VERIFY AGENT: GOAL-SPECIFIC COMPLETION VERIFICATION

  • After each interaction, the Verify Agent checks observed outcomes against the goal-specific question, reports supporting evidence and unmet or uncertain conditions, and guides continuation, revision, advancement, or stopping.Verified completion also triggers memory compression; final-goal verification informs the stop decision.Source: §3.3 · p. 5

3.4 VISUOMOTOR AGENT: GOAL-CONDITIONED EXECUTION

  • The Visuomotor Agent uses observations, historical context, and verification feedback to act toward the goal, which can remain stable across interactions as movements adapt to new evidence.An interaction may execute a short movement sequence before visual reassessment.Source: §3.4 · p. 5

3.5 MEMORY AGENT: PROGRESS-AWARE MULTIMODAL COMPRESSION

  • At verified-completion boundaries, the Memory Agent condenses completed history into route-and-landmark summaries, verification outcomes, and visual keyframes while retaining active-goal evidence and unfinished state.The updated memory replaces the detailed completed segment and supports subsequent goal generation, execution, and verification.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

Experiments compare NavHarness with matched-backbone baselines, test memory compression and framework variants, and evaluate real-world routes. Results report navigation performance, context costs, and illustrative indoor and outdoor behavior.

4.1 EXPERIMENTAL SETUP

  • The benchmarks use 100 episodes each, matched backbones and interfaces, and report navigation error, success, path efficiency, route agreement, and context consumption.R2R-CE uses the Open-Nav subset; RxR-CE generally has longer routes and more detailed instructions.Source: §4.1 · p. 6, §4.2 · p. 7, §4.3 · p. 7

4.2 NAVIGATION PERFORMANCE

  • GPT-6-Astra NavHarness reaches 79.0% SR on R2R-CE and 82.5% SR on RxR-CE, improving over Codex CLI by 5.0 and 6.2 points, respectively.GPT-5.6-Sol gains 4.4 and 6.0 SR points on the same benchmarks.Source: §4.2 · p. 7

4.3 NAVIGATION QUALITY AND CONTEXT EFFICIENCY

  • Goal compression reduces total tokens in all four settings, but savings and navigation trade-offs vary by backbone.GPT-5.6-Sol saves 35.94% on R2R-CE and 56.59% on RxR-CE; GPT-6-Astra saves 5.52% and 4.05%, respectively.Source: Figure 3 · p. 7, §4.3 · p. 8

4.4 FRAMEWORK ABLATION ACROSS BACKBONES

  • Without compression, R2R-CE SR rises from 28%, 63%, and 74% to 32%, 67%, and 79% across the three backbones; compression preserves higher SR than baselines.The comparison does not isolate goal generation from verification.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 REAL-WORLD NAVIGATION

  • 83.3% SR and 1.51 m NE on real-world routes improve over JanusVLN by 33.3 SR points and 0.83 m NE.NavHarness was evaluated on eight routes, with three trials per route.Source: Table 3 · p. 8, §4.5 · p. 8
  • Indoor route stages distinguish clearing the doorway before turning from approaching the chair after it is seen.The route follows corridor and lounge landmarks toward a yellow chair with a white ball.Source: §4.5 · p. 8, Figure 5 · p. 9
  • Outdoor cases show obstacle-responsive adjustment toward a sculpture and landmark-conditioned progress toward trees.For the bicycle route, the robot approaches the landmark before turning and continues through the corridor toward the trees.Source: §4.5 · p. 9, Figure 6 · p. 9

5 LIMITATIONS AND CONCLUSION

NavHarness connects adaptive goals, observed completion checks, and context management while retaining the global instruction. Its current deployment has inference-cost, latency, and visual-coverage constraints.

5 LIMITATIONS AND CONCLUSION

  • NavHarness uses observed task progress to guide the next objective and identify completed history for summarization.The organization retains the global instruction while allowing execution to respond to the encountered environment.Source: §5 · p. 9
  • Verified completion links adaptive goals, action, and context management in the framework.The global instruction remains in context as execution responds to the environment.Source: §5 · p. 9
  • Remote closed-source models incur inference costs and communication latency, while a single RGB camera can leave off-camera obstacles unobserved and lead to unsafe movement.Future work proposes lightweight local VLMs and depth sensing or panoramic observations.Source: §5 · p. 10

A EXTENDED RELATED WORK

Related work covers learned navigation policies, language-model agents, progress monitoring, and context management. NavHarness is positioned as connecting adaptive goals, observable completion, and context boundaries.

A EXTENDED RELATED WORK

B EVALUATION PROTOCOLS

The evaluation protocols distinguish controlled NavHarness comparisons from published results with differing configurations and cohorts. They specify the observations available to the agent and the action interface used in simulation.

B EVALUATION PROTOCOLS

  • The simulation agent receives forward-facing RGB observations and can issue forward, left, right, or STOP primitives, with execution returning an observation and action record.A decision may request a short sequence of primitives.Source: B EVALUATION PROTOCOLS · p. 15
  • The model does not receive the evaluator’s reference path, success label, or destination distance, and only selected archived images are attached to requests.A textual reference to a historical frame does not mean that image is visible to the model.Source: B EVALUATION PROTOCOLS · p. 15
  • The Codex CLI and NavHarness comparisons match backbones, episode lists, and observation/action interfaces; Table 1 disables compression, while Table 2 also evaluates it enabled.Codex CLI retains native context management.Source: B EVALUATION PROTOCOLS · p. 15
  • Published results span different training configurations, visual inputs, executors, and episode cohorts, so the table separates them from the controlled comparison.A full validation split, fixed subset, and previously released-pool subset are distinct cohorts even under the same benchmark name.Source: B EVALUATION PROTOCOLS · p. 15
  • Published configurations include differing views and training setups; NaVid’s RxR figures are transfer results from R2R training, not RxR-trained results.SmartWay/GPT-4o is the original setup, while SmartWay/GPT-5.5 is a reproduction; external results provide context rather than a matched ranking.Source: B EVALUATION PROTOCOLS · p. 15

C AGENT FORMULATION, COORDINATION, AND PROMPT TEMPLATES

NavHarness coordinates goal generation, action execution, verification, and memory updates through an interaction loop. Its update rules preserve active goals and unresolved evidence while using verified completion to advance and compress history.

C AGENT FORMULATION, COORDINATION, AND PROMPT TEMPLATES

  • Algorithm 1 orders goal generation, action execution, verification, memory update, and routing, specifying when history is appended, goals persist or change, and memory is replaced.The prompt templates summarize agent instructions; the algorithm specifies invocation order and shared-state updates.Source: C AGENT FORMULATION, COORDINATION, AND… · pp. 15–16

C.1 INITIALIZING THE NAVIGATION EPISODE

  • Episode initialization uses the first observation to generate an initial goal and verification question, while ongoing episodes retain their active goal and supplied view.State separates persistent memory from recent records; the instruction remains fixed and available to every agent.Source: §C.1 · p. 16

C.2 GENERATING THE NEXT GOAL

  • Goal generation uses the route instruction, current observation, and execution history to set the next pursuable goal and pair it with a question testing completion and expected landmarks or spatial relations.Expected observations are conditions to check, not established facts; the final goal checks arrival without requiring STOP already issued.Source: §C.2 · p. 17
  • A goal and its question persist unless coordination requests an update; completion advances using updated memory, while revising an incomplete goal retains its detailed evidence.Feedback about the previous goal is reset for the replacement but remains in recorded or summarized history.Source: §C.2 · p. 17

C.3 CHECKING OBSERVED COMPLETION

  • Verification assesses the instruction, goal, question, observations, and execution history; it checks performed actions and the observed result rather than treating intended actions as proof of success.The judgment is appended after the executed actions and resulting observation, separating evidence from the feedback it generates.Source: §C.3 · pp. 17–18
  • Feedback supports continuation, goal revision, or advancement after verified completion, while termination and the instructed stopping condition take priority over another goal update.Only completion triggers compression; uncertainty that supports neither completion nor revision leaves the goal active.Source: §C.3 · p. 18

C.4 EXECUTING THE ACTIVE GOAL

  • The Visuomotor Agent acts toward the active goal, but the realized action record and observation—not the requested batch—establish what occurred; STOP has no mechanical verification gate.The environment may end a batch early, and its primitive-action budget is distinct from the external-interaction bound.Source: §C.4 · pp. 18–19

C.5 SUMMARIZING COMPLETED NAVIGATION

  • The Memory Agent compresses completed interaction records into route, landmark, spatial-relation, and verification summaries while carrying unresolved conditions forward; revising an incomplete goal does not compress it.The summarizer preserves relevant uncertainty and distinguishes observations and executed actions from interpretations.Source: §C.5 · pp. 19–20

D METRIC DEFINITIONS AND CONTEXT ACCOUNTING

This section defines navigation and context-accounting measures, including what token and input averages count. It also distinguishes episode-level savings from local input changes during execution.

D METRIC DEFINITIONS AND CONTEXT ACCOUNTING

  • Navigation error measures final distance to the goal, success rate measures successful termination, and SPL combines success with path efficiency.OSR measures reaching the success region at any point; nDTW measures agreement with the reference route.Source: D METRIC DEFINITIONS AND CONTEXT… · p. 20
  • Episode token totals aggregate recorded input, cached input, and output usage across goal, verification, execution, and memory calls, including retries.Table 2 reports mean tokens per episode in thousands; cached input is included once in input totals, and averages exclude incomplete usage records.Source: D METRIC DEFINITIONS AND CONTEXT… · pp. 20–21
  • Image/Dec. counts input image instances per external decision, while Text/Dec. measures input text characters in thousands, including repeated history and internal-agent inputs.Text/Dec. is not token usage, and image counts do not specify visual token consumption.Source: D METRIC DEFINITIONS AND CONTEXT… · p. 21
  • The Goal-to-Off total-token ratio compares mean episode consumption with compression enabled versus disabled; Figure 3 instead shows local input-token changes at selected compression events.Values below 100% indicate lower consumption with compression, and local decreases are not aggregate savings across events.Source: D METRIC DEFINITIONS AND CONTEXT… · p. 21

E REAL-WORLD EVALUATION PROTOCOL

The real-world protocol tests NavHarness on eight challenging routes, with three trials per route, using a camera-equipped Unitree Go2 platform.

E REAL-WORLD EVALUATION PROTOCOL

  • Eight routes of approximately 15–20 m span corridors, sofa areas, classrooms, stairs, laboratories, and outdoor spaces; each is tested three times for 24 trials.Table 3 reports aggregate success rate and mean navigation error across routes.Source: E REAL-WORLD EVALUATION PROTOCOL · p. 21
  • The evaluation uses a Unitree Go2 measuring approximately 70 × 40 cm with a front-mounted Intel RealSense D435i camera about 70 cm above ground.The camera supplies 640 × 480 RGB at configured 30 FPS; depth, infrared, and camera IMU streams are disabled.Source: E REAL-WORLD EVALUATION PROTOCOL · p. 21
  • The platform communicates through ROS 2 Foxy with CycloneDDS, and the aggregate results retain the recorded baseline values.Source: E REAL-WORLD EVALUATION PROTOCOL · p. 21

F ADDITIONAL REAL-WORLD CASES

The additional cases show route transitions, local corrections, and final approaches across six egocentric observations per figure. Their annotations are explanatory reconstructions, and the sequences do not establish individual component effects.

F ADDITIONAL REAL-WORLD CASES

  • Each figure presents six egocentric observations illustrating route transitions, local corrections, and the final approach.Goal and verification annotations are scene-aligned explanatory reconstructions, not verbatim logged outputs.Source: F ADDITIONAL REAL-WORLD CASES · p. 21
  • The sequences show observed behavior but do not independently establish the causal contribution of any individual framework component.Colored outlines are editorial annotations.Source: F ADDITIONAL REAL-WORLD CASES · p. 21
  • One route links a doorway, yellow chair, white table, and black table through landmark-conditioned turns before the agent calls STOP.The final approach has a clear view.Source: Figure 7 · p. 22
  • Another route follows table landmarks, approaches a doorway, makes local corrections, and enters the room before stopping.A further case aligns through a narrow opening formed by door leaves before locating the instructed white chair.Source: Figure 8 · p. 22, Figure 9 · p. 22
  • A stepped-doorway case adapts its local objective to the seating arrangement before approaching the dark green sofa.Panel (h) is the final observation before the recorded stop action.Source: Figure 10 · p. 22

Related papers

Loading 2609.39915v1…