PaperlayerView exact revision on arXiv

Source-linked AI summary

PreAct-Nav: Agentic Reasoning Before Action for Urban Navigation

Jing Xie, Shouwei Ruan, Yubin Wang, Yuxiang Zhang, Junwei Yang, Songchang Jin, Dianxi Shi

arXiv:2610.04916v1cs.ROcs.AI

TL;DR

Urban navigation requires local actions to support distant goals, yet history and language-based future thoughts provide limited guidance and visual evidence. PreAct-Nav guides frozen policies with medium-horizon subgoals, action-conditioned visual predictions, and feedback-updated memory. On UrbanNav, it improves success rate and path-length-weighted success over the original policy, with larger prediction gains on longer routes and routes with more turns.

  • Problem

    History and language-based future thoughts provide limited guidance and visual evidence for determining whether locally plausible actions will support progress toward a distant destination.

    Source: §1 · pp. 1–2

  • Method

    PreAct-Nav equips frozen policies with a VLM reasoner, medium-horizon subgoals, action-conditioned visual predictions, and memory updated from execution feedback.

    Source: §1 · pp. 2–3

  • Results

    On UrbanNav, PreAct-Nav improves success rate by 6.71% and success weighted by path length by 4.19% relative to the benchmark’s original policy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Prediction gains are larger on longer routes and routes with more turns.

    Source: §1 · p. 2

    Visual predictions can be ambiguous or inconsistent, and video generation dominates inference cost, limiting extension of the loop to time-sensitive navigation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Urban navigation requires embodied agents to pursue long-horizon goals through local decisions based on egocentric observations. However, existing agentic navigation methods often struggle to translate distant goals into coherent local decisions in large-scale physical environments. Their reliance on linguistic reasoning over transient observations or limited history constrains anticipation of the consequences of actions and future conditions, despite the importance of such foresight for navigating long and complex urban routes. To bridge this gap, we propose PreAct-Nav, an agentic navigation framework that equips frozen policies with anticipatory reasoning for robust urban navigation. Our central idea is to anchor local decisions in persistent medium-horizon subgoals, assess the consequences of predicted actions before execution, and continually update the reasoning context using actual outcomes. At its core, a navigation memory module maintains the active subgoal and relevant experience across decisions, translating distant goals into actionable intermediate objectives. We further introduce a predictive world sandbox that uses an action-conditioned world model (AC-WM) to forecast world dynamics conditioned on candidate movements. A vision-language model (VLM) reasoner interprets these predictions under the current subgoal to retain or revise actions. After execution, real observations are used to assess outcomes, correct inconsistent assumptions, and update memory to continue or reformulate the subgoal. Extensive evaluations demonstrate that the proposed PreAct-Nav improves action selection through memory updates and visual prediction, with more pronounced gains on longer routes and routes with more turns.

1 INTRODUCTION

Urban navigation requires local movements to remain useful for distant goals, but history and language-based future thoughts provide limited guidance and visual evidence. PreAct-Nav uses medium-horizon subgoals, predicted visual outcomes, and execution feedback to guide action review and memory updates.

1 INTRODUCTION

  • Local movements can be feasible yet fail to support a distant destination, while observed history and language-based future thoughts provide limited guidance and visual evidence.The challenge is connecting intermediate steps across intersections and turns to long-horizon goals while anticipating how actions affect subsequent progress.Source: §1 · pp. 1–2
  • PreAct-Nav is a training-free harness that equips frozen policies with a VLM reasoner to assess predicted visual outcomes before execution and update navigation memory afterward.The reasoner uses the current subgoal and relevant experience to retain or revise a policy action without updating model weights.Source: §1 · p. 2
  • PreAct-Nav improves success rate by 6.71% and success weighted by path length by 4.19% relative to UrbanNav’s original policy.Evaluations use three navigation policies and two VLM reasoners; prediction gains are larger on longer routes and routes with more turns.Source: §1 · p. 2
  • Navigation Memory maintains medium-horizon subgoals and observed experience, then updates subgoal progress using execution feedback.These subgoals decompose long-horizon tasks into objectives that guide local action review and revision.Source: §1 · p. 2
  • The Predictive Sandbox exposes an action-conditioned world model to provide future visual cues for reasoning under the current subgoal.The reasoner uses those cues to assess whether a proposed movement advances the subgoal and to retain or revise the action.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers navigation reasoning, agent memory, and action-conditioned world models. PreAct-Nav’s distinction is coordinating memory and predicted visual cues through a frozen reasoner that reviews and revises movements before execution.

Navigation reasoning

  • Navigation reasoning ranges from learned local-movement policies to explicit instruction reasoning, multi-expert assessment, and retrieved experience with future thoughts.Prior examples include LM-Nav, which composes pretrained models without fine-tuning [Shah et al., 2023], and ReAct and Inner Monologue, which interleave reasoning with feedback [Yao et al., 2023; Huang et al., 2023].Source: §2 · p. 3

Agent memory

  • Agent memory systems reuse stored experience, while PreAct-Nav’s Navigation Memory couples experience with an active medium-horizon subgoal that guides action review and revision.Execution feedback updates the objective and progress cues so memory tracks both prior outcomes and the current navigation purpose.Source: §2 · p. 3

Action-conditioned world models

  • Action-conditioned world models predict observations for specified movements, but PreAct-Nav focuses on how a VLM interprets their visual cues against the current subgoal.Real observations subsequently establish progress and update memory, connecting predictive reasoning with execution feedback.Source: §2 · p. 3

3 METHODOLOGY

PreAct-Nav organizes navigation around medium-horizon subgoals and observed experience, using predicted visual outcomes to review and revise actions before execution. Real execution feedback then updates memory for subsequent decisions.

3 METHODOLOGY

  • PreAct-Nav is a training-free harness for a frozen navigation policy, coordinating a shared VLM reasoner with Navigation Memory and the Predictive Sandbox.The shared VLM is implemented through structured prompts and tool calls, with model weights frozen.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM FORMULATION

  • The reasoner receives the destination, current and recent observations, navigation memory, and policy action; observations and memory remain fixed until execution advances the step.The policy action is designated as the initial action, with successive revisions retained in the reasoning history.Source: §3.1 · p. 4

3.2 NAVIGATION MEMORY

  • Navigation Memory maintains a medium-horizon subgoal and observed experience, with subgoal fields for objective, visual cues, open question, and status.Execution feedback compares real observations with expected cues, updates progress, and determines whether the active objective continues or closes.Source: §3.2 · pp. 4–5

3.3 PREDICTIVE SANDBOX

  • The Predictive Sandbox uses the current observation and proposed action to generate a short predicted rollout for review under the current subgoal.The reasoner assesses prediction quality, links cues to the action, and specifies expected future cues to check after execution; inconclusive predictions defer to observed evidence.Source: §3.3 · p. 5

3.4 AGENTIC REASONING HARNESS

  • The harness previews actions, requests prediction when evidence is unresolved, reviews the result, revises contradicted actions, executes a supported action, and updates memory from real feedback.If no action is supported, the harness retains the policy action.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate PreAct-Nav across navigation tasks, frozen policies, and VLM reasoners, then examine component contributions and route-level behavior. Results show gains on UrbanNav and CityWalker, including larger prediction gains on longer and more turning-intensive routes.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers UrbanNav seen and unseen splits and CityWalker routes, reporting completion, efficiency, goal-distance, path-alignment, and goal-region metrics.UrbanNav has 4,179 test-seen and 4,634 test-unseen episodes; CityWalker uses 154 routes from nine recordings.Source: §4.1 · p. 6, §4 · p. 6

4.2 MAIN RESULTS

  • On UrbanNav test-unseen, Qwen raises SR from 78.14% to 83.38% and SPL from 74.73% to 77.87%, while reducing NE from 5.9659 to 5.3353 m.Both splits improve on all six metrics; the test-unseen SR gain over Open-Nav is 4.14 points.Source: §4.2 · p. 7, Table 1 · p. 7
  • With GPT-5.6 Sol, PreAct-Nav reaches 84.18% SR on test-unseen; on CityWalker, Qwen reaches 79.22% SR and 78.94% SPL.The CityWalker results exceed NavGPT by 5.84 SR points and 8.74 SPL points.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 POLICY TRANSFER

  • On test-unseen, NoMaD gains 2.78 SR points and OmniVLA gains 3.58 points; OmniVLA improves on all six metrics.The NoMaD result primarily reflects better completion and path efficiency; the harness improves navigation without changing policy weights.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 ABLATION STUDY

  • The full method achieves the highest ablation SR at 83.38%, the lowest NE, and the highest nDTW, SDTW, and OSR.Its 77.87% SPL exceeds the 77.24% obtained with the sandbox alone.Source: §4.4 · p. 8, Table 4 · p. 9
  • The full method improves SR and reduces NE in every route-length and turning interval, with larger prediction gains on longer and more turning-intensive routes.SR gain reaches 9.60 points on 30–50 m routes and 7.92 points at 360–540 degrees of cumulative turning; curves are point estimates.Source: §4.4 · p. 8, Figure 3 · p. 9

4.5 QUALITATIVE RESULTS

  • In one example, predicted cues motivate a storefront-to-sidewalk action revision, and the next real observation supports the expected passage.When pedestrians remain in the passage, memory updates the objective to passing on the right while retaining an open question about available space.Source: §4.5 · pp. 8–9, Figure 4 · p. 9
  • PreAct-Nav is also illustrated in an outdoor-plaza real-robot case linking action review and revision with observation-based memory updates.The example is reported in Appendix B.Source: §4.5 · p. 9

5 CONCLUSION AND LIMITATIONS

PreAct-Nav combines medium-horizon subgoals, action-conditioned visual prediction, and execution feedback to review navigation actions before execution. Its robot demonstration illustrates subgoal completion and transition to a new objective, while prediction quality and cost remain limitations.

5 CONCLUSION AND LIMITATIONS

  • Experiments across frozen policies show improved navigation success, with greater prediction gains on longer routes and routes with more turns.The harness maintains medium-horizon subgoals, uses action-conditioned future visual cues, and updates memory from real feedback.Source: §5 · p. 10
  • Visual prediction quality and cost constrain the harness; generated videos can be ambiguous or inconsistent, and video generation dominates inference cost.More reliable, faster predictions are needed to extend the loop to time-sensitive navigation.Source: §5 · p. 10

A NAVIGATION WORKFLOW

  • The workflow updates experience from observations, establishes or reviews a subgoal, previews and revises actions when needed, then executes a selected movement.After execution, the next observation is compared with expected cues and memory is updated.Source: A NAVIGATION WORKFLOW · p. 13

B REAL-WORLD NAVIGATION DEMONSTRATION

  • In a wheeled-robot run, the harness revised a doorway-approaching proposal into a rightward facade route, then marked the exterior-walkway subgoal complete.The next objective was to approach the locker along a clear paved route; predicted rollouts were generated online before execution, while current and next images were real camera observations.Source: B REAL-WORLD NAVIGATION DEMONSTRATION · p. 13

C FULL PROMPT TEMPLATES

Together, the prompt templates define how the reasoning loop establishes subgoals, reviews and revises movements, and updates memory from execution feedback, with structured formats that distinguish predicted evidence from real observations. The memory-update prompt applies this process to following an exterior walkway by checking a proposed movement against the subgoal and using expected and observed cues to revise the action while preserving the objective.

4 Memory Update

  • The rollout prompt asks whether the proposed movement remains outside while following the exterior walkway.Source: §4 · p. 14
  • The executed-action prompt records rightward travel along the facade and expects the entrance to shift left.Source: §4 · p. 14
  • When a contradiction is identified, the prompt revises the movement while preserving the medium-horizon objective.Source: §4 · p. 14
  • The example revised action is to turn right and follow the facade, expecting the entrance to shift left and the exterior walkway to come into view.Source: §4 · p. 14
  • The revised action is reviewed before execution, and the prompt pairs the current observation with the next observation.Source: §4 · p. 14
  • Figure 5 links subgoals, visual previews, action revision, and memory update in a real-world robot route along a building exterior toward a delivery locker.Source: Figure 5 · p. 14

C.1 MESSAGE CONSTRUCTION

  • Each operation packages textual instructions and structured context with chronological real images; predicted frames follow real images, and memory updates use before-and-after images.Shared context fixes observations, experience, and subgoal within a decision while action and review history may change.Source: §C.1 · p. 14

C.2 NAVIGATION MEMORY: ESTABLISHING A SUBGOAL

  • Subgoal establishment returns a route intent, visible anchor, intermediate objective, completion cue, and open question, while the harness maintains status and progress.The compact memory groups intent and objective as Objective, and anchor and completion as Visual cues; creation step and accumulated progress are maintained outside the response.Source: §C.2 · p. 16

C.3 ACTION PREVIEW: OBSERVATION-BASED REVIEW

  • Observation-based review assesses the current movement against real observations and memory, requesting a prediction only for a specific unresolved question that could benefit from it.Missing evidence alone yields UNRESOLVED, and an unseen destination alone does not justify rejecting local movement.Source: §C.3 · pp. 16–17

C.4 ACTION REVIEW: PREDICTED VISUAL EVIDENCE

  • Predicted review evaluates the same movement with forecast frames, distinguishing the action verdict from video quality and treating predictions as fallible rather than factual evidence.A coherent video need not support the action, while an inconclusive video does not itself contradict it.Source: §C.4 · pp. 17–18

C.5 ACTION REVIEW: CONSTRUCTING A REVISION

  • When a contradicted review and remaining budget allow, revision changes the local movement while preserving the subgoal and grounding the new path in currently visible walkable space.The harness validates revisions and returns valid new movements to Action Preview; ABSTAIN requires an empty path.Source: §C.5 · pp. 18–19

C.6 MEMORY UPDATE: EXECUTION FEEDBACK

  • After execution, memory update compares two real images with the movement, subgoal, and expected cues; every real transition is stored with feedback.SUPPORTED indicates compatible progress rather than completion, and ACHIEVED requires the completion cue to be visible.Source: §C.6 · pp. 19–20

C.7 PREDICTIVE SANDBOX: WORLD-MODEL REQUEST

  • The world-model request conditions scene continuation on the current image and proposed movement, while the VLM uses the subgoal and open question to interpret the returned video.The world model receives no destination description, reference future, success label, or subgoal as generation text.Source: §C.7 · p. 20

C.8 CORRESPONDENCE WITH THE METHOD AND FIGURES

  • The same templates support the illustrated parked-vehicle and pedestrian transitions, with case-specific context and responses filling rather than changing the protocol.The examples show prediction-informed revision in one case and contradictory real feedback closing a subgoal before a new one is established in the other.Source: §C.8 · p. 20

D IMPLEMENTATION DETAILS

Implementation uses frozen pretrained models and compact subgoal memory, validates revised waypoint actions, and conditions predicted video rollouts on the proposed movement.

D.1 MODELS AND CONTEXT

  • The default system pairs the UrbanNav policy with Qwen3.8-27B and Cosmos3-Nano; pretrained models receive no weight updates during evaluation.Subgoal state groups route intent and intermediate objective with visual anchors and progress or completion evidence; the implementation retains four recent experience summaries and refreshes an active subgoal after six decisions if unresolved.Source: §D.1 · pp. 20–21

D.2 ACTION REPRESENTATION

  • Revised actions encode waypoint displacements as ranges and bearings, then undergo format, value, confidence, and repetition checks before use.The conversion uses x_k = r_k sin β_k and y_k = r_k cos β_k; invalid or low-confidence revisions fall back to the policy proposal.Source: §D.2 · p. 21

D.3 PREDICTIVE SANDBOX

  • Each sandbox query conditions video generation on the current image, a camera trajectory derived from the proposed movement, and a shared scene-continuation prompt.It returns 60 predicted frames in addition to the input frame, and the query record links each rollout to its conditioning action.Source: §D.3 · p. 22

D.4 URBANNAV EVALUATION

UrbanNav evaluation uses seen and unseen test episodes with egocentric observations and local-waypoint actions.

D.4 URBANNAV EVALUATION

  • The test splits contain 4,179 test-seen and 4,634 test-unseen episodes.Each agent receives a destination description and egocentric observations and produces local waypoints.Source: §D.4 · p. 22
  • Main comparisons allow at most 25 navigation decisions per episode and count success within a three-metre radius.Internal reasoning iterations and tool calls do not count as additional navigation decisions.Source: §D.4 · p. 22
  • Each agent receives the destination description and egocentric observations, then produces a sequence of local waypoints.Source: §D.4 · p. 22

D.5 CITYWALKER EVALUATION

The CityWalker evaluation uses 154 route windows from nine recordings, with fixed endpoints expressed relative to the agent and a 100-decision limit.

D.5 CITYWALKER EVALUATION

  • 154 route windows from nine recordings use fixed route endpoints expressed as rightward and forward displacements in the current frame.The evaluated policy is frozen CityWalker.Source: §D.5 · p. 22
  • Each decision executes at most two metres along the proposed path, and success requires reaching within three metres of the goal.Source: §D.5 · p. 22
  • The frozen CityWalker policy receives five chronological RGB observations and coordinate context; stationary transitions count toward the 100-decision budget.Baselines use task-specific prompt adaptations.Source: §D.5 · p. 22

E METRICS AND COMPLETE RESULTS

This section defines navigation measures and describes full-split reporting and supplementary route-geometry analyses, including results grouped by route length and turning.

E METRICS AND COMPLETE RESULTS

  • SR, SPL, NE, and OSR are standard navigation measures; SPL uses executed waypoint-path length against reference-route length, and NE is measured in metres.OSR indicates whether any executed waypoint enters the three-metre success radius; off-route termination counts as failure before aggregation.Source: E METRICS AND COMPLETE RESULTS · pp. 22–23

E.1 FULL-SPLIT REPORTING

  • Table 1 separates test-seen from test-unseen, while Tables 3 and 4 report test-unseen results; each row is a complete evaluation on its split.Tables 1–4 report six metrics for the main comparison, policy transfer, and component ablation.Source: §E.1 · p. 23

F SUPPLEMENTARY ANALYSES

  • Figure 3 partitions all 4,634 episodes by reference length and cumulative turning, with SR gains measured against the original policy and NE reductions against its NE.The plotted values are point estimates; paired source-video-cluster bootstrap estimates are also retained with the figure data.Source: F SUPPLEMENTARY ANALYSES · p. 23

F.1 ROUTE GEOMETRY

  • Short routes are ≤20 m and long routes >20 m; the independent turning partition uses a 223.2241° threshold and groups of 3,089 and 1,545 episodes.Table 9 covers all 4,634 episodes; its SR/SPL rows compare the Navigation Memory Tool harness with the full method, and aggregate to their overall results.Source: §F.1 · p. 23, Table 9 · p. 23

F.2 PAIRED MEMORY AND PREDICTION PROBES

Paired probes test how memory and predicted-video inputs affect reasoning outputs, separately from full-episode navigation; observed-feedback cue labels are model assessments.

F.2 PAIRED MEMORY AND PREDICTION PROBES

  • Updated subgoal context changes 199 of 1,501 policy-action assessments (13.3%) and 134 retain/revise requests (8.9%); preview-before-revision raises requests from 147 to 214.Among 133 states where both conditions revise, mean endpoint difference is 0.28 m; these are input-sensitivity measurements, not independent accuracy estimates.Source: §F.2 · pp. 23–24
  • At 525 fixed states, the construction probe holds destination, observations, memory, and policy action constant while changing whether a predicted rollout is supplied.Each condition independently constructs a new movement, and these state-level probes are separate from full-episode navigation results.Source: §F.2 · p. 24, Table 10 · p. 24
  • Prediction introduces 81 revisions and withdraws 14; construction revises in both conditions at 133 states and retains the policy in both at 297.Only 45 of 525 construction responses explicitly cite a predictive cue.Source: §F.2 · p. 24

F.3 OBSERVED FEEDBACK

  • Frozen Qwen judges 183 observed cues supported (51.4%), 39 contradicted (11.0%), and 134 unresolved (37.6%) across 356 matched executions.These labels are model assessments, not independently annotated ground truth.Source: §F.3 · p. 24

G INFERENCE COST

Inference costs are profiled separately for predictive rollouts and VLM requests on one NVIDIA H200 GPU, with reported generation time and peak memory use.

G INFERENCE COST

  • Both components are profiled separately on one NVIDIA H200 GPU using BF16 inference.The reasoner uses Qwen3.8-27B with FlashAttention-2, greedy decoding, batch size one, and a 384-token output limit.Source: G INFERENCE COST · pp. 24–25
  • Table 13 reports per-request latency at batch size one, timing each named operation separately from video generation.Action Review receives four predicted frames plus three observed images; Memory Update uses observed before–after transitions.Source: Table 13 · p. 25, G INFERENCE COST · p. 25
  • Predictive sandbox generation takes 11.37 s on average and uses 34.06 GiB of peak allocated GPU memory.The statistics cover 1,700 rollouts of 60 predicted frames each, and timing includes video export.Source: G INFERENCE COST · p. 25, Table 12 · p. 25
  • The reasoner peaks at 51.89 GiB of allocated GPU memory across measured operations.Action Execute adds no VLM call; prediction and revision are conditional, and revised actions return to preview and review.Source: G INFERENCE COST · p. 25

Related papers

Loading 2610.04916v1…