Source-linked AI summary
What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation
Yuhang Peng, Zhikai Wang, Ziyi Wang, Jiaru Zhong, Xiaoji Zheng, Jingzhi Huang, Yi Wang, Jiangtao Gong
TL;DR
Zero-shot VLN raises the question of how smaller open models should retain and organize navigation history across decisions. The paper combines task, temporal, and spatial memories, and its complete system achieves the highest SR and SPL among the evaluated Qwen3-based zero-shot baselines.
Problem
The paper asks how navigation history should be represented for smaller open models that must connect current observations with instruction progress and earlier evidence across actions.
Source: §1 · p. 1Method
The system combines instruction progress, pose-anchored observation history, and accumulated geometry in complementary task, temporal, and spatial memories.
Source: §1 · pp. 1–2Results
0.263 SR and 0.196 SPL are the complete system's results, the highest among the evaluated Qwen3-based zero-shot baselines.
Source: §5.3 · p. 8Takeaways & Limitations
LAW2 complements endpoint metrics by exposing changes in directional responsiveness and route fidelity under instruction interventions.
Source: §6 · p. 10Alternative memory mechanisms and improved extraction and compression of task-relevant information remain future-work directions.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Zero-shot vision-and-language navigation (VLN) requires an agent to use past observations without navigation-specific training. We study what information an 8B-scale navigation system should retain and how it should be organized for subsequent decisions. Our agent combines instruction progress, hierarchically compressed observation history anchored to poses, and accumulated geometry in complementary task, temporal, and spatial memories. An 8B observer summarizes observations, and an 8B reasoner uses the memories to select waypoint intents or request additional views, a geometric controller executes the selected targets. A semantic information perspective motivates the distinction between retaining evidence, making it accessible to the reasoner, and acquiring further observations. On R2R-CE val-unseen, the complete system achieves the highest among the evaluated Qwen3-based zero-shot baselines. Removing temporal memory reduces SR and sharply increases average path length from 16.9 to 39.1m, removing the spatial-map and planning component reduces SR as well. Real-world experiments on a Unitree Go2 show that our zero-shot method achieves higher success rates than the post-trained baseline in both corridor and office environments, suggesting better transfer to the evaluated real-world settings. As a complementary evaluation, LAW2 uses paired instruction interventions to examine behavioral responses, instruction following ability beyond endpoint success. These results support preserving task progress, observation history, and geometry in structured memory for zero-shot navigation with smaller open models.
1 INTRODUCTION
The paper asks how smaller open models should represent navigation history, then proposes complementary memories for instruction progress, pose-anchored observations, and explored geometry. It evaluates this design in an 8B-observer/8B-reasoner system and introduces LAW2 to assess behavioral responses to instruction changes.
1 INTRODUCTION
- The paper asks how navigation history should be represented for smaller open models, noting that replacing GPT-5 with Qwen3-VL-8B in reproduced HSGM reduces R2R-CE val-unseen SR from 0.463 to 0.049.Source: §1 · p. 1
- An 8B observer summarizes observations while an 8B reasoner combines summaries, instruction, and spatial memory to select waypoint intents or request additional views.The design rationale distinguishes retaining task-relevant evidence from making it accessible to the reasoner and acquiring further views.Source: §1 · p. 2
- The proposed representation combines instruction progress, pose-anchored observation history, and accumulated geometry in task, temporal, and spatial memories.Temporal memory uses hierarchical summaries anchored to poses; spatial memory contains an occupancy map and committed target.Source: §1 · pp. 1–2
- The system evaluation includes component ablations, integration with CA-Nav, and real-world deployment to examine the memory representation's utility.Source: §1 · p. 2
- LAW2 complements navigation metrics with paired tests of directional responsiveness, route-information dependence, and paraphrase sensitivity.Its interventions include direction reversals, route-information removal, and paraphrases that preserve meaning.Source: §1 · p. 2
2 RELATED WORK
Prior VLN work spans benchmark tasks, agentic systems, and structured memory representations. This paper focuses on combining temporal records with accumulated geometry for an 8B-scale reasoner used without navigation-specific training.
2.1 VISION-AND-LANGUAGE NAVIGATION
- R2R-CE extends instruction-following navigation from discrete graphs to continuous environments, while REVERIE additionally requires identifying an instruction-referred object.Spatial VLN systems include online topological mapping with obstacle-aware planning and combinations of global topological and local metric representations.Source: §2.1 · p. 2
2.2 AGENTIC NAVIGATION AND MEMORY
- Agentic VLN combines language or vision-language models with perception, planning, and execution, while the paper focuses on organizing task progress, pose-tied history, and geometry as complementary evidence.CA-Nav uses a value map to score instruction-relevant locations, and BSC-Nav explores brain-inspired spatial memory.Source: §2.2 · p. 2
2.3 SPATIAL AND TEMPORAL MEMORY FOR NAVIGATION
- Prior memory approaches structure history through hierarchical panoramic encoding, spatial grids, semantic maps, or grounded world graphs with selective retrieval.The paper emphasizes complementary temporal records and accumulated geometry for an 8B-scale reasoner; Figure 1 shows task, temporal, and spatial memory feeding waypoint selection or preview.Source: §2.3 · p. 3, Figure 1 · p. 3
3 METHODOLOGY
The method organizes instruction progress, pose-anchored observation history, and accumulated geometry into complementary memories, then uses memory-conditioned reasoning to select and execute navigation intents. It separates evidence availability, retention, and use while allowing additional views when needed.
3.1 PROBLEM FORMULATION
- The agent operates in a partially observable environment, updating internal memory from prior memory, action, and observation rather than receiving the complete environment state.The internal memory is not necessarily an exact Bayesian belief state; benchmark success rate alone also does not establish compliance with all route instructions.Source: §3.1 · pp. 3–4
3.2 DESIGN RATIONALE
- The design rationale distinguishes missing observations, discarded evidence, and difficulty interpreting retained evidence, motivating acquisition, retention, and organization mechanisms.Previewing can acquire missing evidence, but does not guarantee that it resolves a query or is correctly interpreted and retained.Source: §3.2 · p. 4, §3.4 · p. 6
3.3 SPATIOTEMPORAL MEMORY
- The memory comprises task context, temporal observations, and accumulated spatial evidence, without explicitly maintaining a posterior over environment states.Updates use RGB observations, aligned depth, and camera poses with known camera intrinsics.Source: §3.3 · p. 4
3.3.1 TASK MEMORY
- Task memory retains the instruction, ordered subgoals, and active-subgoal index, which advances when the reasoning module judges the current subgoal complete.The instruction and decomposition are initialized once; subsequent updates change only the stage index.Source: §3.3.1 · p. 5, §3.4 · p. 6
3.3.3 SPATIAL MEMORY
- Spatial memory combines a ground-plane occupancy map, the trajectory accumulated from observed poses, and a committed world-coordinate target.Map cells are free, occupied, or unknown; the map supports obstacle-aware planning and frontier exploration, while pose anchoring links summaries to trajectory segments.Source: §3.3.3 · p. 5
3.3.2 TEMPORAL MEMORY
- Temporal memory retains recent observations and execution history while compressing earlier observations into pose-anchored summaries across subgoal transitions.The recent window keeps eight observations; each eight-observation chunk overlaps the preceding chunk by two frames, and five pending summaries are compressed together.Source: §3.3.2 · p. 5, §3.4 · p. 6
3.4 MEMORY-CONDITIONED NAVIGATION POLICY
- The policy selects waypoint intents or requests PREVIEW views from memory, while a geometric controller executes targets through obstacle-aware paths.Each step updates memory; a new target is selected when none is committed, and primitive actions continue until the target is released or STOP is issued.Source: §3.4 · p. 6
3.5 ACTION ABSTRACTION
- Waypoint prediction and selection perform substantially better than primitive-action prediction, although the reported comparison does not isolate context length as its cause.Intent decisions may span multiple primitive steps, reducing the number of target-selection decisions when an intent covers several steps on average.Source: §3.5 · p. 7
4 EVALUATION PROTOCOL
The evaluation uses standard R2R-CE navigation metrics and LAW2 paired instruction interventions. LAW2 compares transformed instructions with originals under the same scene, start pose, and execution settings.
4 EVALUATION PROTOCOL
- R2R-CE evaluation reports SR, SPL, NE, OSR, and nDTW, with RSR reported only when OSR is positive and both metrics share complete oracle-success coverage.RSR measures successful stopping conditional on reaching the goal region; it does not separately identify perception, progress-estimation, or stopping errors.Source: §4 · p. 7
- LAW2 pairs each transformed instruction with its original under the same scene, start pose, and execution settings to assess behavioral response to instruction changes.Its interventions are FLIP (82 episodes), GOAL-ONLY (174 episodes), and four PARAPHRASE variants for each of 200 instructions.Source: §4 · p. 7
- Figure 2 presents qualitative results on R2R in its top panel and real-world experiments in its bottom panel.The caption identifies the settings, but does not specify a quantitative comparison.Source: Figure 2 · p. 8
5 EXPERIMENTS
The experiments evaluate the memory design in simulation, on a Unitree Go2, and through paired instruction interventions. The complete system leads evaluated Qwen3-based zero-shot baselines, while ablations and real-world tests provide further evidence about its components and scope.
5.1 SIMULATION EXPERIMENT / 5.2 REAL-WORLD EXPERIMENT
- The simulation comparison uses an 8B observer and Qwen3-VL-8B reasoner against post-training and zero-shot baselines; the Go2 evaluation compares against AwareVLN in corridor and office environments.The Go2 communicates with the A100 server remotely, with FAST-LIVO2 reconstruction and localization supplying evaluation poses.Source: §5.1 · p. 8, §5.2 · p. 8
5.3 MAIN RESULTS
- SR of 0.263 and SPL of 0.196 are the highest among evaluated Qwen3-based zero-shot baselines; post-trained models score higher in simulation.The comparison is between complete systems with different perception modules and action interfaces.Source: §5.3 · p. 8
- Adding temporal memory to CA-Nav raises SPL from 0.097 to 0.126 and nDTW from 0.162 to 0.225, while SR changes from 0.243 to 0.247.Average path length falls from 27.82 to 23.50 m.Source: §5.3 · p. 8
- In corridor and office environments, the system achieves higher SR and lower NE than AwareVLN.The Go2 streams onboard observations to a remote server and executes returned actions; environments are reconstructed and localized for evaluation.Source: §5.2 · p. 8, §5.3 · p. 9
5.4 ABLATION STUDY
- Removing temporal memory lowers SR from 0.263 to 0.190, SPL from 0.196 to 0.112, and nDTW from 0.379 to 0.201.Average path length rises from 16.9 to 39.1 m and steps from 131 to 292, while OSR rises from 0.328 to 0.475.Source: §5.4 · p. 9
- Removing spatial memory lowers SR to 0.205 and SPL to 0.142; removing both spatial and temporal components yields SR of 0.185 and nDTW of 0.202.Task memory is retained in every variant.Source: §5.4 · p. 9
- Removing PREVIEW lowers SR from 0.263 to 0.190 with nearly unchanged nDTW, while the intent-action variant scores 0.263/0.196 versus 0.125/0.085 for primitive actions.Token consumption also decreases without PREVIEW.Source: §5.4 · p. 9
5.5 INSTRUCTION-INTERVENTION ANALYSIS
- LAW2 tests directional responsiveness, route-information dependence, and paraphrase sensitivity, with different patterns across follow rates, endpoint success, and route fidelity.Under FLIP, the system has the highest mean follow rate across paired conditions; GOAL-ONLY and PARAPHRASE show that endpoint success and nDTW can change differently.Source: §5.5 · p. 9
6 CONCLUSION
The conclusion presents complementary task, temporal, and spatial memories as a design for zero-shot navigation with 8B-scale models. It also identifies LAW2 as a complement to endpoint metrics and points to alternative memories and improved information extraction as future work.
6 CONCLUSION
- The proposed design combines instruction progress, pose-anchored observation history, and accumulated geometry in complementary memories for zero-shot navigation.The complete system achieves the highest SR and SPL among evaluated Qwen3-based zero-shot baselines; ablations indicate utility from temporal history and spatial mapping with planning.Source: §6 · p. 9
- LAW2 supplements endpoint metrics by exposing changes in directional responsiveness and route fidelity under instruction interventions.The paper specifies intervention protocols and plans to release implementation, configurations, and benchmark records after acceptance.Source: §6 · p. 10, REPRODUCIBILITY STATEMENT · p. 10
- Future work will explore alternative memory mechanisms, including native memory and test-time training, and improved extraction and compression of task-relevant information.Generative AI tools were used to construct and review LAW2's PARAPHRASE portion and to improve manuscript readability.Source: §6 · p. 10, AI USE STATEMENT · p. 10
A ADDITIONAL EXPERIMENT RESULTS
The additional experiments cover ablations, VLM-call and token costs, real-world navigation, and paired instruction-intervention benchmarks.
A ADDITIONAL EXPERIMENT RESULTS
- Table 2 evaluates model variants across navigation performance, path efficiency, step budget, and average token consumption per episode.The ablation setup replaces memory components with minimal baselines.Source: Table 2 · p. 13
- Table 3 compares HSGM and Ours on VLM-call frequency and token consumption per call or step and per episode.The accompanying text reports a larger input-token load per decision for HSGM than for Ours.Source: Table 3 · p. 13
- Table 4 evaluates corridor and office navigation for simple and complex instructions, reporting NE and SR within 2.0 m.Complex instructions require a turn or passing a doorway before reaching the target.Source: Table 4 · p. 13
- Table 5 reports initial-turn obedience on 82 selected R2R Val Unseen episodes, comparing original instructions with versions that reverse one direction word.It includes Follow, Opposite, Straight, MeanFollow, and WordEffect; success at the original goal is auxiliary.Source: Table 5 · p. 14
- Table 6 evaluates GOAL-ONLY on 174 episodes, retaining the final annotated sub-instruction and measuring SR against the unchanged target and nDTW against the original route.Source: Table 6 · p. 14
- Table 7 reports PARAPHRASE results on 200 R2R Val Unseen episodes across same-register, terse, colloquial, and landmark-re-description rewrites.NE is reported in meters.Source: Table 7 · p. 14
B.1 MOTIVATION AND EVALUATION DESIGN
LAW2 uses matched instruction interventions to examine navigation behavior, with episode selection, transformations, and scoring designed around distinct instruction changes.
B.1 MOTIVATION AND EVALUATION DESIGN
- LAW2 evaluates instruction controllability through paired interventions, with each transformed instruction matched to its original under fixed scene, pose, goal, trajectory, and execution settings.FLIP tests directional responsiveness, GOAL-ONLY route-information dependence, and PARAPHRASE robustness to meaning-preserving rewrites.Source: §B.1 · pp. 14–15, Table 8 · p. 15
B.2 SOURCE EPISODES AND INSTRUCTION ALIGNMENT
- The benchmark uses R2R-CE val-unseen episodes and FGR2R human-annotated sub-instructions, with GOAL-ONLY and PARAPHRASE drawn from a fixed stratified 200-episode manifest.The manifest allocates 50 episodes each to doorway, walking, stairs, and abstract stopping conditions; FLIP candidates are selected separately from 1,839 episodes.Source: §B.2 · p. 15
B.3.1 CANDIDATE SELECTION
- FLIP candidates must contain a single movement-command direction, support a 45°–135° turn in the reference trajectory, and allow an opposite-direction probe that passes a local navigability check.The filter yields 94 candidates, from which 41 per direction are retained: 82 episodes across 11 scenes.Source: §B.3.1 · p. 16
B.3.2 INSTRUCTION TRANSFORMATION RULES
- FLIP reverses only the selected direction word, and both conditions use the same v1-3 start orientation.A paired original run with a different orientation is invalid.Source: §B.3.2 · p. 16
B.3.3 SCORING OBEDIENCE
- Departures of at least 30° are scored as following or opposing the supplied direction, while straight or insufficient departures remain in the denominator as non-turning outcomes.WordEffect measures the net excess of following over opposing across paired conditions; original-goal SR is auxiliary because the reversed command changes the requested route.Source: §B.3.3 · p. 16
B.4.1 COHORT SELECTION
- GOAL-ONLY excludes 24 episodes with one merged segment and two with exactly two segments ending in a bare stopping command, leaving 174 episodes selected independently of agent performance.Source: §B.4.1 · p. 17
B.4.2 STRICT FINAL-SEGMENT EXTRACTION
- GOAL-ONLY extracts the final raw FGR2R segment using its character span and removes only surrounding whitespace, without restoring or supplementing text.Source: §B.4.2 · p. 17
B.4.3 EVALUATION
GOAL-ONLY compares matched transformed and original instructions, while PARAPHRASE rewrites boundary-aligned instruction segments and validates their structure and content.
B.4.3 EVALUATION
- GOAL-ONLY compares transformed instructions with matched originals on the same 174 episodes, keeping the target and reference trajectory unchanged.SR measures stopping near the original target, while nDTW measures fidelity to the original route; nDTW changes do not establish compliance with a uniquely specified alternative route.Source: §B.4.3 · p. 17
B.5.1 GENERATION
- PARAPHRASE generates four rewrite styles for each source instruction and preserves segment order, viewpoint alignment, and route boundaries.The styles are same-register, terse, colloquial, and landmark re-description.Source: §B.5.1 · p. 17
B.5.2 AUTOMATIC VALIDATION
- All 800 rewrites passed hard validation, with mean lengths of 130.7 characters for ORIG and 138.6, 105.8, 186.2, and 159.4 for A1–A4, respectively.Automatic checks reject structural failures, while lexical flags trigger review rather than rejection because lexical overlap alone does not establish semantic equivalence.Source: §B.5.2 · p. 17
B.5.3 REVIEW AND LIMITATIONS
The paper distinguishes the information available to a receiver from its realized computation and task performance, while noting limits on semantic guarantees and review independence.
B.5.3 REVIEW AND LIMITATIONS
- Generation and semantic review both used Claude Opus 5.5, limiting the independence of the assessment.The review assessed 800 rewrites with 90 hidden controls; all controls received expected judgments, and 199 of 200 A4 rewrites were initially judged equivalent before one was corrected.Source: §B.5.3 · p. 18
C A SEMANTIC INFORMATION PERSPECTIVE ON EMBODIED AI
- The semantic-information perspective separates a communicated task message, the receiver’s existing information, its computation, and later environment interaction.It is used as a design rationale, not as a claim that communication bounds characterize memory cost or navigation performance.Source: C A SEMANTIC INFORMATION PERSPECTIVE… · p. 18, §C.1 · p. 18
C.1.1 RECEIVER MODEL
- The receiver model separates internal configuration, current information state, processing mechanism, and output-selection settings without prescribing a particular architecture or memory organization.The state update uses the prior state, preceding action, and current observation; the initial message and initial observation enter the initial state.Source: §C.1.1 · p. 18
C.1.2 SEMANTIC SUFFICIENCY AND REALIZED COMPUTATION
- Semantic sufficiency means the premises entail a task-relevant proposition across compatible hypotheses, but does not guarantee that the receiver computes it or selects an appropriate action.Correctness in the actual environment also requires the premises to hold there.Source: §C.1.2 · p. 19
C.1.3 RETAINED INFORMATION AND NEW EVIDENCE
- Making an existing semantic consequence explicit preserves the kernel, whereas additional interpreted evidence may exclude previously compatible hypotheses.This idealized additive update requires consistent combined premises; actual information-state updates may be lossy or revise earlier assumptions.Source: §C.1.3 · p. 19
- The framework distinguishes retaining evidence, organizing it for computation, and acquiring new evidence, but does not prescribe storage or identify the cause of an unsuccessful decision.Source: §C.1.3 · p. 19
C.2 INTERACTION AND TASK ACHIEVEMENT
This section formalizes task satisfaction as an outcome of receiver–environment interaction over a finite horizon. It distinguishes local judgments from requirements that the executed actions must satisfy.
C.2 INTERACTION AND TASK ACHIEVEMENT
- The interaction model separates environment state, executed action, and observation, while representing the receiver’s internal state separately.Actions can change the physical environment, affect observations, or do both.Source: §C.2 · p. 19
- After each new observation, the receiver updates its information state; the resulting policy acts without direct access to the complete environment state.Its internal state need not be an exact Bayesian belief, and optimal inference or action selection is not assumed.Source: §C.2 · p. 20
- The finite-horizon objective evaluates whether the task requirements are met, with early termination represented by an absorbing terminal state.Initialization uses an initial-state distribution and initial-observation model, followed by subsequent interaction; the objective assigns zero intermediate rewards and a terminal task-satisfaction reward.Source: §C.2 · pp. 19–20
- J_QG(R) is the probability that the receiver’s executed action set satisfies QG over trajectories induced by receiver computation and environment interaction.The environment models, initialization distribution, horizon, and message are held fixed; the formulation uses zero intermediate rewards and a terminal task-satisfaction reward.Source: §C.2 · p. 20
- Q_t concerns a judgment during interaction, whereas Q_G specifies what the executed action set must achieve.A receiver’s completion declaration alone does not establish task satisfaction.Source: §C.2 · p. 20