Source-linked AI summary
MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation
Xueyao Wei, Yifei Wang, Delun Li, Shaoan Wang, Junzhi Yu, Wenjie Song, Xingyu Chen
TL;DR
Long-horizon VLN must preserve useful history under limited context while representing where observations occurred and reusing experience across episodes. MemoryVLN combines history compression, pose-aware representations, hierarchical memory, and retrieval, improving success rate over state-of-the-art baselines on both benchmarks.
Problem
Long-horizon VLN requires preserving useful historical experience within limited context, while existing methods rarely unify history compression, ego-aware representation, cross-context memory updates, and retrieval.
Source: §1 · p. 2Method
MemoryVLN compresses visual history, combines 2D appearance, 3D geometry, and 6D poses, organizes in-episode and context-level memory, and retrieves relevant experience for action prediction.
Source: §1 · p. 2Results
MemoryVLN improves success rate over state-of-the-art baselines by approximately 2% on R2R-CE and 6.5% on RxR-CE.
Source: §1 · p. 3Takeaways & Limitations
MemoryVLN’s hierarchical memory mechanism yields stable gains for long-horizon navigation on R2R-CE and RxR-CE.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Large vision-language models (LVLMs) have shown strong potential for embodied vision-language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose \textbf{MemoryVLN}, a unified memory-driven navigation system built around memory construction and management. During memory construction, \emph{Long Sequence Compression} employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited token budget. \emph{Ego-Aware Memory Representation} further integrates 2D semantic tokens, 3D geometric tokens, and 6D agent poses, allowing each memory to encode both what was observed and where it was observed. During memory management, \emph{Hierarchical Memory Bank} maintains in-episode and context-level memories through cross-context warm-start and confidence-controlled write-back. \emph{Memory Enhancement} then retrieves the Top-$K$ relevant memories according to the current navigation state and fuses them with the current visual tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2\% and 6.5\% in successful rate).
1 Introduction
Long-horizon VLN requires preserving useful historical experience under limited context, not merely processing more frames. MemoryVLN unifies memory construction and management, and improves success rate on both R2R-CE and RxR-CE.
Motivation
- Long-horizon navigation must preserve useful evidence such as distant landmarks, visited paths, incorrect branches, and spatial layout within a limited context budget.The challenge is preserving useful history rather than simply seeing more frames.Source: §1 · p. 2
Research gap
- Prior LVLM-based VLN methods address history through sequences, recurrent or latent states, or task-specific aggregation, but rarely unify compression, ego-aware representation, memory updating, and retrieval.The paper identifies redundancy, context forgetting, spatial misalignment, and poor cross-context transfer as remaining vulnerabilities.Source: §1 · p. 2
Proposed approach
- MemoryVLN combines sequence compression, spatially grounded memory tokens, hierarchical in-episode and context-level banks, and retrieval with residual fusion for action prediction.Compression allocates a limited token budget to dense recent and sparse representative distant observations; the representation combines 2D tokens, 3D tokens, and 6D poses.Source: §1 · p. 2
Contributions
- The paper frames MemoryVLN as a unified system covering memory construction and management across the full memory process.Its stated contributions cover compression, representation, storage, update, retrieval, and reuse.Source: §1 · p. 2
Experiments
- MemoryVLN improves success rate over state-of-the-art baselines by approximately 2% on R2R-CE and 6.5% on RxR-CE.The reported gains validate the memory designs for long-horizon navigation.Source: §1 · p. 3
2 Related Work
Prior navigation research has developed history encoders, maps, geometric representations, and memory mechanisms, but the cited approaches generally do not combine compression, spatial grounding, and reusable cross-context memory. MemoryVLN brings these memory functions together in a hierarchical system.
Embodied Navigation
- Classical VLN uses data augmentation, history or map encoding, waypoint prediction, transfer, and history reuse, but typically relies on bounded histories, maps, or episode-specific planning.The paper contrasts these approaches with a unified memory that can be compressed, spatially grounded, and reused.Source: §2 · p. 3
- LLM- and LVLM-based agents strengthen reasoning and action generation, navigation foundation models improve context modeling, and world models support planning through future prediction.MemoryVLN covers memory compression, representation, storage, retrieval, and update without additional navigation data.Source: §2 · p. 3
Memory for Long-Horizon and Persistent Navigation
- Cognitive models and Memory Networks motivate hierarchical storage, but general memory mechanisms do not directly address visual redundancy, pose misalignment, or unreliable write-back in navigation.Navigation memory must decide what to retain, compress, retrieve, or transfer as observations accumulate online.Source: §2 · p. 3
- Existing navigation memory spans episodic encoders, trajectory reuse, maps, selective storage, and continual retrieval, while most methods center on single episodes, fixed implicit states, or static maps.MemoryVLN instead combines short- and long-term pools with a persistent Context-Level Bank, thresholded retrieval, warm start, confidence gating, and LRU replacement.Source: §2 · p. 3
Spatial and Geometric Representations for Navigation
- Semantic maps, BEV or grid representations, 3D feature fields, RGB-D models, and implicit memories capture spatial structure, but often depend on depth or preconstructed maps and support episode-specific use.MemoryVLN binds appearance, geometry, and pose in retrievable memory tokens.Source: §2 · p. 3
- DUSt3R and VGGT provide geometric priors but do not decide which observations persist or how experience is reused across contexts.MemoryVLN fuses frozen VGGT geometry with 2D visual tokens and 6D poses, then stores the resulting tokens in an updatable hierarchical bank.Source: §2 · p. 3
3 Method
MemoryVLN represents long-horizon navigation using compact, pose-aware historical tokens organized into in-episode and persistent context-level banks. Its pipeline compresses history, constructs and updates memory, and retrieves relevant experience for action prediction.
3.1 Problem Formulation
- Long-horizon decisions use evidence from the current observation, the trajectory, and retained experience from previous episodes in the same environment.MemoryVLN separates this history into an In-Episode Bank and a persistent Context-Level Bank.Source: §3.1 · pp. 3–4
3.2 Overview
- The pipeline has memory construction and memory management stages, with instruction tokens retained while only growing visual history is selectively compressed.The four modules bound the visual-token budget, attach appearance, geometry, and pose, organize experience, and retrieve relevant history for the Qwen2.5-VL action-prediction backbone.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
3.3 Long Sequence Compression
- Long Sequence Compression uses an Ebbinghaus-inspired retention prior and a sublinear square-root budget schedule to preserve dense recent evidence and sparse distant observations.It distributes budget across temporal chunks with higher weights for more recent chunks, then samples key frames within chunks; probabilistic sampling reduces near-duplicate clusters, while sampled-chunk recent frames and trajectory endpoints are retained.Source: §3.3 · p. 5
3.4 Ego-Aware Visuospatial Representation
- Ego-Aware Visuospatial Representation combines ViT appearance tokens, frozen VGGT geometry tokens, and each retained frame’s relative 6D pose.The aligned geometric stream is added as a residual, keeping the pretrained ViT representation primary while encoding visual content, pose, and structural relations across historical viewpoints.Source: §3.4 · pp. 5–6
3.5 Hierarchical Memory Bank
- The Context-Level Bank retrieves a cosine-similar prior above a threshold for warm start, while in-episode short- and long-term pools preserve recent detail and compress older experience.At episode end, confidence-gated write-back limits persistent storage to reliable candidates; a full context bank replaces its least-recently-used entry.Source: §3.5 · p. 6
3.6 Memory Enhancement
- Memory Enhancement retrieves top-k in-episode entries using a query pooled from current-frame tokens, weights them by cosine similarity, and fuses their summary with the current representation.The current observation remains the primary signal while relevant distant evidence is injected.Source: §3.6 · p. 6
3.7 Training and Inference
- MemoryVLN uses standard VLN trajectories without DAgger rollouts or additional navigation data, with a frozen VGGT geometry branch and jointly optimized task-specific components.Inference executes compression, pose injection, hierarchical memory maintenance, top-k enhancement, and confidence-gated write-back online; the in-episode bank resets per episode while the context-level bank persists within its environment scope.Source: §3.7 · p. 7
4 Experiments
MemoryVLN improves success over the strongest data-comparable baselines on both continuous-navigation benchmarks, while component ablations and qualitative examples examine its memory design. The experiments evaluate the system on unseen environments and test simulated and real-world navigation.
4.1 Experimental Setup
- Evaluations use R2R-CE and RxR-CE val-unseen, with standard VLN training trajectories and no DAgger rollouts or additional navigation data.The reported metrics are NE, OS, SR, SPL, and nDTW; baseline results follow their respective published settings, so some are not fully data-matched.Source: §4.1 · p. 7, §4.2 · p. 7
4.2 Main Results
- SR improves by approximately 2% on R2R-CE and 6.5% on RxR-CE over the strongest data-comparable baselines.MemoryVLN also substantially reduces navigation error on R2R-CE and achieves strong SR and SPL; the RxR-CE benchmark is longer and more linguistically complex.Source: §4.2 · p. 7, Table 1 · p. 8
4.3 Ablation Study
- Removing Long Sequence Compression causes out-of-memory failures on both benchmarks as historical visual tokens grow with trajectory length.Compression is therefore necessary for long-horizon inference, not merely an efficiency optimization.Source: §4.3 · p. 8
- Restoring ego-pose encoding improves most reported metrics, including R2R-CE SR by 2.01 points and RxR-CE nDTW by 1.51 points.The ablation indicates that pose information helps disambiguate visually similar observations from different locations.Source: §4.3 · p. 8
4.4 Qualitative Analysis
- In Habitat, MemoryVLN reaches the target where JanusVLN deviates and fails; it also completes two instructed routes in a real indoor environment.The authors suggest the real-world examples indicate transfer of memory mechanisms learned in simulation to physical environments.Source: Figure 3 · p. 9, §4.4 · p. 10
4.3 Ablation Study
- The In-Episode Bank raises R2R-CE SR and SPL from 53.34 and 49.06 to 54.76 and 50.26, respectively.On RxR-CE, it improves SR, SPL, and nDTW by 1.15, 2.29, and 1.69 points.Source: §4.3 · p. 9
- The Context-Level Bank improves R2R-CE OS, SR, and SPL by 1.52, 0.60, and 0.61 points, and RxR-CE SR, SPL, and nDTW by 0.47, 1.33, and 0.56 points.RxR-CE NE increases slightly; the authors report gains in navigation success and trajectory quality overall.Source: §4.3 · pp. 9–10
5 Conclusion
The paper identifies the lack of hierarchical navigation memory—not context size alone—as a core bottleneck in long-horizon VLN. MemoryVLN turns history into compact, spatially grounded, retrievable, reusable experience, with future work proposed on more persistent and map-like memory.
5 Conclusion
- The paper frames the core bottleneck of long-horizon VLN as the lack of Hierarchical Navigation Memory, rather than context size alone.MemoryVLN uses compression, ego-aware visuospatial representation, and memory enhancement to transform observations into compact, spatially grounded, retrievable, and reusable experience.Source: §5 · p. 10
- Future work may explore more persistent lifelong memory, more explicit map-like memory, and tighter integration between memory and planning.Source: §5 · p. 10
Appendix
- The supplementary document is intended as a standalone PDF accompanying the code-and-data package, with essential material retained in the main paper.It contains no links to external code repositories.Source: Appendix · p. 14
A.1 Task, Data, and Evaluation Protocol
The task evaluates continuous navigation on R2R and RxR in Matterport3D environments using Habitat. Episodes have a 400-action cap, with memory persistence restricted by episode and environment boundaries.
A.1 Task, Data, and Evaluation Protocol
- Evaluation uses continuous-navigation R2R and RxR on Matterport3D environments in Habitat, with reported main and ablation results on val-unseen.Training uses standard VLN trajectories only, without DAgger rollouts or additional navigation data.Source: §A.1 · p. 14
- At each step, the agent receives an instruction and egocentric RGB observation and predicts MOVEFORWARD, TURNLEFT, TURNRIGHT, or STOP.Closed-loop evaluation is capped at 400 actions per episode.Source: §A.1 · p. 14
- The In-Episode Bank resets at episode boundaries, while the Context-Level Bank persists only within its associated environment.This prevents information from one scene entering another scene’s evaluation.Source: §A.1 · p. 14
A.2 Metrics
The appendix defines navigation metrics, describes implementation settings, and documents the experiment environment and randomness controls.
A.2 Metrics
- NE is final geodesic distance to the goal; SR measures final-state success, OS uses the closest visited state, SPL combines success with path efficiency, and nDTW measures trajectory similarity.The paper reports these metrics to capture final accuracy, goal visitation, efficiency, and adherence to the reference route.Source: §A.2 · pp. 14–15
A.3 Implementation Configuration
- The reported implementation uses Qwen2.5-VL as the action-prediction backbone and keeps the VGGT geometry branch frozen; evaluation commands may override configuration values.The submitted configuration records model and memory settings.Source: §A.3 · p. 15, Table 3 · p. 15
A.4 Software Environment and Randomness
- The released setup specifies software versions and seed 42, but full deterministic mode is not enabled, so reruns are not guaranteed to be bitwise identical.Reported training uses bfloat16, AdamW, and a global batch size of 64 across 64 GPUs.Source: §A.4 · pp. 15–16
B More Ablation Studies
Paired evaluation reports gains on both benchmarks, with statistical significance on RxR but not R2R. Additional R2R comparisons show how the gains vary across training-data stages, while retrieval interpretation distinguishes short- and long-term memory.
B More Ablation Studies
- The paired analysis matches outputs by scene and episode, using 1,839 R2R pairs and 3,669 RxR pairs, with 3,570 valid pairs for RxR SPL.The RxR SPL analysis excludes 99 pairs with non-finite values; confidence intervals use 20,000 paired-bootstrap resamples.Source: B More Ablation Studies · p. 16
- R2R SR and SPL improve by 2.0 and 1.2 percentage points without significance; RxR SR and SPL improve by 6.2 and 5.2 points, with both differences significant.SR significance is assessed with the exact McNemar test and SPL with the two-sided Wilcoxon signed-rank test.Source: B More Ablation Studies · p. 16
- At 46.4% training data, MemoryVLN exceeds JanusVLN by 4.2 SR and 7.1 SPL points; reported-final gains are 2.0 SR and 1.2 SPL points.The dashed line marks the last data-matched comparison; the reported-final point is not training-data matched.Source: B More Ablation Studies · p. 16, Figure 4 · p. 17
C.1 Interpretation of Memory Retrieval
- For long trajectories, the short-term pool retains nearby observations while the long-term pool stores consolidated history; context-level reuse requires sufficient same-environment similarity.Pose and retrieved history can help distinguish visually similar rooms at different scene positions.Source: §C.1 · p. 17
C.2 Failure Case Analysis
Four reproducible failed rollouts expose different breakdowns in room selection, final-doorway grounding, context reuse, and instruction progress. Together, they point to mismatches among instruction state, landmarks, location, and stopping decisions rather than a single execution issue.
C.2 Failure Case Analysis
- The R2R rollout reaches the entrance and kitchen but takes the wrong branch instead of the dining room, ending 28.61 m from the goal without collisions.The analysis attributes the error to failure to bind later observations to the requested room and landmark and to verify branch consistency.Source: §C.2 · p. 18
- After the second stair segment, the R2R agent misses the first-right massage-room entrance and stops early at 60 steps versus 82 for the successful reference.The task requires jointly preserving ordered stair and turn constraints and verifying doorway orientation, room semantics, and ordinal constraints before stopping.Source: §C.2 · p. 18
- Episode 1204 failed at 5.39 m NE yet entered the Scene Bank; episode 1206 retrieved it at cosine similarity 1.0000, though disabling the bank reproduced the same failure.The trace supports a missing reliability filter, not the conclusion that the Scene Bank alone caused episode 1206's error.Source: §C.2 · p. 18
- On the 204-step RxR reference, MemoryVLN stops after 44 steps; its 16.40 m NE and 12.49 m Oracle NE accompany forgotten later subgoals and premature stopping.The case motivates explicit subgoal-progress state and a pre-STOP check for later landmarks in order.Source: §C.2 · pp. 18–19
- Across four cases, failures arise from semantic-topological selection, terminal grounding, memory quality control, and progress tracking—not a single collision or action-execution issue.The proposed remedies combine branch verification, target-consistency checks before stopping, reliability-gated write-back, and explicit subgoal progress.Source: §C.2 · pp. 17, 19
C.3 More Qualitative Results
Qualitative Habitat comparisons show MemoryVLN reaching targets in examples where JanusVLN fails, while real-office demonstrations show successful instruction following on a quadruped. The real-world evidence is qualitative rather than a large-scale physical-world benchmark.
C.3 More Qualitative Results
- Across four identical Habitat instructions, MemoryVLN reaches the target region and stops successfully where JanusVLN deviates, fails to reach the goal, or stops incorrectly.The examples span repeated turns, room transitions, stairs and corridors, and long instructions requiring precise localization.Source: §C.3 · p. 19, Figure 6 · p. 21
- A Unitree GO2 follows five natural-language office instructions to targets including a printer, coffee machine, water dispenser, and conference-room podium; this remains a qualitative demonstration.The paper presents these real-world examples as evidence of applicability beyond Habitat appearance, not as a large-scale physical-world benchmark.Source: §C.3 · p. 19, Figure 7 · p. 22
- Figure 5 pairs egocentric observations with top-down trajectories for four failure types: branch confusion, doorway grounding, context-reuse error, and long-instruction progress loss.The maps mark the goal, shortest path, and executed trajectory.Source: Figure 5 · p. 20