PaperlayerView exact revision on arXiv

Source-linked AI summary

MemVLN: Streaming Episodic Memory for Vision-Language Navigation

Yan Wang, Jiangyong Huang, Baoxiong Jia, Siyuan Huang

arXiv:2610.35066v1cs.CV

TL;DR

Streaming VLM navigation needs persistent memory to use observations from earlier in long episodes, but existing memory approaches face representation, compatibility, or capacity constraints. MEMVLN combines bounded episodic memory with geometry features and trains memory through masked, multi-scale trajectory summarization. It achieves state-of-the-art performance on most reported VLN-CE metrics under official-trajectory training, and improves success on six held-out physical episodes from 17% to 40%, or 47% after post-training.

  • Problem

    Streaming VLM agents lack persistent memory for long-horizon navigation, while prior approaches face representation, streaming-compatibility, or capacity constraints.

    Source: §1 · p. 2

  • Method

    MEMVLN combines a bounded, time- and action-pose-annotated visual memory bank with a frozen geometry encoder and masked, multi-granularity trajectory summarization.

    Source: §1 · p. 2

  • Results

    MEMVLN achieves state-of-the-art performance on most reported VLN-CE metrics under official R2R/RxR training trajectories and raises held-out physical success from 17% to 40%, then 47% after post-training.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Simulation analyses characterize when episodic memory is useful, while the physical evaluation reports improved success on held-out episodes in one scene.

    Source: §1 · p. 2, §5 · p. 9

    The physical study tests six held-out episodes in one scene, with substantial collision rates, and cross-scene generalization remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-language navigation (VLN) requires an embodied agent to follow language instructions through 3D environments. Maintaining persistent episodic memory remains a challenge for VLM-based streaming agents in long-horizon navigation. We present MEMVLN, a streaming episodic memory framework that maintains a bounded bank of compressed perceptual features annotated with timesteps and action-derived relative poses, and produces compact memory tokens via cross-attention. We pair the proposed memory with a frozen pretrained geometry encoder. To learn the memory module, we design a masking-based trajectory summarization task with multi-granularity sub-trajectory decomposition, encouraging route reconstruction exclusively from episodic memory. Under navigation training restricted to the official R2R/RxR trajectories, MEMVLN achieves state-of-the-art performance on most reported metrics on standard VLN-CE benchmarks. Long-horizon analyses further characterize when episodic memory is useful. For physical evaluation, we collect 30 instruction-conditioned episodes in one physical scene, use 24 as demonstrations for post-training, and hold out six for testing. On the held-out episodes, MEMVLN improves success over the VLM baseline from 17% to 40%, and post-training further raises it to 47%.

1 INTRODUCTION

Persistent memory remains a constraint for streaming VLM navigation over long episodes, while existing approaches trade away observation detail or do not fit monocular continuous settings. MEMVLN combines bounded episodic memory, geometry features, and memory-specific trajectory summarization, and is evaluated in simulation and on a physical robot.

Motivation

  • Existing memory methods either compress history into recurrent states, rely on observation-level representations coupled to panoramas or maps, or retain capacity limits in streaming formulations.These constraints motivate memory designed for streaming video and long-horizon navigation.Source: §1 · p. 2

Approach

  • MEMVLN stores streaming perceptual features with timesteps and action-derived relative poses, then uses cross-attention to produce fixed-size memory tokens for autoregressive action prediction.When the bank reaches capacity, consecutive frames are merged; a frozen pretrained StreamVGGT encoder and a lightweight adapter add spatial features without changing the underlying LLM architecture.Source: §1 · p. 2
  • Masked trajectory summarization trains the memory module to describe traversed paths using memory tokens as visual evidence, with hierarchical sub-trajectories providing supervision at multiple temporal scales.Local visual tokens are masked during summarization, encouraging route reconstruction exclusively from episodic memory.Source: §1 · p. 2

Experiments

  • Under official R2R/RxR navigation-training trajectories, MEMVLN achieves state-of-the-art performance on most VLN-CE metrics and improves held-out physical success from 17% to 40%, reaching 47% after post-training.The physical evaluation uses 30 episodes in one scene: 24 demonstrations and six held-out tests.Source: §1 · p. 2

2 RELATED WORK

Related work spans navigation memory, pretrained vision-language models, and increasingly practical monocular streaming formulations. MEMVLN is positioned as a bounded, episode-local visual memory for streaming VLN.

Vision-Language Navigation

  • VLN is established as following language instructions through 3D environments, with foundational work defining the task across embodied-AI settings.The passage lists a broad set of subsequent foundational VLN studies, including Anderson et al. (2018b), Ku et al. (2020), and Krantz et al. (2020).Source: §2 · p. 2
  • Earlier VLN work includes data augmentation and simulation or real-world data scaling, while evaluation has expanded to increasingly complex scenarios.This paper focuses on continuous-environment VLN and learning navigation memory without massive data scaling.Source: §2 · p. 3

Vision-Language Models for Navigation

  • Pretrained VLMs have become a pathway toward general-purpose VLN, with work spanning topological, explicit 3D, and trajectory-level modeling, followed by monocular streaming and real-world deployment.Retaining long visual histories directly increases input context and computation, motivating dedicated memory mechanisms.Source: §2 · p. 3

Memory for Embodied Agents

  • Navigation memory ranges from recurrent hidden states and explicit observation or spatial representations to growing within-episode histories, cross-episode scene memory, and perceptual-cognitive memory for manipulation.MEMVLN instead targets a bounded episode-local visual bank with time/pose-grounded latent readout and memory-specific trajectory-summary supervision.Source: §2 · p. 3

3 METHOD

MEMVLN predicts actions from recent visual context, language, and compact episodic memory, with a frozen geometry encoder supplying spatial features. Its training combines navigation with masked trajectory summarization and multi-scale sub-trajectory supervision.

3.1 MODEL OVERVIEW

  • The frozen StreamVGGT encoder supplies camera-token representations, which a lightweight trainable adapter aligns and fuses with native visual features into local hybrid tokens.Recent observations form the short-term context; memory tokens complement it with accumulated visual history.Source: §3.1 · p. 4
  • MEMVLN augments a streaming VLM with a bounded memory bank and frozen geometry features, predicting actions from memory tokens, recent context, and the instruction.The short-term context supports local perception, while retrieved memory summarizes accumulated navigation history.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 MEMORY MODULE

  • The memory bank stores episode-specific visual features with timestep and action-derived relative pose; at capacity, it merges the most similar consecutive frames to keep the bank bounded.Pose is dead-reckoned from executed discrete actions, without ground-truth pose or external odometry at inference.Source: §3.2 · p. 4
  • Learnable queries and cross-attention resample the bank into a fixed-size memory-token set, while spatial downsampling reduces per-frame patch features and temporal and pose embeddings ground the representation.The design combines long-horizon evidence, bounded processing, and spatio-temporal grounding; queries remain instruction-agnostic while the downstream LLM interprets memory conditionally.Source: §3.2 · p. 5

3.3 DATA AND TRAINING

  • Training jointly optimizes navigation action prediction and trajectory summarization with a unified cross-entropy loss; navigation trajectories also merge consecutive atomic actions within a three-step window.During summarization, local visual tokens are masked so memory tokens alone provide visual evidence for reconstructing navigational semantics.Source: §3.3 · p. 5
  • FGR2R sub-instructions and waypoints are mapped to nearest points on continuous trajectories, creating sub-trajectory splits that supervise memory at multiple temporal scales.This supplements whole-trajectory summarization with finer-grained supervision.Source: §3.3 · p. 5

4 EXPERIMENTS

Under training restricted to official R2R/RxR trajectories, MEMVLN achieves state-of-the-art performance on most reported benchmark metrics. Ablations, long-horizon analyses, and a physical-robot evaluation examine its components and performance across settings.

4.1 MAIN SIMULATION RESULTS

  • On RxR, MEMVLN reaches 55.4% SR and 48.4% SPL, while nDTW rises from 59.1 to 64.6 and NE falls from 6.46 to 5.59 m.On R2R, it reaches 55.0% SR and 50.5% SPL, exceeding JanusVLN by 2.2 and 1.3 pp; OSR rises from 58.0% to 61.2%, with comparable NE (5.19 vs. 5.17 m).Source: §4.1 · p. 6, Table 2 · p. 7

4.2 ABLATION STUDIES

  • With default N_b=100 and N_m=300, R2R-CE SR improves by 4.1 pp and SPL by 3.3 pp over zero memory, with the best SR and SPL in both capacity ablations.Source: Table 4 · p. 7, §4.2 · p. 7
  • Memory alone improves success on both benchmarks; geometry adds spatial information, and trajectory-summary supervision adds 5.3 pp SR and 4.7 pp SPL over memory plus geometry on R2R.The full model exceeds geometry-only by 5.8 pp SR and 5.3 pp SPL on R2R, while lowering NE from 6.04 to 5.19 m.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 LONG-HORIZON ANALYSIS

  • Across instruction-complexity groups, memory raises SR and lowers NE, with its largest SR gain on very complex instructions: 40.5% to 46.5%, while NE falls from 9.07 to 8.01 m.The full model reaches 49.1% SR and 7.14 m NE, an 8.6 pp SR gain over baseline; memory also improves both metrics across instruction types.Source: Figure 3 · p. 8, §4.3 · p. 8
  • On backtracking episodes, memory raises SR from 43.2% to 50.9% and lowers NE from 7.86 to 6.91 m; the full model reaches 52.8% SR and 5.98 m NE.Memory also gains 5.2 pp SR on stop-condition subsets and 4.5 pp on landmark-passing subsets.Source: §4.3 · p. 8

4.4 REAL-WORLD EXPERIMENT

  • On G1, MEMVLN achieves 40.0% SR versus Base’s 16.7%, while reducing CR from 56.7% to 46.7%.Post-training with 24 demonstrations raises SR from 40.0% to 46.7%, with CR remaining at 46.7%.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 QUALITATIVE RESULTS

  • Qualitative examples show an RxR-CE route from a closet through a hallway to a staircase and a G1 rollout past chairs, stopping at a bucket beside a lamp.Figure 5 presents selected observations, actions, instructions, and top-down trajectories for RxR-CE and G1.Source: Figure 5 · p. 9, §4.5 · p. 9

4.4 REAL-WORLD EXPERIMENT

  • Fixed-input replay on one RTX 4090 gives mean policy latencies of 503.2 ms for Base and 691.0 ms for full MEMVLN; memory adds 300 LLM-input tokens independent of route length.Source: §4.4 · p. 9

5 CONCLUSION

MEMVLN combines bounded episodic memory with a frozen geometry encoder and trains memory through multi-granularity trajectory summarization. It reports strong VLN-CE results and a physical evaluation on held-out episodes from one scene.

5 CONCLUSION

  • MEMVLN achieves state-of-the-art performance on most reported VLN-CE metrics with navigation training restricted to official R2R/RxR trajectories.The physical evaluation uses one scene, with 24 demonstrations and six held-out episodes.Source: §5 · p. 9
  • Multi-granularity trajectory summarization trains bounded memory by masking local visual tokens and reconstructing routes exclusively from episodic memory.The memory is paired with a frozen pretrained geometry encoder.Source: §5 · p. 9
  • Simulation analyses characterize episodic memory’s utility across instruction complexities and navigation scenarios.Source: §5 · p. 9

6 LIMITATIONS AND FUTURE WORK

The paper identifies limitations in instruction-agnostic memory consolidation, training strategies, and the scope of its physical evaluation. It proposes task-aware consolidation, broader deployment settings, and safety-aware post-training as future directions.

6 LIMITATIONS AND FUTURE WORK

  • Instruction-agnostic retrieval and similarity-based consolidation may discard task-relevant observations during long episodes.The bounded memory bank does not use the instruction to guide these operations.Source: §6 · p. 9
  • The physical study tests six held-out episodes in one scene, reports substantial collision rates, and leaves cross-scene generalization untested.Source: §6 · p. 9
  • Broader training strategies for trajectory-summarization supervision remain under-explored.The paper names broader deployment settings and safety-aware post-training as future directions.Source: §6 · p. 9

A ADDITIONAL IMPLEMENTATION DETAILS

The implementation details specify data capture, sequential episode loading, memory-bank configuration, and simulation and real-world training protocols. The default memory uses bounded frame storage and a fixed number of readout tokens.

A ADDITIONAL IMPLEMENTATION DETAILS

  • Real-world post-training uses 24 robot-collected demonstrations and weights corrected non-stop actions at 3.0, accepted actions at 0.5, and stop actions at 2.0.The demonstrations use an RGB-D camera on the G1 robot; the correction-weighted objective focuses adaptation on states requiring human intervention.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 18
  • The streaming episode sampler assigns complete episodes to devices and feeds their steps sequentially to preserve chronological memory updates.This supports continuous memory streams alongside distributed training and task mixing.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 18
  • The default bank stores each observation’s visual feature, timestep, and action-derived relative pose, merging the most similar consecutive frames when capacity is exceeded.The defaults are N_f=100 patch tokens per frame, N_b=100 frame entries, and N_m=300 memory tokens.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 18
  • Simulation training freezes the visual and geometry encoders while adapting the LLM with LoRA and training the memory module and geometry adapter from scratch.The training protocol supplements the default configuration described in the implementation section.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 18

B INFERENCE LATENCY AND COMPUTATIONAL COST

The section measures fixed-input inference cost, describes the bounded memory-token interface, and reports memory-capacity ablations. The measured complete pipeline has higher policy latency than the Base policy under the tested setup.

B INFERENCE LATENCY AND COMPUTATIONAL COST

  • The latency diagnostic replays 102 fixed decisions from three frozen R2R traces on one RTX 4090, excluding sensor capture, environment stepping, and low-level control.Each variant is timed in two fresh processes after a 20-decision warm-up; the diagnostic does not compute navigation metrics.Source: B INFERENCE LATENCY AND COMPUTATIONAL… · p. 19, Table 8 · p. 19
  • Mean policy latency rises from 503.2 to 691.0 ms (37.3%), corresponding to a 1.45 Hz sequential high-level policy rate on the tested GPU.For the 16-frame subset, mean latency is 534.9 ms for Base and 731.7 ms for Full MEMVLN, a 36.8% increase.Source: Table 8 · p. 19, B INFERENCE LATENCY AND COMPUTATIONAL… · p. 19

C.1 MEMORY CAPACITY

  • The memory-capacity ablation covers R2R-CE and RxR-CE; its zero-capacity setting retains the geometry encoder and trajectory-summary supervision.The stated default is N_b=100, N_m=300, and N_f=100 patch tokens per stored frame; the supplied passage does not give ablation values.Source: Table 9 · p. 20, §C.1 · p. 20

B INFERENCE LATENCY AND COMPUTATIONAL COST

C.2 LONG-HORIZON ANALYSIS PROTOCOL

The protocol groups RxR-CE episodes by instruction complexity and by keyword-matched instruction types. It also identifies the real-world instruction and per-instruction result tables.

C.2 LONG-HORIZON ANALYSIS PROTOCOL

  • 3,669 RxR-CE Val-Unseen episodes are grouped into four punctuation-count bins: 1–3, 4–6, 7–10, and more than 10 marks.The bins contain 239, 819, 1,113, and 1,498 episodes, respectively.Source: §C.2 · p. 20
  • Back, Stop, and Pass subsets use case-insensitive keyword matching with word boundaries, and episodes may belong to multiple subsets.The subsets match backtracking, stopping or hallway terms, and passing or crossing terms; All denotes the complete set.Source: §C.2 · p. 20
  • Table 10 lists the real-world evaluation instructions by ID, and Table 11 reports per-instruction SR/CR across five repeated trials for each method.Bold identifies the best nonzero SR and lowest CR in each instruction row.Source: Table 11 · p. 21, Table 10 · p. 21

C.3 REAL-WORLD RESULTS

In the physical G1 evaluation, MEMVLN improves aggregate success over Base, and post-training raises success further while leaving collision rate unchanged. Instruction-level results remain mixed, and collisions remain a challenge.

C.3 REAL-WORLD RESULTS

  • All 30 instruction-conditioned episodes were collected in the same physical scene; 24 demonstrations were used for post-training and six distinct episodes were held out.Each held-out instruction was repeated five times per method; CR measures contact safety independently of task completion.Source: §C.3 · p. 21
  • Real-world inference runs on a host workstation with an RTX 4090 GPU, receiving robot-mounted RealSense D455 RGB observations over Wi-Fi and returning predicted actions to the G1.Source: §C.3 · p. 21
  • 46.7% SR is achieved by MemVLN-PT versus 16.7% for Base across 30 held-out trials; CR is 46.7% for both MemVLN and MemVLN-PT.MemVLN succeeds in 12 of 30 runs (40.0% SR), while MemVLN-PT succeeds in 14 of 30 (46.7% SR).Source: §C.3 · p. 21
  • MemVLN-PT improves SR over MemVLN on I3 and I5, leaves it unchanged on I2 and I4, and lowers CR on I2 and I6 while raising it on I1 and I3.All four methods obtain 0% SR on I4, which requires selecting the second door on the right after passing an intermediate door.Source: §C.3 · p. 21
  • MemVLN and MemVLN-PT both have a 46.7% CR, and post-training leaves aggregate CR unchanged despite improving aggregate SR.The authors identify contact-aware control and richer supervision for fine-grained spatial grounding as future work.Source: §C.3 · p. 21

C.4 MORE QUALITATIVE RESULTS

The qualitative results section presents selected observations and actions from multi-step navigation rollouts in simulation and from two real-world G1 rollouts, alongside their instructions.

C.4 MORE QUALITATIVE RESULTS

  • Figure 6 presents selected observations and actions from R2R-CE navigation rollouts.The supplied description does not report specific outcomes from the rollouts.Source: §C.4 · p. 22, Figure 6 · p. 23
  • Figure 7 presents selected observations and actions from RxR-CE navigation rollouts.The supplied description does not report specific outcomes from the rollouts.Source: §C.4 · p. 22, Figure 7 · p. 24
  • Figure 8 shows selected observations and actions from two G1 real-world navigation rollouts, together with their instructions.Source: Figure 8 · p. 25

Related papers

Loading 2610.35066v1…