PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan

arXiv:2607.07608v1cs.ROcs.CV

TL;DR

VLA models struggle with long-horizon manipulation, while existing memory methods leave historical experience outside the model’s native latent reasoning space. LaMem-VLA weaves dual-scale latent memory tokens into VLA reasoning, achieving 97.6% average success on LIBERO and 73.9% average success on SimplerEnv-Bridge.

  • Problem

    Existing memory-augmented VLAs leave historical experience outside the model’s native continuous latent reasoning space, leaving open how memory can be represented and consumed within that space.

    Source: §1 · p. 2

  • Method

    LaMem-VLA organizes history into complementary short- and long-term vaults, retrieves context-relevant evidence, compresses it into latent memory tokens, and weaves them into the VLA embedding sequence.

    Source: §1 · pp. 2–3

  • Results

    97.6% average success on LIBERO, 1.1 points above MemoryVLA, and 73.9% average success on SimplerEnv-Bridge, 16.6 points above CogACT.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors report that dual-scale latent memory improves VLA robustness beyond policy-side memory conditioning, especially when actions depend on task progress and historical cues.

    Source: §1 · p. 3

    Empirical validation is limited to simulated environments; real-world robot experiments are planned for a future version.

    Source: §1 · p. 3

Abstract

from arXiv · show

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

1 INTRODUCTION

LaMem-VLA addresses the difficulty of long-horizon manipulation by representing historical experience as latent memory that is retrieved and woven into VLA reasoning. It separates memory into short-term visual evidence and long-term semantic and action-continuity evidence, both consumed as latent tokens.

1 INTRODUCTION

  • Mainstream VLAs rely primarily on current observations, leaving them prone to struggle with long-horizon manipulation requiring transitions, completed steps, and the current task phase.LaMem-VLA frames the alternative as context-native robotic memory stored, retrieved, and consumed in the model’s continuous embedding space.Source: §1 · p. 2
  • Prior methods either extend the observation window, with growing costs and a fixed memory ceiling, or retrieve external-bank history as auxiliary policy context that remains outside native VLA reasoning.The separation prevents historical evidence from being interleaved with multimodal reasoning before action formation.Source: §1 · p. 2
  • Short-term memory preserves visually grounded evidence such as object locations and state changes, while long-term memory preserves task progress, contextual semantics, and action continuity; both become latent tokens.Their distinction is in provenance and function, not representation form.Source: §1 · p. 2
  • LaMem-VLA uses a curator, seeker, condenser, and weaver to organize dual memory vaults, retrieve relevant evidence, compress it into fixed-length latent tokens, and insert those tokens before action queries are resolved.The resulting memory-grounded queries condition a diffusion-based action expert [34] to generate temporally aware action sequences.Source: §1 · p. 2
  • LaMem-VLA achieves 97.6% average success on LIBERO, beating MemoryVLA [23] by 1.1 points and CogACT [15] by 4.4 points, and π0 [1] by 3.5 points on the first four suites.On SimplerEnv-Bridge, it achieves 73.9% average success, surpassing CogACT [15] by 16.6 points and π0 [1] by 4.7 points. Validation is currently limited to simulated environments; real-world experiments are planned for a future version.Source: §1 · pp. 2–3

2 RELATED WORK

Prior VLAs use single-stream or hierarchical policies but mainly predict from current observations and instructions; memory-augmented approaches address history in several ways, while LaMem-VLA integrates retrieved experience into native latent reasoning.

Vision-Language-Action (VLA) Models

  • VLA policies divide into single-stream action-token prediction and hierarchical high-level reasoning with a separate trajectory-generating policy [40] [41] [42] [43] [3] [48] [49].Single-stream models autoregressively predict discretized actions; hierarchical models use generative policies such as diffusion or flow matching [1] [15] [50] [9].Source: §2 · p. 3
  • Mainstream VLAs rely on current observations and instructions, leaving them without explicit task-progress awareness for long-horizon manipulation.The paper characterizes these policies as Markovian and says their limited historical context constrains robustness on long-horizon tasks.Source: §2 · p. 3

Memory Mechanisms for Robotic Control

  • History methods expand observation windows [18] [19] [20] [21] [22] or compress experience into proxy representations [51] [52] [54] [55], trading context costs or detail for temporal awareness.Windowed methods incur increasing compute and memory costs and discard evidence beyond a fixed horizon; aggressive abstraction can lose perceptual detail and semantic context.Source: §2 · p. 3
  • External memory banks retrieve compact historical evidence [23] [24] [25], but weak current cues can undermine retrieval, and policy-side conditioning limits direct participation in token-level reasoning.Retrieved evidence is commonly used as auxiliary context rather than integrated into native reasoning and action formation.Source: §2 · pp. 3–4
  • LaMem-VLA retrieves evidence from complementary short-term visual and long-term semantic vaults, distills it into latent memory tokens, and interweaves them with visual, language, and action tokens.This context-native design lets perceptual evidence and long-range task progress jointly shape VLA reasoning without a growing context.Source: §2 · p. 4

3 METHOD

LaMem-VLA addresses the short-horizon bias of observation-conditioned VLA policies by storing historical experience in dual-scale vaults, reconstructing it as latent memory tokens, and weaving those tokens into action reasoning. Its pipeline separates memory storage, context-aware retrieval, fixed-length reconstruction, and action generation.

3.1 PROBLEM FORMULATION

  • The task is modeled as language-conditioned Markovian control: the policy takes instruction and current visual observation as input and predicts a chunk of 7-DoF end-effector actions.Each action comprises relative translation, relative rotation, and gripper state.Source: §3.1 · p. 4

3.2 LAMEM-VLA MODEL ARCHITECTURE

  • LaMem-VLA weaves short- and long-term latent memory tokens into the VLA sequence so historical experience participates directly in reasoning and action generation.The framework uses a curator, seeker, condenser, and weaver to bring memory into the model’s native embedding space rather than supplying it only as auxiliary policy-side context.Source: §3.2 · p. 5

3.3 LATENT MEMORY CURATOR

  • The curator stores compressed visual evidence in a short-term key-value vault and action hidden states in a long-term vault to preserve task-progress and action-continuity information.When vault capacity is exceeded, adjacent redundant units are consolidated; the short-term stream selects a pair by cosine similarity and averages its keys and values, and the same updating strategy applies to the long-term vault.Source: §3.3 · pp. 5–6

3.4 LEARNING TO SYNTHESIZE LATENT MEMORY WITH DUAL-SCALE VAULT

  • The seeker derives a shared retrieval query from current multimodal cognition and uses it to retrieve context-relevant units from both vaults, passing them onward as visual and semantic evidence.Short-term units are ranked by cosine similarity, while long-term units are ranked by comparing their mean-pooled representations with the mean-pooled query; retrieved evidence is not inserted verbatim into the VLA sequence.Source: §3.4 · p. 6
  • The condenser reconstructs retrieved evidence into query-conditioned, fixed-length short- and long-term memory tokens in the same embedding space as VLA reasoning.This bounded interface makes injected memory length independent of how many units were retrieved.Source: §3.4 · pp. 6–7

3.5 INTERWEAVING MEMORY INTO ACTION REASONING IN LATENT SPACE

  • The weaver injects both memory streams into the VLA input sequence, where self-attention combines them with observation, instruction, and action queries before a diffusion expert decodes continuous action chunks.The expert performs conditional denoising on action chunks using the memory-grounded action tokens.Source: §3.5 · p. 7

4 EXPERIMENT

Experiments evaluate LaMem-VLA on SimplerEnv-Bridge and LIBERO, then ablate its memory sources, integration path, and retrieval budget. The reported results favor dual latent memory and direct latent-native integration.

4.1 IMPLEMENTATION DETAILS

  • LaMem-VLA uses a 7B Prismatic VLM and an approximately 300M-parameter diffusion action expert to predict continuous 7-DoF actions in chunks of 16.It takes one 224 × 224 third-person RGB observation and an instruction; each memory vault holds up to 16 units, with top-8 retrieval and 8 short-term plus 4 long-term latent tokens. Inference uses DDIM [63] with 10 denoising steps.Source: §4.1 · p. 7

4.2 SIMULATED EVALUATION ON SIMPLERENV-BRIDGE

  • 73.9% is LaMem-VLA’s highest average success rate on SimplerEnv-Bridge, a 16.6 gain over CogACT [15] and above π0 [1] and SemanticVLA.Per-task success rates are 83.3% on Put Spoon on Towel, 75.0% on Put Carrot on Plate, 41.7% on Stack Cube, and 95.8% on Put Eggplant in Basket. Evaluation reports average success across 24 trials per task.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 SIMULATED EVALUATION ON LIBERO

  • 97.6% is LaMem-VLA’s best overall LIBERO success rate, 1.1 points above MemoryVLA [23] and 4.4 above CogACT [15].LaMem-VLA scores 98.8% on Spatial, 99.0% on Object, 97.2% on Goal, 95.8% on Long-10, and 97.0% on Long-90. It also improves over MemoryVLA by 2.4 points on Long-10 and 1.4 on Long-90, without the extra proprioceptive and wrist-camera inputs used by starred methods; evaluation averages 50 rollouts per task.Source: §4.3 · p. 8, Table 2 · p. 9

4.4 ABLATION STUDY

  • 73.9% on SimplerEnv and 97.0% on LIBERO-90 are the full dual-memory results, versus 57.3% and 92.1% when both memory streams are removed.The ablation attributes complementary roles to short-term visual evidence about object states and transient cues, and long-term semantic memory about task progress and action continuity.Source: Table 3 · p. 9, §4.4 · p. 9
  • 73.9% on SimplerEnv and 97.0% on LIBERO-90 are the best results with latent-native memory, versus 71.9% and 94.8% with external policy-side memory conditioning.Raw retrieved evidence also improves over the memory-free baseline of 57.3% and 92.1%, but remains below LaMem-VLA; the authors suggest uncompressed retrieval can add redundant tokens.Source: Table 4 · p. 9, §4.4 · p. 9
  • K = 8 gives the best retrieval-budget results: 73.9% on SimplerEnv and 97.0% on LIBERO-90, compared with 71.8% and 96.2% at K = 12.Increasing K from 2 to 4 raises success from 66.7% to 70.8% on SimplerEnv and from 94.4% to 95.9% on LIBERO-90; the paper attributes the K = 12 decline to redundant or weakly related evidence and added compression burden.Source: Table 5 · p. 9, §4.4 · pp. 9–10

5 CONCLUSION

LaMem-VLA makes robotic history part of the VLA’s native latent context, using complementary short-term visual and long-term semantic memory to improve temporal awareness. Experiments in two simulation platforms demonstrate its superior performance.

5 CONCLUSION

  • LaMem-VLA is a dual latent memory framework designed to reduce VLA models’ temporal short-horizon bias.Its core idea is to make robotic history part of the model’s native latent context rather than an external condition attached after multimodal reasoning.Source: §5 · p. 10
  • LaMem-VLA represents retrieved short-term visual and long-term semantic history as compact latent memory tokens in the VLA input sequence.This lets past visual evidence and task-progress cues interact with the current observation, language instruction, and action queries through the same embedding space.Source: §5 · p. 10
  • Experiments in two simulation platforms demonstrate LaMem-VLA’s superior performance.The results suggest that context-native latent memory is a practical direction for building VLA systems with stronger temporal awareness.Source: §5 · p. 10

Related papers

Loading 2607.07608v1…