Source-linked AI summary
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, Shuicheng Yan
TL;DR
Long-horizon robotic manipulation remains difficult because VLA models and existing memory approaches do not natively integrate historical experience into latent reasoning. LaMem-VLA weaves short- and long-term latent memory into VLA inputs, achieving higher average success across LIBERO and SimplerEnv than baselines.
Problem
VLA models struggle with long-horizon manipulation, while existing memory approaches keep historical experience outside the VLA’s native latent reasoning space.
Method
LaMem-VLA organizes history into short-term and long-term vaults, retrieves relevant evidence, condenses it into fixed-length latent tokens, and weaves them into VLA reasoning.
Results
97.6% average success on LIBERO and 73.9% on SimplerEnv-Bridge, surpassing CogACT by 4.4 and 16.6 points, respectively.
Takeaways & Limitations
The results support representing robotic history as context-native latent memory that participates directly in multimodal reasoning and action generation.
Takeaways & Limitations
Empirical validation is limited to simulated environments, with real-world robot experiments left for future work.
Abstract
from arXiv · showhide
Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.
LaMem-VLA
The LaMem-VLA section identifies the paper’s authors, led by Hongyu Qu and Jianzhe Gao.
- Hongyu Qu is listed as the first author.
- Jianzhe Gao is listed as the second author.
- The remaining listed authors are Xiaobin Hu, Shaohuan Yang, Xinlei Yu, Rui Yan, Wenguan Wang, Xiangbo Shu, and Shuicheng Yan.
1 INTRODUCTION
LaMem-VLA addresses the difficulty of long-horizon manipulation by treating historical experience as context-native latent memory that is stored, retrieved, and consumed within the VLA embedding space. It organizes history into complementary short-term and long-term vaults and weaves dual-scale memory into VLA reasoning for action generation, achieving strong results on LIBERO and SimplerEnv-Bridge.
- Motivation: VLA models struggle with long-horizon manipulation because current observations omit transitions, completed operation steps, and the task’s current phase.
- Limitations of Existing Methods: Existing memory-augmented VLAs use historical frames, video sequences, or auxiliary memory banks, incurring context-length costs and keeping memory outside native latent reasoning.
- LaMem-VLA: LaMem-VLA represents, retrieves, and consumes historical experience as context-native latent memory in the same continuous embedding space used for perception, language understanding, reasoning, and action formation.
- LaMem-VLA: LaMem-VLA organizes robotic history into complementary short-term and long-term memory vaults, then weaves dual-scale memory into VLA reasoning through four coordinated modules.
- Experiments: 97.6% average success on LIBERO exceeds MemoryVLA [23] by 1.1 points, CogACT [15] by 4.4 points, and π0 [1] by 3.5 points on the first four suites.
- Experiments: 73.9% average success on SimplerEnv-Bridge surpasses CogACT [15] by 16.6 points and π0 [1] by 4.7 points.
2 RELATED WORK
Related work frames VLA models as visual-language-to-action policies with single-stream and hierarchical designs, while existing memory mechanisms address Markovian limitations through temporal context or retrieval. These approaches can leave retrieval unreliable and historical experience outside native token-level reasoning, motivating latent-memory integration.
- Vision-Language-Action (VLA) Models: VLA models map visual observations and language instructions to robot actions and are broadly categorized as single-stream or hierarchical architectures.Single-stream models directly discretize continuous actions.
- Memory Mechanisms for Robotic Control: Mainstream VLA policies assume Markovian decision-making, predicting actions mainly from current observations and lacking explicit awareness of task progress.Existing historical-context methods include temporally extended observations interleaved with language tokens.
- Memory Mechanisms for Robotic Control: Retrieval-based memory methods use the current query to select compact task-relevant evidence from an auxiliary memory bank for action prediction.The current observation and language instruction drive memory access and policy conditioning.
- LaMem-VLA: LaMem-VLA organizes historical experience into dual memory vaults, retrieves task-relevant evidence, compresses it into fixed-length latent memory tokens, and integrates it with multimodal inputs.The framework uses a vision-language encoder before memory retrieval and condensation.
- Memory Mechanisms for Robotic Control: Query-driven retrieval can become unreliable when present observations provide weak cues or contain visually similar distractors.This limitation arises because memory access is driven primarily by the current query.
3 METHOD
LaMem-VLA addresses the short-horizon bias of Markovian VLA policies by storing, retrieving, and reconstructing historical experience as dual-scale latent memory native to VLA reasoning. Its curator, seeker, condenser, and weaver integrate short-term visual and long-term task-progress evidence with current multimodal inputs for action generation.
- Motivation: Markovian VLA policies predict action chunks from the current instruction and visual observation, making long-horizon control vulnerable to missing historical transitions, subtasks, and progress cues.Each action is a 7-DoF end-effector control vector, but the supplied formulation emphasizes dependence on instantaneous inputs.
- Memory Vaults: LaMem-VLA organizes historical experience into complementary short-term and long-term vaults, then retrieves and reconstructs it as compact latent memory tokens.The short-term vault stores compressed visual evidence as key-value units, whereas the long-term vault stores action hidden states encoding task progress and action continuity.
- Latent Memory Condenser: The condenser transforms retrieved short-term and long-term evidence into query-conditioned latent tokens in the same C-dimensional embedding space as VLA reasoning.Because the reconstructed tokens have fixed length, injected memory remains independent of retrieval size while retaining evidence from both vaults.
- Latent Memory Weaver: The weaver prepends latent short-term and long-term memory tokens to current image and language tokens, after which action tokens are passed to a diffusion-based action expert.This native latent integration lets historical experience participate directly in VLA reasoning rather than serving as raw auxiliary policy context.
- Latent Memory Seeker: The seeker builds a context-aware query from current visual-linguistic cognition and uses it to retrieve Top-K units from both memory vaults without inserting raw memories into the VLA sequence.A masked query builder preserves the original multimodal hidden states, and cosine-similarity retrieval produces short-term visual and long-term semantic evidence.
4 EXPERIMENT
LaMem-VLA achieves strong manipulation performance on SimplerEnv-Bridge and LIBERO, outperforming reported baselines. Ablations show that dual-scale latent memory, latent-native integration, and a moderate retrieval budget are important for performance.
- SimplerEnv-Bridge: LaMem-VLA reaches 73.9% average success on SimplerEnv-Bridge, a 16.6-point gain over CogACT and above π0 and SemanticVLA.Per-task success rates are 83.3% on Put Spoon on Towel, 75.0% on Put Carrot on Plate, 41.7% on Stack Cube, and 95.8% on Put Eggplant in Basket.
- LIBERO: LaMem-VLA achieves 97.6% average success across LIBERO’s five suites, improving over MemoryVLA by 1.1 points, CogACT by 4.4 points, and π0 by 3.5 points on the first four suites.The LIBERO evaluation uses a Franka robot across Spatial, Object, Goal, Long-10, and Long-90 suites.
- Dual-scale latent memory: The full dual-scale memory design performs best, reaching 73.9% on SimplerEnv and 97.0% on LIBERO-90, while removing both memory streams causes the largest degradation.The ablation compares full memory with removing short-term memory, long-term memory, or both streams.
- Latent-native memory integration: Latent-native memory integration outperforms policy-side conditioning: the memory-free baseline scores 57.3% on SimplerEnv and 92.1% on LIBERO-90, versus 71.9% and 94.8% with external memory conditioning.The study also evaluates directly feeding raw retrieved evidence to the action policy.
- Retrieved memory units: A retrieval budget of K = 8 yields the best performance, reaching 73.9% on SimplerEnv and 97.0% on LIBERO-90, whereas K = 12 reduces results to 71.8% and 96.2%.Increasing K from 2 to 4 improves SR from 66.7% to 70.8% on SimplerEnv and from 94.4% to 95.9% on LIBERO-90.
5 CONCLUSION
LaMem-VLA is a dual latent memory framework that reduces the temporal short-horizon bias of vision-language-action models by integrating robotic history into their native latent context. It uses complementary short-term visual and long-term semantic memory represented as compact latent tokens within the VLA input sequence.
- 5 CONCLUSION: LaMem-VLA reduces the temporal short-horizon bias of vision-language-action models through a dual latent memory framework.Its central design makes robotic history part of the model’s native latent context rather than an external condition added after multimodal reasoning.
- 5 CONCLUSION: Complementary short-term visual memory and long-term semantic memory organize the retrieved robotic history.The framework represents this retrieved history as compact latent memory tokens.
- 5 CONCLUSION: The compact latent memory tokens are inserted directly into the VLA input sequence.This places historical experience within the model’s native latent context during reasoning.