PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation

Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, Gao Huang

arXiv:2508.19236v2cs.ROcs.CV

TL;DR

Mainstream VLA models overlook temporal dependencies important for long-horizon robotic manipulation. MemoryVLA introduces a cognition-inspired working-memory and Perceptual–Cognitive Memory Bank framework with a memory-conditioned diffusion action expert, achieving state-of-the-art performance across simulation and real-world tasks.

  • Problem

    Mainstream VLA models rely on current observations and overlook temporal dependencies in long-horizon robotic manipulation.

  • Method

    MemoryVLA uses VLM-derived perceptual and cognitive tokens as working memory, retrieves and consolidates historical context in a Perceptual–Cognitive Memory Bank, and conditions a diffusion action expert.

  • Results

    MemoryVLA consistently surpasses CogACT and π0 across 150+ tasks on three robots, including 71.9% on SimplerEnv-Bridge and a 26-point gain over CogACT on real-world long-horizon temporal tasks.

  • Takeaways & Limitations

    Temporal memory modeling supports MemoryVLA’s reported performance on challenging long-horizon robotic manipulation tasks.

Abstract

from arXiv · show

Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it and struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working memory to buffer short-lived representations for immediate control, while the hippocampal system preserves verbatim episodic details and semantic gist of past experience for long-term memory. Inspired by these mechanisms, we propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation. A pretrained VLM encodes the observation into perceptual and cognitive tokens that form working memory, while a Perceptual-Cognitive Memory Bank stores low-level details and high-level semantics consolidated from it. Working memory retrieves decision-relevant entries from the bank, adaptively fuses them with current tokens, and updates the bank by merging redundancies. Using these tokens, a memory-conditioned diffusion action expert yields temporally aware action sequences. We evaluate MemoryVLA on 150+ simulation and real-world tasks across three robots. On SimplerEnv-Bridge, Fractal, LIBERO-5 suites and Mikasa-Robo, it achieves 71.9%, 72.7%, 96.5%, and 41.2% success rates, respectively, all outperforming state-of-the-art baselines CogACT and pi-0, with a notable +14.6 gain on Bridge and +11.8 gain on Mikasa-Robo. On 12 real-world tasks spanning general skills and long-horizon temporal dependencies, MemoryVLA achieves 84.0% success rate, with long-horizon tasks showing a +26 improvement over state-of-the-art baseline. Project Page: https://shihao1895.github.io/MemoryVLA

1 INTRODUCTION

MemoryVLA addresses temporal dependencies in long-horizon robotic manipulation by combining cognitive-inspired memory with VLA action generation. It reports state-of-the-art results across simulation and real-world evaluations.

  • Motivation: Mainstream VLA models rely on current observations, overlooking temporal dependencies in long-horizon manipulation tasks.Push Buttons states can look nearly identical before and after pushing, requiring temporal modeling.
  • Approach: MemoryVLA combines working memory, a Perceptual–Cognitive Memory Bank, and a diffusion action expert to model long-horizon temporal dependencies.The framework uses VLM-derived perceptual and cognitive representations, memory retrieval and fusion, and consolidation of similar entries.
  • Results: 71.9% and 72.7% success rates on SimplerEnv Bridge and Fractal surpass CogACT by 14.6 and 4.6 points, respectively, while LIBERO reaches 96.5%.MemoryVLA also outperforms π0 on these evaluations.
  • Results: 41.2% success rate on Mikasa-Robo outperforms π0 by 11.8 points.The evaluation covers multiple tasks and reports the highest performance across tasks.
  • Results: 85% and 83% scores on real-world general and long-horizon temporal tasks exceed CogACT by 9 and 26 percentage points, respectively.The real-world evaluation spans 12 tasks across Franka and WidowX robots.

2 RELATED WORKS

Prior work has explored VLA models and temporal modeling, but temporal modeling remains incompletely explored in robotic manipulation. MemoryVLA’s architecture provides a memory-based approach to this gap.

  • Vision-Language-Action Models: VLA models have advanced through visual foundation models, large-scale datasets, and pretrained vision-language models.The related-work passage situates these developments within progress in robotic manipulation.
  • Temporal Modeling in Robotics: Temporal modeling has been extensively studied in computer vision and autonomous driving but remains not fully explored in robotic manipulation.This establishes the specific research gap addressed by MemoryVLA.
  • MemoryVLA: MemoryVLA encodes observations and instructions into tokens, retrieves historical context from a memory bank, and conditions a diffusion transformer to predict future actions.The architecture combines short-term working memory with long-term perceptual and cognitive memory.

3 METHOD

MemoryVLA maps current visual and language inputs to working-memory tokens, retrieves and fuses relevant temporal context from a Perceptual–Cognitive Memory Bank, and uses the result to generate future action sequences.

  • Problem formulation: MemoryVLA maps an RGB observation and language instruction to a sequence of future actions through an end-to-end policy.Each action contains relative translation, relative Euler-angle rotation, and a binary gripper state.
  • Vision-language cognition: The VLM encodes current inputs into perceptual and cognitive tokens that jointly form working memory.The perceptual stream represents visual information, while the cognitive stream processes visual tokens with the language instruction.
  • Perceptual-cognitive memory bank: The PCMB complements present-timestep working memory by storing fine-grained perceptual details and high-level cognitive semantics across up to L entries per stream.The memory bank is introduced because working memory alone lacks temporal dependencies.
  • Memory retrieval: At each timestep, working-memory tokens query the PCMB with attention and timestep positional encoding to retrieve decision-relevant historical information.Separate perceptual and cognitive memory tensors produce retrieved embeddings through Transformer layers.
  • Memory gate fusion: Learned gates adaptively fuse retrieved perceptual and cognitive embeddings with current working-memory representations.Sigmoid activation and element-wise multiplication produce memory-augmented perceptual and cognitive features.
  • Memory consolidation: When stored entries exceed capacity, the system merges the most similar adjacent entries in each stream by averaging their vectors to reduce redundancy.The consolidated representations are both passed to the action expert and updated into the PCMB.
  • Memory-conditioned action expert: A memory-conditioned diffusion action expert uses the augmented representations to predict future action sequences with semantic and perceptual attention.The expert predicts T = 16 actions and conditions denoising through cognitive and perception-attention layers.

4 EXPERIMENTS

MemoryVLA is evaluated across simulation and real-world manipulation settings spanning multiple robots, benchmarks, tasks, and variations. Across these evaluations, it consistently outperforms the reported baselines, with especially large gains on Bridge, Mikasa-Robo, and long-horizon real-world tasks.

  • Experimental Setup: The evaluation covers 3 robots, 6 benchmarks, more than 150 tasks, and more than 500 variations across simulation and real-world settings.Experiments include four simulation benchmarks, general and long-horizon temporal real-world evaluations, and robustness and generalization tests.
  • SimplerEnv: 71.9% average success on SimplerEnv-Bridge is a +14.6-point gain over CogACT-Large and surpasses π0.Bridge uses a WidowX robot, with results reported from repeated task trials.
  • SimplerEnv: 72.7% overall success on SimplerEnv-Fractal improves CogACT by +4.6 points across Visual Matching and Visual Aggregation settings.The setting-level averages are 77.7% for Visual Matching and 67.7% for Visual Aggregation.
  • LIBERO: 96.5% overall success on LIBERO improves CogACT by +3.3 points and surpasses π0 across five suites.Suite averages range from 93.4% on Long-10 to 98.4% on Spatial and Object, using only third-person RGB inputs.
  • Mikasa-Robo: MemoryVLA achieves the highest Mikasa-Robo performance, averaging an 11.8% improvement over the previous state of the art.The largest reported task-specific improvement is +41.0% on ShellGameTouch.
  • Real-World Evaluation: On real-world tasks, MemoryVLA scores 85% on general tasks and 83% on long-horizon temporal tasks, exceeding CogACT by +9 and +26 points.Evaluations use Franka and WidowX robots across six general and six long-horizon temporal tasks.
  • Ablation Studies: Ablations on SimplerEnv-Bridge report 71.9% with combined perceptual and cognitive memory, memory length 16, gate fusion, and token-merge consolidation.Cognitive-only and perceptual-only memory reach 63.5% and 64.6%, while FIFO consolidation reaches 66.7%.

5 CONCLUSION

MemoryVLA combines cognitive-science-inspired memory with a diffusion action expert to model temporal dependencies in robotic manipulation. Across simulation and real-world evaluations, it consistently outperforms CogACT and π0 while showing robustness under diverse OOD conditions.

  • MemoryVLA uses a hippocampus-like Perceptual–Cognitive Memory Bank with working memory and a memory-conditioned diffusion action expert for temporally aware manipulation.Future directions include memory reflection for embedding-space reasoning and lifelong memory consolidation for scalable generalization.
  • Across 150+ tasks with 500+ variations on three robots, MemoryVLA consistently surpasses CogACT and π0 across SimplerEnv, LIBERO, and real-world evaluations.
  • MemoryVLA demonstrates strong robustness and generalization under diverse real-world OOD conditions, including unseen backgrounds, distractors, objects, containers, lighting, and occlusions.The evaluation covers Pick Place Order and Clean Restaurant Table.

B.2 SIMULATION EVALUATION

Simulation experiments evaluate MemoryVLA under OOD shifts in pick-and-move and hinge-like manipulation tasks. Performance remains strong under moderate shifts but degrades most under unseen camera views.

  • Pick-and-move tasks: MemoryVLA maintains strong performance across most simulated OOD shifts in Pick Coke Can and Move Near, with the largest degradation under unseen camera views.The variants include unseen backgrounds, distractors, lighting, textures, and camera views.
  • Pick-and-move tasks: 42.0% is MemoryVLA’s success rate for Pick Coke Can under unseen camera views, compared with 92.0% in the base setting.
  • Hinge-like object manipulation: MemoryVLA generalizes well under moderate shifts in Open/Close Drawer and Place Apple Into Drawer, but performance drops notably when camera views change.

C.2 TRAINING DATA

The evaluation uses multiple simulation benchmarks and real-world data collected on Franka and WidowX robots, with task-specific training, sampling, preprocessing, and evaluation protocols. Memory retrieval is assessed through temporally ambiguous manipulation cases.

  • Simulation benchmark data: BridgeData v2 contains roughly 60,000 teleoperated WidowX trajectories covering language-conditioned tabletop skills such as picking, placing, pushing, stacking, and folding.
  • Simulation benchmark data: RT-1 contains roughly 130,000 episodes spanning 700+ tasks, while LIBERO provides 130 Franka simulation tasks across five suites with 50 demonstrations each.
  • Simulation benchmark data: Mikasa-Robo comprises five memory-dependent manipulation tasks with 250 demonstrations per task and uses Δend-effector control.
  • Real-world data: Real-world demonstrations use fixed third-person RGB views on Franka and WidowX robots, with 50–150 demonstrations per general task and 200–300 per long-horizon temporal task.Frames are downsampled to 224 × 224 and retained based on end-effector translation, orientation change, or a maximum frame gap.
  • Training and evaluation protocols: Training and evaluation use task-specific step counts, grouped or sequential episode sampling, memory lengths, validation schedules, and benchmark-specific trial protocols.Real-world memory length is 16 for general tasks and 256 for long-horizon temporal tasks; LIBERO and Mikasa-Robo do not use action ensembling.
  • Memory retrieval case studies: MemoryVLA attends to earlier decisive frames in Change Food and Shell Game Touch when the current observation cannot identify the next action.These cases test retrieval of motion trends and previously revealed object locations rather than redundant visual history.

F DATA LENGTH STATISTICS

Table 9 summarizes action-length distributions across simulation and real-world task suites using maximum, minimum, median, and average statistics. Real-world results additionally include motion-filtered statistics.

  • Action-length statistics cover SimplerEnv Bridge and Fractal, LIBERO Spatial/Object/Goal and Long-10/Long-90, and real-world General and Temporal suites.
  • Real-world filtered statistics remove frames with negligible end-effector motion, defined by translation below 1 cm and rotation below 0.4 rad between consecutive frames.

G ADDITIONAL ABLATION STUDY

Additional ablations examine memory length, cognitive-token count, fusion and consolidation strategies, while efficiency tests assess inference overhead. The results support a moderate memory length, limited benefit from additional cognitive tokens, and lightweight deployment cost.

  • Memory Length: 256 memory entries achieve the best performance on the representative Clean Table & Count real-world temporal task.Temporal tasks have substantially longer episodes than the other evaluated settings, motivating larger memory lengths.
  • Cognitive Tokens: Increasing cognitive tokens from 1 to 4 does not improve performance.The authors attribute this to the semantic capacity of a single 4096-dim EOS token.
  • Fusion and Consolidation: Additional ablations extend fusion-type and consolidation-strategy comparisons to LIBERO-Long-90 and a real-world long-horizon task.
  • Combined Ablations: The SimplerEnv-Bridge ablations vary memory type, length, retrieval, fusion, and consolidation, with gray rows marking the default configuration.
  • Inference Efficiency: 0.194 s latency and 82.5 Hz throughput on RTX 4090 represent a 3.6% overhead increase over the baseline.Memory usage is 16.6 GB, only 0.8 GB above the baseline.
  • Inference Efficiency: The lightweight memory module keeps retrieved memory size and cross-attention cost small through single-token cognitive entries, perceptual compression, and redundancy merging.

I ZERO-SHOT TASK GENERALIZATION

The paper adds zero-shot task-generalization experiments and temporal-context baseline comparisons across Mikasa-Robo, LIBERO, and SimplerEnv. MemoryVLA achieves good unseen-task generalization and higher success rates than the included temporal-context baselines.

  • Zero-Shot Generalization: MemoryVLA achieves good zero-shot task generalization from Apple To Basket to Eggplant To Basket, Blush To Basket, and Apple To Plate.
  • Temporal-Context Comparisons: Temporal-context baseline comparisons cover Mikasa-Robo, LIBERO, and SimplerEnv for fair evaluation across benchmarks.
  • Mikasa-Robo: MemoryVLA substantially outperforms the reproduced CronusVLA baseline across Mikasa-Robo tasks.CronusVLA aggregates multi-frame VLM features through a sliding-window module.
  • LIBERO: MemoryVLA obtains higher average success rates than TTF-VLA, TraceVLA, 4D-VLA, CronusVLA, and MAP-VLA on LIBERO.
  • SimplerEnv: MemoryVLA achieves higher success rates than TraceVLA, RoboVLMs, and CronusVLA on SimplerEnv.

K TASK DETAILS

The evaluation spans diverse simulation and real-world manipulation suites, including general skills and long-horizon tasks whose correct actions depend on temporal information. Benchmark descriptions specify task variations, visual conditions, and suite composition.

  • Benchmark Scope: The benchmark design records task templates, variation types, and variation counts to characterize evaluation diversity and difficulty.
  • Real-World Tasks: The 12 real-world tasks are divided into General and Long-horizon Temporal suites.
  • General Tasks: General real-world tasks test insertion, placement, stacking, alignment, balance, grasp stability, and constrained manipulation.
  • Long-Horizon Temporal Tasks: Long-horizon tasks test ordered memory, temporal sequencing, reversible-action tracking, and decisions requiring information from earlier steps.Change Food can make the next action ambiguous from a single frame, requiring recall of earlier steps.
  • SimplerEnv: SimplerEnv contains Bridge and Fractal suites, with Bridge covering four WidowX tabletop tasks and Fractal evaluating four Google-robot tasks under visual-matching and visual-aggregation protocols.
  • LIBERO: LIBERO comprises Spatial, Object, Goal, Long-10, and Long-90 suites covering varied placements, objects, operations, and extended tasks.

M.1 REAL-WORLD EVALUATION

Qualitative evaluation visualizes MemoryVLA rollouts across real-world general and long-horizon tasks, SimplerEnv, and all LIBERO suites. The figures provide representative trajectories spanning the paper’s evaluation settings.

  • Real-World Long-Horizon Tasks: Figures 13 and 14 show representative real-world rollouts on long-horizon temporal tasks, including Seq Push Buttons, Change Food, Guess Where, Clean Table & Count, Pick Place Order, and Clean Restaurant Table.
  • Real-World General Tasks: Figure 15 presents qualitative results on real-world general tasks such as Insert Circle, Egg in Pan, Egg in Oven, Stack Cups, Stack Blocks, and Pick Diverse Fruits.
  • Simulation Tasks: Figures 16 and 17 visualize representative results on SimplerEnv-Bridge and SimplerEnv-Fractal, while Figure 18 spans all five LIBERO suites.
Loading 2508.19236v2…