SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

Cheng Yin; Wang Xu; Junpeng Yang; Sikyuen Tam; Hanyu Liu; Yuan Yao; Xiangrui Zeng; Junbo Cui; Yequan Wang; Zhouping Yin; Yankai Lin

Source-linked AI summary

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

Cheng Yin, Wang Xu, Junpeng Yang, Sikyuen Tam, Hanyu Liu, Yuan Yao, Xiangrui Zeng, Junbo Cui, Yequan Wang, Zhouping Yin, Yankai Lin

arXiv:2609.05533v1Sep 2, 2026cs.CVcs.LGcs.RO

TL;DR

Long-horizon manipulation requires access to observations from minutes earlier, while existing memory mechanisms commit to what information to retain before future needs are known. SimpleMemVLA keeps sampled history intact as timestamped native video and uses generated-sub-task hidden states to condition a flow-matching action head. It achieves state-of-the-art results across four memory benchmarks, outperforms matched alternatives, and shows evidence that its policy reads visual history.

Problem
Long-horizon manipulation is partially observable, but retrieval, compression, and recurrent-state mechanisms must decide what to retain before future decisions reveal which evidence matters.
Method
SimpleMemVLA uses intact sampled history in the backbone’s pretrained timestamped-video format and routes generated-sub-task hidden states to a standard flow-matching action head.
Results
SimpleMemVLA sets a new state of the art on four memory benchmarks; on RoboMME it reaches 88.3% versus 31.5%, 22.6%, and 20.6% for matched retrieval, compression, and recurrent-state rebuilds.
Takeaways & Limitations
Native video context can serve as an effective memory interface for long-horizon VLA control without a dedicated memory module or loss on evaluated general-purpose control suites.
Takeaways & Limitations
Exact streaming’s background prefill and key–value cache continue to grow with accumulated history, limiting sustainable decision rate as episodes lengthen.

Abstract

from arXiv · show

Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earlier. Existing memory mechanisms: retrieval banks, learned compressors, recurrent states must decide what to keep from the past before knowing what a future decision will require. This was motivated by the assumption that minute-scale history is too large to process directly, which modern VLM backbones no longer make true. In this work, we introduce SimpleMemVLA, a VLA without a dedicated memory module. It keeps the sampled history intact and passes it to the backbone in the timestamped video format the backbone was pretrained to process; the hidden states of a generated sub-task then form the only channel from history to a standard flow-matching action head. Since consecutive decisions share most of their history, prefilling the shared prefix during action execution keeps latency close to a single-frame VLA. SimpleMemVLA sets a new state of the art on four memory benchmarks without cost on general-purpose control. Holding the backbone and training setup fixed, it outperforms retrieval, compression and recurrent-state mechanisms by a wide margin, and causal interventions confirm that the policy genuinely reads its history. Code available at https://github.com/wadeKeith/SimpleMemVLA

1 INTRODUCTION

Long-horizon manipulation requires policies to use observations from minutes earlier, but existing memory mechanisms commit to what to retain before future needs are known. SimpleMemVLA instead exposes intact timestamped history to the pretrained video backbone and achieves strong memory-task results without dedicated memory machinery.

Motivation: Long-horizon manipulation is partially observable because the next action may depend on observations from minutes earlier.General-purpose VLAs commonly condition actions on a single image or sub-second observation window.
Motivation: Retrieval, compression, and recurrent-state mechanisms may omit, distort, or overwrite evidence before a future decision reveals its relevance.The paper calls this premature information selection write-time commitment.
Motivation: A 60 s history occupies roughly 5.6k tokens of a 262k-token context window, so minute-scale history need not be compressed into a separate memory representation.Modern VLM backbones can process temporally ordered video through native multimodal interfaces.
Approach: SimpleMemVLA keeps sampled history intact in timestamped video format, lets the backbone identify relevant evidence, and routes generated-sub-task hidden states to a standard flow-matching action head.It uses no dedicated memory module and makes the generated sub-task the only channel from history to actions.
Results: 88.3% on RoboMME exceeded matched retrieval, compression, and recurrent-state rebuilds at 31.5%, 22.6%, and 20.6%, respectively.The comparison holds the backbone and training setup fixed.
Results: Masking task-relevant history changes policy outputs, whereas masking irrelevant segments does not, supporting genuine use of visual history.The policy also adapts to edited or previously unseen visual histories without parameter updates.

2 RELATED WORK

Related VLA work has advanced action generation and policy architecture, while memory research addresses the need to preserve earlier observations in partially observable tasks. SimpleMemVLA differs by presenting minute-scale timestamped video directly to native backbone attention rather than using a dedicated memory mechanism.

VLA research directions: Generalist VLA research has progressed from discretized actions and control-specific tokenizers to continuous diffusion and flow-matching experts.These developments concern how policies generate actions from available observations.
Memory mechanisms: Memory designs address partially observable tasks because the information needed for a decision may appear only in earlier observations.Existing designs differ in what they preserve before future requirements are known.
Memory mechanisms: Symbolic pipelines store structured representations but discard information outside their predefined vocabulary during extraction.This limitation can remain even with perfect perception.
SimpleMemVLA: SimpleMemVLA presents minute-scale timestamped video directly to the backbone, allowing native attention to select relevant past evidence for each decision.This approach parallels results reported for sliding-window streaming video understanding.

3 SIMPLEMEMVLA

SimpleMemVLA formulates memory-dependent manipulation as history-conditioned control and retains sampled visual history in the backbone’s native timestamped video interface. A generated textual sub-task compresses selected history for the action expert, while shared-prefix prefill makes deployment practical.

3.1 PROBLEM SETUP AND OVERVIEW: Memory-dependent manipulation requires conditioning actions on history because identical current observations can demand different actions after different past events.The policy states the current sub-task before producing an action chunk.
3.1 PROBLEM SETUP AND OVERVIEW: The architecture sends head-camera history through the native video channel, wrist views through the image channel, and generated sub-task states to a flow-matching action head.It uses standard VLA components rather than a dedicated memory module.
3.2 RETAINING HISTORY FOR READ-TIME SELECTION: History is rebuilt as a subsampled window of up to K = Twfv frames, with native plaintext timestamps preserving temporal grounding for self-attention.The window covers the last Tw seconds and uses a sampling rate fv much lower than the native frame rate.
3.2 RETAINING HISTORY FOR READ-TIME SELECTION: The input format separates past video from present wrist images, while low-rate sampling keeps minute-scale history within a modest token budget.An episode-scale window keeps early sampled evidence available near the episode end.
3.3 A NARROW TEXT CHANNEL FROM HISTORY TO ACTION: The generated one-sentence sub-task provides the compact conditioning signal through fused token embeddings and hidden states from the backbone response.The flow-matching expert uses this signal together with the current proprioceptive state to generate an action chunk.
3.3 A NARROW TEXT CHANNEL FROM HISTORY TO ACTION: Training jointly supervises the demonstrated action chunk and an offline-generated sub-task target, using the same targets across mechanism variants.This controls annotation quality in the within-stack memory comparison.
3.4 EXACT STREAMING INFERENCE: Shared-prefix prefill overlaps history processing with action execution, so the next decision processes only newly arrived temporal content before decoding.This reduces decision-time latency without changing policy output.
3.4 EXACT STREAMING INFERENCE: The exact streaming cache and background prefill still grow with accumulated history, although a sliding-window-attention variant keeps both bounded by episode length.The bounded variant requires training with the same sliding-window attention during training and inference.

4 EXPERIMENTS

Across four memory-centric benchmarks, SimpleMemVLA achieves the strongest reported performance while matching or exceeding top general-purpose control results. Matched and intervention-based analyses attribute these gains to preserving and using visual history, with streaming reducing the latency cost of long contexts.

Benchmark performance: SimpleMemVLA achieves the highest reported performance on all four memory-centric benchmarks, ties LIBERO’s best average, and leads zero-shot LIBERO-Plus transfer.The results report 97.5% on LIBERO and 78.4% on LIBERO-Plus, 5.3 points above the strongest memory-augmented model.
Benchmark performance: 94.0% on RMBench is 11.0 points above the strongest specialist, using one model across the nine-task comparison set.Its average rises from 91.6% in the single-memory setting to 97.0% in the multi-memory setting.
Benchmark performance: 88.3% on RoboMME exceeds the strongest non-oracle baseline by 43.7 points, while MIKASA-Robo reaches 74.0%, 29.6 points above the strongest prior VLA.The lead spans counting, permanence, reference, imitation, occlusion, and demonstrated-sequence demands.
Benchmark performance: 63.6% TSR and 72.1% CSR on RoboMemArena exceed the strongest prior model by 17.4 points across trajectories averaging more than one thousand control steps.The largest gains occur in Occlusion and Counting, while SimpleMemVLA does not lead on Transferring.
Attribution: Masking task-relevant history changes outputs while masking irrelevant history does not, and edited histories alter behavior without parameter updates.These interventions support genuine visual memory and inference-time visual in-context learning.
Efficiency: Streaming reduces decision latency from 1.02 s to 0.68 s in a minute-scale configuration, with background prefill fitting within the action-execution interval.At 45 minutes of history, latency is 1.18 s with streaming versus 32.1 s with full recomputation.
Efficiency: Exact streaming’s background prefill and key–value cache still grow with episode length, motivating streaming-native sliding-window attention for bounded deployment costs.The SWA variant uses a bounded attention window during training and inference.

5 CONCLUSION

SimpleMemVLA uses pretrained native video context as memory, preserving timestamped history and routing task-relevant information to a standard flow-matching action head. It achieves strong memory performance, supports causal history use, and avoids sacrificing evaluated general-purpose control.

SimpleMemVLA exposes minute-scale timestamped visual history directly to a pretrained VLM and uses generated-subtask hidden states to condition a flow-matching action head.The approach avoids a dedicated memory module while preserving perceptual detail and temporal structure until task-conditioned read time.
88.3% on RoboMME versus 31.5%, 22.6% and 20.6% for matched retrieval, token-compression and recurrent-state rebuilds.The comparison uses the same stack and attributes the gain to the memory interface.
History-editing experiments redirect outputs using unseen edited videos without parameter updates, confirming dependence on specific past observations and emergent visual in-context learning.
The method establishes state-of-the-art results across four memory-centric benchmarks without sacrificing performance on evaluated general-purpose suites.

A IMPLEMENTATION DETAILS

The implementation uses a shared native-video backbone and training setup, with offline sub-task labels shared across controlled comparisons. Practical deployment is bounded by context length, pixel-budget handling, and the reported benchmark comparison protocols.

All benchmark suites use Qwen3.5-4B with native video and plaintext timestamps, shared conditioning and losses, plus AdamW with separate backbone and action-head learning rates.The learning rates are 10^-5 and 5 × 10^-5, respectively, with cosine decay and gradient clipping.
Offline cloud-LLM sub-task labels are stored per frame and shared by all three mechanism variants, while the deployed policy never queries the annotator.Annotation is therefore a shared training-data cost and absent at inference.
The listed benchmark rows combine published results, official leaderboard entries, and suite-specific evaluation protocols rather than one uniform retraining procedure.The passage specifies separate provenance for RMBench, RoboMME, RoboMemArena, MIKASA-Robo and LIBERO-Plus.
The backbone’s 262k-position limit corresponds to about 48 minutes of history at the deployment sampling rate.

B FULL ROBOMME RESULTS

The full RoboMME materials document task-wise success rates, method-family rankings, and controlled memory-interface variants. Figures and tables distinguish SimpleMemVLA from reference-only human and oracle rows and from excluded within-stack rebuilds.

Figure 10 presents all 24 rows in source-table order, with colors encoding memory families and orange hatched bars identifying SimpleMemVLA.
Table 8 reports complete task-level RoboMME success rates for SimpleMemVLA and its three controlled variants.The table uses percentage success rates and marks ranks among the remaining 21 methods.

C LIBERO-PLUS: PROTOCOL DETAILS AND BREAKDOWNS

LIBERO-Plus evaluation follows a fixed, exhaustive protocol across 10,030 tasks and reports performance by suite, difficulty, and perturbation-related dimensions. Semantic suites and higher difficulty levels show larger degradation, while noise effects depend on whether geometric evidence remains available.

Protocol: 10,030 tasks are evaluated with one trial each, frozen initial states, fixed environment seeds, ten settling steps, suite-specific budgets, and simulator-based success checks.The seven dimensions contain different task counts, so the reported Total is success-weighted by task count.
By suite and difficulty: 23.0 points and 21.9 points are lost by Goal and Long under perturbation, compared with 16.4 for Spatial and 14.9 for Object.
By suite and difficulty: Success declines monotonically from 90.5% at Level 1 to 56.3% at Level 5.The passage describes this as graded degradation rather than a bimodal solved/unsolved split.
Noise breakdown: The 74.2% Sensor Noise result varies by corruption type according to whether visual corruption preserves geometric evidence.Motion and glass blur smear textures and edges, while the broader benchmark pattern favors intact scene geometry and consistent streams.

D.1 THE GAINS TRACK MEMORY LOAD

SimpleMemVLA’s advantage increases as tasks require more memory, while matching the best reactive VLAs when memory is not the bottleneck. Its remaining failures cluster in precision control, fine-grained re-identification, and instruction or positional perturbations.

RMBench’s margin over Mem-0 widens from +38.8 on single-memory tasks to +68.5 on multi-memory ones.
On RoboMME, the margin grows from +24.7 on Counting to +44.5 on Reference, where the evidence is entirely off-frame.
On MIKASA-Robo, the lead over the best prior VLA stretches from 12 to 39 points as RememberColor increases from 3 to 9 candidates.
LIBERO reaches 97.5%, on par with the best reactive VLAs, where memory is not the bottleneck.
Residual failures concentrate in precision control, fine-grained re-identification, and perturbations that rewrite instructions or attack positional evidence.

D.2 WHY THE MACHINERY FALLS SHORT

The controlled comparisons attribute prior memory methods’ weakness to write-time abstraction: they discard order, timestamps, stale frames, or percepts before future needs are known. Raw-stream attention instead defers selection until read time.

Fourteen RoboMME variants share one π0.5 backbone yet reach 11.6–44.5%, versus 88.3% for the timestamped stream.
Within the authors’ stack, retrieval, compression, and recurrent-state rebuilds fall to 20.6–31.5% when only the memory interface differs.
GroundSG reaches 84.1% even with ground-truth VLM outputs, remaining below the raw stream’s 88.3% because abstraction precedes the need.
Deployment edits identify discarded evidence: shuffled frames remove temporal order, zeroed timestamps impair counting, and graying interventions show reliance on percepts.
Attention over the raw stream avoids the tested families’ write-time choice by deferring selection to read time.

E BOUNDED-COST STREAMING WITH SLIDING-WINDOW ATTENTION

Sliding-window attention bounds both per-decision compute and state by evicting old softmax-attention cache blocks while preserving episode-wide linear-attention state. It gives constant-cost deployment with a small accuracy trade-off on the hardest memory tasks.

Window design: SWA changes only the 8 softmax-attention layers, limiting them to the most recent L = 5,888 tokens while linear-attention layers retain the full stream.
Timestamping: Absolute timestamps keep cached tokens valid as the window slides, because emitted tokens are never relabeled or re-encoded.
Deployment: Each decision appends one history unit, evicts one oldest unit, forks the decision suffix, and reuses cached stream state.
Cost: The SWA decision takes 0.92 s within a 0.96 s real-time budget, with resident cache capped at the 5,888-token window.
Training: Episode-level packing trains shared full-history trunks with B = 8 anchor branches, matching deployment’s linear-attention state construction.
Results: The deployment scores 91.0 versus 94.0 overall, with differences concentrated in two hardest memory tasks while streamed inference reproduces full-prompt decisions exactly.