PaperlayerView exact revision on arXiv

Source-linked AI summary

EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies

Ganlin Yang, Zhangzheng Tu, Yuqiang Yang, Sitong Mao, Junyi Dong, Tianxing Chen, Jiaqi Peng, Jing Xiong, Jiafei Cao, Jifeng Dai, Wengang Zhou, Yao Mu, Tai Wang

arXiv:2606.20092v2cs.CV

TL;DR

Long-horizon manipulation requires memory because task-relevant visual evidence can become occluded or unobservable. EventVLA combines visual anchors with foresight-driven keyframe memory and outperforms existing memory-augmented VLAs across simulation and real-world tasks.

  • Problem

    Long-horizon VLA manipulation requires retaining transient task-relevant visual evidence beyond the current observation, while existing memory approaches face latency, information-bottleneck, or redundancy limitations.

  • Method

    EventVLA combines foundational visual anchors with a foresight-driven Keyframe Evidence Memory module that selectively stores predicted task-critical visual events.

  • Results

    75.2% average success on RoboTwin-MeM, with up to 80% success on real-world bimanual tasks, while outperforming existing memory-based VLAs.

  • Takeaways & Limitations

    EventVLA provides an end-to-end approach for evaluating and addressing non-Markovian memory requirements in long-horizon robotic manipulation.

  • Takeaways & Limitations

    The bounded event buffer may saturate and evict early cues in exceptionally long-horizon tasks exceeding 10 minutes with high event densities.

Abstract

from arXiv · show

Memory remains a critical bottleneck for long-horizon robotic manipulation, as standard Vision-Language-Action (VLA) policies often fail when task-relevant cues become occluded or unobservable over time. While existing memory-augmented methods utilize historical context, they either suffer from severe information bottlenecks, incur high latency via decoupled dual systems, or rely on unselective buffers that accumulate massive visual redundancies. To address these limitations, we introduce EventVLA, an end-to-end framework founded on the concept of sparse visual evidence memory that comprises two core components: foundational visual anchors to retain initial and short-term contexts, and a dynamic Keyframe Evidence Memory (KEM) module. Specifically, KEM directly predicts future keyframe probabilities from the VLA's latent embeddings to autonomously capture and store sparse, task-critical visual events. This foresight-driven mechanism empowers the policy to dynamically evaluate the future causal utility of current observations, preserving transient visual evidence before it becomes unobservable. Furthermore, we propose RoboTwin-MeM, a diagnostic benchmark specifically designed to evaluate non-Markovian manipulation tasks with interactive visual evidence. Extensive evaluations show that across 17 memory-requiring simulation tasks and 4 real-world bimanual tasks, EventVLA achieves an average success rate improvement of +40% over state-of-the-art memory-augmented VLAs.

1 Introduction

EventVLA addresses the failure of predominantly Markovian VLA policies on long-horizon manipulation by sparsely preserving task-critical visual evidence. It combines foundational visual anchors with foresight-driven keyframe selection and introduces RoboTwin-MeM to evaluate genuinely non-Markovian scenarios.

  • Motivation: VLA policies predominantly assume task-relevant information remains persistently visible, although dynamic workspaces require retaining intermediate states for later actions.An example is remembering an item's original location after it has been displaced.
  • Sparse Visual Evidence Memory: Foundational visual anchors retain the initial frame and a short-term history window, reducing redundancy while supplying global layout and local motion cues.These heuristics solve structurally simple memory-requiring tasks but fail in complex interactive scenarios.
  • EventVLA: EventVLA combines foundational visual anchors with a dynamic Keyframe Evidence Memory module in an end-to-end framework that autonomously captures transient interaction-driven events.KEM replaces rigid rule-based heuristics with foresight-driven, data-driven keyframe predictions over upcoming observations.
  • RoboTwin-MeM: RoboTwin-MeM is a diagnostic simulation benchmark for genuinely non-Markovian manipulation, comprising 8 challenging tasks whose required intermediate keyframes scale from 1 to 5.It targets transient evidence that existing benchmarks can often solve using basic visual anchors alone.
  • Results: 75.2% average success rate is achieved on RoboTwin-MeM, while EventVLA reaches 67.8% on RMBench and sets a new state-of-the-art on conventional memory-oriented tasks.The passage describes EventVLA as vastly outperforming existing methods on the newly introduced benchmark.

2 Related Work

Recent VLA foundation models are highly generalizable but fundamentally memoryless, motivating memory-augmented approaches for non-Markovian tasks with transient or occluded visual information. Existing benchmarks often fail to directly evaluate explicit memory reasoning at sufficient scale.

  • Memory-Augmented VLAs: VLA foundation models achieve remarkable generalizability but operate under a strict Markovian assumption, limiting performance when critical visual information is transient or occluded.
  • Memory-Augmented VLAs: Memory-augmented VLAs have emerged across three paradigms, including dual-system Memory-VLAs that use a high-level VLM for planning.
  • Memory Benchmarks: Standard simulation suites emphasize long-horizon execution rather than explicit memory reasoning because task-relevant information typically remains persistently visible.
  • Memory Benchmarks: Recent memory-centric benchmarks remain limited by small scale, exclusive reinforcement-learning focus, or observable states.

3 EventVLA Framework

EventVLA models long-horizon manipulation as non-Markovian by conditioning actions on sparse visual evidence memory. Its memory combines foundational scene and short-term anchors with foresight-driven event keyframes predicted from VLA hidden states and trained end-to-end.

  • Sparse visual evidence memory: EventVLA augments reactive action generation with an external memory buffer that selectively stores historical frames when current task evidence becomes occluded or unobservable.The buffer minimizes informational and computational redundancy while preserving essential visual evidence.
  • Foundational visual anchors: The memory unites a permanent initial-scene anchor with a K-frame short-term history, preserving layout, motion, and task-progression cues.These deterministic anchors cannot capture unpredictable transient evidence arising during complex interactions.
  • Keyframe Evidence Memory: KEM is a lightweight parallel head that predicts future keyframe probabilities from the VLA transformer’s action-conditioned hidden states.Because these states encode visual observations and future execution queries, the head supports proactive keyframe selection.
  • Keyframe Evidence Memory: Chunk-wise foresight predicts task-critical events across the future horizon, capturing transient evidence that step-wise classification could miss before it disappears.EventVLA writes a keyframe when its predicted probability crosses a threshold, then bounds event memory with FIFO eviction.
  • Training and inference: The unified temporally ordered sequence is encoded by the VLM to model correlations across sparse frames, while KEM is trained end-to-end with automated event labels, smoothed BCE, and a teacher-to-student curriculum.Dense online predictions are distilled into discrete write events to enforce sparsity and prevent redundant memory growth.

4 RoboTwin-MeM Benchmark

RoboTwin-MeM is a diagnostic simulation benchmark for evaluating whether VLA policies capture and retain transient visual evidence during manipulation. Its unified, n-parameterized design supports scalable evaluation of intermediate memory demands, including fleeting evidence and sequence-tracking tasks.

  • Benchmark Design: RoboTwin-MeM evaluates transient visual-evidence retention within the RoboTwin 2.0 simulation platform built on the SAPIEN physics engine.It is designed as a diagnostic simulation benchmark for VLA policies.
  • Benchmark Design: The benchmark unifies automated data synthesis and policy evaluation while providing fine-grained language annotations for each action-observation pair.The annotations describe low-level interactions and state transitions, supplying structured, dense supervision.
  • Memory Demands: RoboTwin-MeM isolates and quantifies intermediate memory demands by requiring policies to memorize unpredictable visual evidence revealed during execution.Unlike suites solvable through static anchors or short-term histories, it explicitly tests active memorization of mid-execution evidence.
  • Memory Demands: Tasks parameterized by n test transient memory, sequence tracking, and counting beyond trivial history concatenation.Pick the Unhidden Block (n = 3) and Cover Blocks Hard (n = 4) expose essential evidence briefly before it disappears when the cover closes.

5 Experiments

Experiments show that EventVLA’s visual anchors and KEM are complementary: anchors perform strongly on RMBench, while KEM is necessary for intermediate-state memory on RoboTwin-MeM. The method also preserves reactive control on Markovian tasks and outperforms reactive and memory-augmented baselines in real-world manipulation.

  • RMBench and visual anchors: 67.8% average success rate makes foundational visual anchors state of the art on RMBench’s simple memory-required tasks.These tasks primarily depend on persistent spatial layouts and fixed motion styles.
  • RMBench and visual anchors: Removing the initial frame or short-term history reduces success to 33.7% and 23.8%, confirming both are necessary for visual anchoring.The initial frame provides global spatial reference, while short-term history supplies motion cues.
  • RoboTwin-MeM and KEM: 18.0% average success with visual anchors alone on RoboTwin-MeM demonstrates that fixed historical windows are inadequate for intermediate-state memory.RoboTwin-MeM is explicitly designed to test intermediate-state memory.
  • Markovian benchmarks: 83.8% versus 80.0% on Easy and 81.6% versus 78.0% on Hard shows EventVLA improves over QwenOFT on Markovian RoboTwin-2.0 tasks.The memory mechanism complements standard closed-loop reactive execution rather than degrading it.
  • RoboTwin-MeM and KEM: 75.2% to 24.9% follows latent-memory replacement, 48.8% follows binary targets, and 53.4%, 32.0%, 31.1%, and 13.6% follow buffer or horizon restrictions.The ablations identify information bottlenecks, unstable prediction targets, redundant frame flooding, premature eviction, and truncated foresight as failure modes.
  • Real-world evaluation: 0% to 10% success for reactive π0.5 contrasts with πMEM’s 50% on Find Block Easy, 30% on Pick-X-Times, and 40% on Pick in Order.The physical evaluation covers four non-Markovian tasks on the ARX ACONE bimanual robot, with each task tested over 20 trials.

6 Limitations

EventVLA’s bounded event buffer limits scalability in exceptionally long-horizon tasks with high event densities, risking saturation and premature eviction of early historical cues. Future work will explore hierarchical memory or compressed representations for massive event sequences.

  • Scalability: EventVLA’s bounded event buffer limits scalability in exceptionally long-horizon tasks, such as those exceeding 10 minutes with high event densities.These conditions create exceptionally large event sequences.
  • Scalability: Buffer saturation can cause premature eviction of early historical cues in massive, high-density event sequences.The limitation arises when the bounded buffer cannot retain all accumulated events.
  • Future work: Future work will investigate hierarchical memory or compressed representations to manage massive event sequences.These approaches are proposed to address the scalability limitation.

7 Conclusion · Appendix

EventVLA addresses non-Markovian long-horizon manipulation with sparse visual evidence memory, combining rule-based visual anchors and foresight-driven KEM to capture transient task-critical events without dense-memory redundancy. The paper also introduces RoboTwin-MeM as a diagnostic benchmark for intermediate memory capabilities.

  • 7 Conclusion: EventVLA is an end-to-end framework for non-Markovian long-horizon manipulation.
  • 7 Conclusion: Its sparse visual evidence memory combines rule-based visual anchors with a foresight-driven Keyframe Evidence Memory module.
  • 7 Conclusion: KEM proactively captures task-critical transient visual events.
  • 7 Conclusion: The framework avoids the redundancy associated with dense memory.
  • 7 Conclusion: RoboTwin-MeM is proposed as a diagnostic benchmark for evaluating intermediate memory capabilities.
  • 7 Conclusion: The paper reports extensive evaluations across 17 simulation tasks.

A Implementation Details of EventVLA … B Experimental Setups and Benchmarks

EventVLA trains and deploys sparse keyframe memory end-to-end, using automated VLM annotations, curriculum-based supervision, and temporal post-processing to preserve nonredundant visual evidence. Its annotation pipeline is designed for scalable, reliable labeling of long-horizon robotic tasks.

  • A.1 Training Formulations and Curriculum Strategy: Qwen3-VL parses demonstration videos and task descriptions to extract timestamps of task-critical intermediate events for ground-truth keyframe supervision.Temporal ambiguity around each event is addressed through temporally smoothed annotations.
  • A.1 Training Formulations and Curriculum Strategy: The Keyframe Evidence Memory loss averages binary cross-entropy across the future action horizon, aligning predicted probabilities with temporally smoothed keyframe labels.The framework jointly optimizes memory supervision and continuous action generation, balanced by coefficient λ.
  • A.1 Training Formulations and Curriculum Strategy: A teacher-forcing curriculum mixes ground-truth keyframes with thresholded model predictions, stabilizing early training while reducing train-test distribution shift.The transition probability is controlled by α, moving the policy toward autonomous memory updates.
  • A.2 Online Inference and Post-Processing: Online inference converts clustered chunk-wise keyframe probabilities into sparse memory writes using confidence thresholding and one-dimensional non-maximum suppression.Candidate peaks must be local maxima within a temporal neighborhood.
  • A.2 Online Inference and Post-Processing: A temporal cooldown validates candidate peaks only when sufficiently separated from the latest committed timestamp, preventing redundant writes from rapid consecutive events.This cascading extraction ties memory allocation to novel interactive evidence and keeps the buffer highly sparse.
  • A.3 Automated Keyframe Annotation Pipeline: Qwen3-VL-235B-A22B-Instruct-FP8 runs locally on 8 NVIDIA A800 GPUs with vLLM to provide scalable automated keyframe annotation.The pipeline avoids the prohibitive cost of dense manual frame annotation for long-horizon tasks.
  • A.3 Automated Keyframe Annotation Pipeline: The annotation pipeline uniformly samples episodes, concatenates multi-view observations, and prompts the VLM to identify ordered, representative task-state transitions.Examples include stable grasps, placements, and recognizable moments in repeated pick/place cycles; outputs are constrained to JSON keyframe steps.
  • A.3 Automated Keyframe Annotation Pipeline: Cross-validation against precise algorithmic labels found negligible temporal variance despite evaluation horizons often exceeding 1500 to 2000 steps, supporting reliable scalable deployment.The reported variance is naturally accommodated by the temporally smoothed soft labels.

B.1 RoboTwin-MeM Benchmark Details … C.1 Detailed Per-Task Breakdown on RMBench

RoboTwin-MeM isolates non-Markovian manipulation through occlusion, temporal delay, and task-dependent intermediate evidence requirements, while EventVLA uses benchmark-specific memory configurations and achieves near-perfect performance on several RMBench tasks. The extended results also detail real-world evidence capture and per-task baseline comparisons.

  • B.1 RoboTwin-MeM Benchmark Details: RoboTwin-MeM evaluates genuinely non-Markovian manipulation by enforcing strict visual occlusions and temporal delays that hide task-critical evidence.Examples include hidden block colors, covered object identities, and randomly generated spatial information.
  • B.1 RoboTwin-MeM Benchmark Details: 8 bimanual tasks span 430 to 1,544 average steps per episode, with difficulty parameterized by n ∈[1, 5] intermediate keyframe events.The benchmark’s task statistics include average steps, required keyframe count n, and language instructions.
  • B.1 RoboTwin-MeM Benchmark Details: Cover Blocks Hard, Pick Objects in Order, and Press Button Keyframe require retaining transient visual evidence for later sequential manipulation decisions.These tasks involve inspecting hidden attributes, remembering them after covers close, or translating randomized number cards into counting and pressing actions.
  • B.2 Real-world Tasks Details: Expanded real-world rollouts show when task-critical intermediate keyframes emerge and when EventVLA autonomously commits them to its dynamic event buffer.Examples include exposing a hidden block, reading a randomized number, and observing a transient visual cue.
  • B.3 Network Architecture and Hyper-parameters: 80,000 training steps use AdamW, with a 1 × 10−5 learning rate for the base VLM and 1 × 10−4 for newly initialized heads.The differential learning-rate strategy is applied to stabilize convergence.
  • B.3 Network Architecture and Hyper-parameters: For RMBench, EventVLA is trained with initial and short-term visual anchors only, because the benchmark primarily evaluates foundational visual anchoring.The listed anchor positions are o_0, o_t−30, and o_t−15.
  • B.3 Network Architecture and Hyper-parameters: For RoboTwin-MeM, the full KEM module is activated with a teacher-forcing curriculum whose probability α decays linearly from 1.0 to 0.0.The configuration governs the online memory extraction pipeline and bridges train-test distribution shift.
  • C.1 Detailed Per-Task Breakdown on RMBench: 96% Rearrange Blocks, 95% Put Back Block, 96% Swap Blocks, and 97% Cover Blocks success rates show EventVLA near-perfect performance on RMBench structural tasks.The detailed task-level table compares EventVLA with non-memory and prior memory-augmented baselines across the RMBench suite.

C.2 Extended Ablation Analysis and Inference Efficiency · D Qualitative Visualizations · D.1 Simulation Rollouts in RoboTwin-MeM

EventVLA’s ablations identify explicit image memory, soft temporal supervision, sparse buffer management, and sufficient foresight as essential for reliable keyframe capture. Efficiency measurements and qualitative RoboTwin-MeM rollouts show that dynamic KEM preserves transient evidence while maintaining real-time operation for high-level planning.

  • C.2 Extended Ablation Analysis and Inference Efficiency: Explicit raw-image concatenation avoids the severe information bottleneck caused by compressing multiple distinct events into one latent vector.The bottleneck becomes critical for complex tasks requiring retention of n ≥3 events.
  • C.2 Extended Ablation Analysis and Inference Efficiency: Soft raised-cosine labels provide temporal tolerance, whereas hard binary labels create sparse supervision, penalize adjacent valid frames, and can prevent essential memory writes.Hard-label supervision destabilizes the predictive head in environments with execution variability.
  • C.2 Extended Ablation Analysis and Inference Efficiency: NMS-driven event sparsity and adequate buffer capacity are both necessary to prevent redundancy, saturation, and premature FIFO eviction of foundational evidence.A strictly minimal buffer such as Nmax = 2 cannot retain evidence needed by complex multi-stage tasks.
  • C.2 Extended Ablation Analysis and Inference Efficiency: Shrinking the execution chunk size truncates KEM’s look-ahead horizon, weakening prediction and neutralizing proactive memory commitment.The action chunk size governs the foresight window for anticipating transient events.
  • C.2 Extended Ablation Analysis and Inference Efficiency: 0.94 Hz throughput and 1.09 seconds latency characterize full EventVLA inference, compared with 2.91 Hz and 0.36 seconds for the purely reactive QwenOFT baseline.External visual anchors reduce average throughput to 1.07 Hz, while EventVLA remains suitable for high-level planning.
  • D Qualitative Visualizations: Qualitative visualizations cover successful execution sequences across all 8 strictly non-Markovian RoboTwin-MeM simulation tasks.Figures 6 and 7 divide the demonstrations across four memory-intensive tasks each.
  • D.1 Simulation Rollouts in RoboTwin-MeM: KEM proactively triggers sparse memory writes when transient evidence appears, locking critical intermediate states into the event buffer before they become unobservable.Examples include observing a hidden block color after lifting an opaque cover and reading a randomized number.
  • D.1 Simulation Rollouts in RoboTwin-MeM: The rollouts show EventVLA bridging temporal gaps by preserving transient visual evidence and guiding subsequent long-horizon manipulation.The event buffer retains intermediate states needed after the original observations disappear.

D.2 Real-World Robot Execution Sequences

EventVLA is qualitatively validated on the real-world ARX ACONE bimanual robot through four memory-intensive manipulation tasks. The executions show that sparse visual evidence memory preserves critical cues under occlusion and randomized placements, enabling complex multi-stage behavior with non-Markovian remembering and spatial generalization.

  • Real-world execution sequences: Four real-world tasks—Find Block Easy, Pick-X-Times, Find Block Hard, and Pick in Order—were successfully completed by EventVLA on the ARX ACONE bimanual robot.Figure 8 presents qualitative execution sequences for these memory-intensive manipulation tasks.
  • Visual evidence retention: EventVLA robustly captured and retained critical intermediate visual cues despite real-world occlusions and randomized spatial placements.Examples include reading a randomized number to determine counting logic and observing a stick pointing at bottles to memorize an in-context sequence.
  • Long-horizon execution: The policy used sparse visual evidence memory to execute complex, multi-stage physical tasks with strong non-Markovian remembering and spatial generalization.These capabilities were demonstrated across the qualitative real-world execution sequences.
Loading 2606.20092v2…