PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark

Huashuo Lei, Wenxuan Song, Huarui Zhang, Jieyuan Pei, Jiayi Chen, Haodong Yan, Han Zhao, Pengxiang Ding, Zhipeng Zhang, Lida Huang, Donglin Wang, Yan Wang, Haoang Li

arXiv:2605.10921v1cs.RO

TL;DR

Existing robotic memory benchmarks lack multimodal memory annotations, broad and structurally complex tasks, and real-world evaluation. RoboMemArena addresses these gaps and introduces PrediMem, whose explicit keyframe memory and predictive coding achieve the highest TSR and CSR among evaluated methods.

  • Problem

    Existing robotic memory benchmarks lack multimodal memory annotations, broad task coverage and structural complexity, and real-world evaluation.

  • Method

    The paper introduces RoboMemArena with multimodal annotations, long-horizon tasks, and real-world evaluation, alongside PrediMem, a dual-system memory VLA with predictive decoding.

  • Results

    PrediMem achieves the highest task and subtask success rates among evaluated methods on RoboMemArena.

  • Takeaways & Limitations

    RoboMemArena makes memory a central requirement, with 68.9% of subtasks depending on past observations, while PrediMem improves sensitivity to physical state transitions without added inference-time cost.

Abstract

from arXiv · show

Memory is a critical component of robotic intelligence, as robots must rely on past observations and actions to accomplish long-horizon tasks in partially observable environments. However, existing robotic memory benchmarks still lack multimodal annotations for memory formation, provide limited task coverage and structural complexity, and remain restricted to simulation without real-world evaluation. We address this gap with RoboMemArena, a large-scale benchmark of 26 tasks, with average trajectory lengths exceeding 1,000 steps per task and 68.9% of subtasks being memory-dependent. The generation pipeline leverages a vision-language model (VLM) to design and compose subtasks, generates full trajectories through atomic functions, and provides memory-related annotations, including subtask instructions and native keyframe annotations, while paired real-world memory tasks support physical evaluation. We further design PrediMem, a dual-system VLA in which a high-level VLM planner manages a memory bank with recent and keyframe buffers and uses a predictive coding head to improve sensitivity to task dynamics. Extensive experiments on RoboMemArena show that PrediMem outperforms all baselines and provides insights into memory management, model architecture, and scaling laws for complex memory systems.

1 Introduction

RoboMemArena addresses limitations in robotic memory benchmarks by combining multimodal memory annotations, long-horizon and diverse tasks, and real-world evaluation. The paper also introduces PrediMem, a dual-system memory VLA with hierarchical memory and predictive decoding, and evaluates it alongside baselines and variants.

  • Motivation: Memory enables robots to accomplish long-horizon, complex tasks in partially observable environments.The introduction identifies memory as a critical component of robotic intelligence.
  • Limitations: Existing robotic memory benchmarks lack multimodal annotations, task coverage and structural complexity, and real-world evaluation.The paper motivates RoboMemArena by identifying limitations in prior benchmark datasets and evaluation settings.
  • RoboMemArena: RoboMemArena contains 26 simulated tasks across four memory-dependent categories, averaging 1,076 steps per task and 68.9% history-dependent subtasks.Its pipeline uses VLM-based subtask composition, atomic-function trajectory generation, and subtask-instruction and keyframe annotations.
  • RoboMemArena: RoboMemArena supplements scalable simulation with five challenging real-world memory tasks, including demonstrations lasting over three minutes.The real-world benchmarks provide physical evaluation complementary to the simulated benchmarks.
  • PrediMem: PrediMem is a dual-system VLA pairing a high-level VLM planner with a low-level VLA actor and managing recent and keyframe memory buffers.A predictive coding head improves sensitivity to keyframe choice and understanding of event dynamics and task progression.
  • Experiments: Experiments evaluate representative baselines and PrediMem variants on RoboMemArena, providing insights into memory management, model architecture, and scaling laws.The experiments concern memory-augmented robotic manipulation.

2 Related Work

Existing robotic manipulation benchmarks cover broad objects, scenes, and skills, but many tasks remain locally observable and do not isolate memory as the central bottleneck. Recent memory-oriented benchmarks move closer to this goal, while large-scale VLA pretraining strengthens language-conditioned manipulation backbones with multi-frame and future-aware extensions.

  • Existing robotic manipulation benchmarks cover broad objects, scenes, and skills, but many tasks remain locally observable and do not isolate memory as the central bottleneck.
  • Recent memory-oriented benchmarks move closer to isolating memory as the central bottleneck.
  • Large-scale VLA pretraining has produced strong language-conditioned manipulation backbones.
  • Recent VLA extensions add multi-frame context and future-aware action modeling.

3 RoboMemArena

RoboMemArena is a 26-task robotic memory benchmark spanning four memory-demand categories, long-horizon trajectories, multimodal annotations, and paired real-world evaluation. Its scalable pipeline combines VLM task decomposition, autonomous execution, and event-focused keyframe extraction, while benchmark statistics show substantial memory dependence and broad task coverage.

  • Task Suite: 26 tasks cover transferring, occlusion, counting, and sequential execution, targeting failure modes where correct actions depend on information no longer visible.Tasks require remembering source–target mappings, completed transfers, hidden objects, repetition counts, and outcomes spanning multiple operations.
  • Real-world Tasks: Five paired real-world tasks evaluate counting, occlusion, sequential execution, hidden-target tracking, and human-demonstration-conditioned memory on the AgileX Cobot Mobile Aloha Platform.The tasks are Pour Bottle ×2, Brush Plates with Swap, Transfer Objects, Shell Game, and Imitate Human to Make Breakfast.
  • Data Generation: The generation pipeline uses VLM-driven subtask decomposition, AnyGrasp-based autonomous execution with retry checking, and multi-conditioned keyframe extraction.VLM proposals are manually refined when unsuitable; AnyGrasp dispatches estimated 6-DoF poses to primitives; failed subtasks are retried with updated poses.
  • Keyframe Annotation: Gripper-state transitions and kinematic inflections select information-bottleneck frames that capture task progress while avoiding redundant dense video storage.The resulting compact, event-focused annotations provide temporal supervision for VLMs.
  • Benchmark Comparison: RoboMemArena is the only benchmark among 14 established benchmarks satisfying all eight compared feature criteria, including multimodal memory supervision, scalable generation, and paired real-world evaluation.The comparison identifies native keyframes, automated trajectory generation, and physical validation as benchmark-level strengths.
  • Scale and Memory Dependence: 68.9% of subtasks are history-dependent, with 104 of 151 subtasks across 26 tasks requiring information from earlier subtasks or observations.RoboMemArena also collects 2,600 successful long-horizon visual trajectories and 15,100 keyframe-aligned short segments, averaging 1,076 steps per task.

4 PrediMem: Building Hierarchical Memory with Predictive Coding

PrediMem is a hierarchical embodied-memory framework that couples a high-level planner and low-level policy with keyframe-grounded memory and predictive coding. It preserves decision-critical events beyond recent observations while retaining standard dual-system inference cost.

  • Architecture: PrediMem combines System 2 planning, System 1 execution, a keyframe-grounded memory bank, and an auxiliary predictive coding head.Its pipeline asynchronously couples S2, which predicts keyframes and dispatches subtasks, with S1, which executes the current subtask.
  • Hierarchical Memory: The memory bank combines a long-term keyframe buffer with a recent sliding window, allowing decision-critical events to persist beyond recent observations.S2 uses the current observation and memory bank to predict the current subtask and decide which frames become keyframes.
  • Predictive Coding: Predictive coding predicts the subsequent frame representation from current visual features to improve sensitivity to abrupt state transitions at keyframes.The mechanism addresses the tradeoff between over-storing frames and missing transitions that cause downstream errors.
  • Training Objective: S2 training uses the combined objective LS2 = Ltext + 0.1LPre, while S1 follows the official flow-matching objective.Ltext covers next-token prediction for subtask generation and keyframe decisions.
  • Inference: During inference, PrediMem removes the predictive coding head and retains standard dual-system architecture and cost while asynchronously executing the two systems.The predictive coding path is training-only, so its capability gains do not add inference-time architectural cost.

5 Experiments

Experiments show that PrediMem achieves the strongest RoboMemArena performance by combining keyframe memory with predictive coding, while end-to-end training and larger memory systems improve robotic memory performance. Ablations and real-world evaluation further support the importance of explicit memory, predictive state sensitivity, and scalable high-level reasoning.

  • Benchmark comparison: 21.5% average TSR and 38.7% average CSR are achieved by reactive π0.5, while closed-source GPT-5.4 reaches only 8.7% TSR on robotic memory-intensive tasks.Reactive policies perform relatively well when local visual regularities suffice, but closed-source agents transfer poorly to unseen robotic memory scenarios.
  • Benchmark comparison: PrediMem achieves the highest TSR and CSR among all methods by combining an explicit keyframe bank with predictive coding for more precise subtask management.The keyframe bank preserves task-relevant events beyond the recent observation window, while predictive coding increases sensitivity to physical state transitions.
  • Ablations: Removing the keyframe bank causes a broader performance drop than removing predictive coding, especially on occlusion and sequence tasks requiring earlier placements, inspections, or ordering decisions.Predictive coding is particularly important for subtle state transitions such as drawer closure, object disappearance, repeated pouring, and ordered-step completion.
  • Ablations: 0.1 yields the best performance as the coefficient balancing predictive loss LPre and instruction-tuning loss Ltext.The ablation varies the predictive-loss weight while evaluating the resulting model performance.
  • Scaling and real-world evaluation: A 3–5 frame recent window is sufficient for most short-term changes, larger S2 backbones consistently improve performance across tasks, and real-world average success rises from 20% with π0.5 to 40% with MemER.PrediMem further improves real-world performance beyond MemER by using explicit keyframe memory and predictive coding.

6 Conclusion … B VLM Training Prompt and JSON Format

RoboMemArena combines multimodal, keyframe-centered supervision, scalable long-horizon generation, and real-world evaluation, while PrediMem improves memory sensitivity through predictive coding. The appendices specify VLM prompts and strict JSON formats for generating subtasks and inferring current actions from visual history.

  • 6 Conclusion: RoboMemArena combines keyframe-centered multimodal supervision, scalable long-horizon trajectory generation, and paired real-world memory evaluation.It is designed as a diverse and challenging robotic memory benchmark.
  • 6 Conclusion: 68.9% of subtasks depend on past observations, making memory central rather than optional across RoboMemArena.
  • 6 Conclusion: PrediMem uses a dual-system memory framework whose predictive coding objective increases hidden-state sensitivity to physical state transitions without adding inference-time cost.
  • A VLM Input Prompt for Data Generation: The data-generation appendix uses a VLM prompt to decompose coarse tasks into executable subtasks and assign each to a predefined planner.The prompt grounds subtasks in the visible scene and preserves dependencies involving earlier placements, occluded objects, or counted actions.
  • A VLM Input Prompt for Data Generation: The generation prompt supports the predefined planners Move, Place, Pour, Open, and Close, and requires strict JSON output.
  • A VLM Input Prompt for Data Generation: Generated subtask records contain exactly one subtasks field with ordered entries specifying step_id, subtask, planner, and target.
  • B VLM Training Prompt and JSON Format: The training prompt infers the current primitive action from historical keyframes and a recent 5-timestep visual window containing agentview_rgb and eye_in_hand_rgb images.The recent window is primary evidence, and keyframe positions must be empty when no keyframe appears inside it.

C Asynchronous Inference Protocol

PrediMem uses asynchronous dual-system inference: S2 updates subtasks and keyframe decisions from recent visual history and memory, while S1 executes action chunks at a higher control rate using the freshest subtask.

  • Asynchronous inference: S2 asynchronously processes recent frames and memory M_t to emit a subtask and keyframe decision, overwriting the buffered subtask with each newer result.S2’s outputs drive the evolving high-level plan without blocking low-level control.
  • Asynchronous inference: S1 combines the current observation o_t with S2’s freshest subtask c_t to produce the next action chunk a_t.The low-level VLA action head operates at the higher control frequency.
  • Runtime profile: 1.06 Hz and 3.40 Hz are the respective S2 and S1 update rates, with each S2 update overlapping roughly 2.92 S1 chunks.This rate separation enables high-frequency action execution under asynchronously refreshed subtask predictions.
  • Recovery: Memory M_t retains prior events, allowing the agent to recover without restarting after asynchronous updates.Recovery is enabled by persistent memory rather than requiring a full task restart.

D Memory-Dependent Subtask Ratio Annotation

RoboMemArena labels subtasks as memory-dependent when earlier observations or task states are necessary to resolve the current high-level decision. Using this definition, 104 of 151 subtasks are memory-dependent, yielding a benchmark ratio of 68.9%.

  • Definition: A subtask is memory-dependent when the current observation alone cannot determine the correct action and earlier observations or subtasks are required.Removing execution history or observations would make the correct high-level decision ambiguous.
  • Benchmark ratio: 68.9% of RoboMemArena subtasks are memory-dependent, comprising 104 of 151 total subtasks.The numerator counts memory-dependent subtasks across all tasks, while the denominator sums all subtasks.
  • Annotation procedure: Subtasks are classified through task descriptions, keyframe annotations, and subtask decompositions, using manual inspection or an LLM-assisted rubric with manual checks.Small benchmarks are manually inspected, while larger benchmarks receive an LLM-assisted first pass before ambiguous cases are checked manually.
  • Memory-free subtasks: A subtask is memory-free when the required object and target state are visible in the current observation.Examples include picking up a visible cup or placing a visible cup on the table.
  • Memory-demand forms: Memory demand includes occlusion, counting, transferring, and sequence dependencies.These require remembering hidden object locations, prior action counts, source–target mappings, or completed prerequisite subtasks.

E Benchmark Task Details

Table S2 presents the 26 RoboMemArena benchmark tasks, organized by memory type, average total timesteps, and key challenges. The task descriptions continue across the following page.

  • Task Overview: 26 RoboMemArena tasks are described with their corresponding memory types, average total timesteps, and key challenges.The table provides an overview of the benchmark task set.
  • Task Overview: The benchmark task descriptions continue on the next page.This continuation is indicated directly in the provided passage.
  • Task Overview: Table S2 includes a continued portion of the benchmark task descriptions.The continuation retains the same table title and task-description context.

F Reactive Policy Failure Modes

Reactive policies fail on memory-dependent drawer tasks because they cannot track previously checked drawers or retain instruction-level constraints across visual-state resets. These failures explain the quantitative gap between reactive and memory-augmented policies.

  • Failure modes: Reactive policies exhibit two concrete failures on memory-dependent drawer tasks.The quantitative gap between reactive and memory-augmented policies maps to these failures.
  • Failure modes: They cannot distinguish whether a drawer has already been checked after the visual state resets.The drawer’s visual state becomes insufficient for identifying prior inspection.
  • Failure modes: They cannot preserve the instruction-level constraint that all drawers must be opened before final placement.This prevents the policy from maintaining the task requirement across the sequence.

G Real-World Task Details · H Verification-Step Distribution

The supplementary evaluation details a five-task physical-robot suite characterized by memory types, trajectory lengths, and task challenges. Verification-step counts range from 3–9 per task, with most tasks exceeding five steps to provide resolution for CSR-based memory comparisons.

  • G Real-World Task Details: Five physical-robot tasks form the real-world evaluation suite.The task overview organizes them by memory type, average total timesteps, and key challenges.
  • G Real-World Task Details: The real-world task descriptions report memory types for each physical-robot task.These categories support analysis of memory demands across the evaluation suite.
  • G Real-World Task Details: The task overview includes average total timesteps and key challenges for the five physical-robot tasks.These attributes characterize trajectory length and task difficulty in the physical evaluation.
  • G Real-World Task Details: Representative snapshots summarize the physical-robot task settings and example execution frames.The demonstrations also depict the dual-arm platform layout in a format similar to standard real-robot evaluation overviews.
  • H Verification-Step Distribution: Each task contains 3–9 verification steps.The verification-step distribution spans multiple counts across the task suite.
  • H Verification-Step Distribution: Most tasks exceed the long-horizon threshold of five verification steps.This distribution emphasizes tasks requiring extended verification sequences.
  • H Verification-Step Distribution: The verification-step histogram gives CSR sufficient resolution to distinguish agents with different memory capabilities.The distribution is designed to make cumulative success rate comparisons informative.
Loading 2605.10921v1…