PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Haoran Zhang, Jianing Yang, Chelsea Finn, Nima Fazeli, Joyce Chai

arXiv:2603.04639v3cs.ROcs.AI

TL;DR

RoboMME addresses limited and inconsistent evaluation of memory for long-horizon, history-dependent robotic manipulation. It introduces a unified benchmark and controlled π0.5-based VLA suite, finding that memory effectiveness is task-dependent rather than universally dominated by one design.

  • Problem

    Existing memory methods use different policy backbones and inconsistent evaluation protocols, while benchmarks lack diverse and challenging memory requirements for robotic manipulation.

  • Method

    RoboMME provides 16 tasks across temporal, spatial, object, and procedural memory, alongside memory-augmented VLA models based on the π0.5 backbone that compare memory representations and integration mechanisms under controlled settings.

  • Results

    No single memory representation or integration strategy consistently dominates; symbolic memory excels at counting and short-horizon reasoning, whereas perceptual memory is crucial for motion-centric and time-sensitive behaviors.

  • Takeaways & Limitations

    Memory designs should be understood as task-dependent, with perceptual memory combined with memory-as-modulator offering the best reported balance between performance and computational efficiency.

  • Takeaways & Limitations

    RoboMME focuses on tabletop manipulation with fixed assets and mainly evaluates a single pretrained backbone, π0.5, leaving mobile manipulation and alternative architectures for future study.

Abstract

from arXiv · show

Memory is critical for long-horizon and history-dependent robotic manipulation. Such tasks often involve counting repeated actions or manipulating objects that become temporarily occluded. Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms; however, their evaluations remain confined to narrow, non-standardized settings. This limits systematic understanding, comparison, and progress measurement. To address these challenges, we introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models in long-horizon, history-dependent scenarios. Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy that evaluates temporal, spatial, object, and procedural memory. We further develop a suite of 14 memory-augmented VLA variants built on the π0.5 backbone to systematically explore different memory representations across multiple integration strategies. Experimental results show that the effectiveness of memory representations is highly task-dependent, with each design offering distinct advantages and limitations across different tasks. Videos and code can be found at our website https://robomme.github.io.

1. Introduction

RoboMME addresses the lack of standardized, diverse evaluations for memory-augmented robotic manipulation by introducing a benchmark and controlled VLA model suite. Results show that memory effectiveness depends strongly on the task and representation.

  • History-dependent manipulation requires retaining and reusing information because immediate perception alone is insufficient for tasks involving repeated actions, occlusion, or demonstrations.
  • Existing memory methods use symbolic, perceptual, or recurrent representations, but differing backbones and evaluation protocols obscure which designs generalize across tasks.
  • RoboMME contains 16 long-horizon tasks spanning temporal, spatial, object, and procedural memory, with 1,600 demonstrations and 770k high-quality timesteps.
  • The study develops 14 memory-augmented VLA variants based on the π0.5 backbone to compare memory representations and integration mechanisms under controlled conditions.
  • No single memory design performs consistently well across all tasks: symbolic memory excels at counting and short-horizon reasoning, whereas perceptual memory benefits time-sensitive and motion-centric behaviors.

2. Related Work

Prior benchmarks and policies only partially test history-dependent robotic manipulation. RoboMME is positioned as a unified, challenging testbed for systematic comparison.

  • Many existing manipulation benchmarks permit high success from current observations, so they do not explicitly require true history-based reasoning.
  • MemoryBench focuses on spatial recall but has only three simple, nearly solved tasks, while MIKASA-Robo remains short-horizon and lacks sufficient high-quality imitation-learning demonstrations.
  • Recent memory-based policies are commonly evaluated on narrow, self-designed tasks, hindering systematic comparison and understanding.
  • Existing approaches are categorized as symbolic, perceptual, or recurrent according to how they represent interaction history.

3. RoboMME Benchmark

RoboMME constructs non-Markovian, long-horizon tabletop tasks around four cognitive memory dimensions. Its suites and dataset are designed to expose history dependence across varied manipulation demands.

  • RoboMME tasks are intentionally non-Markovian: identical current observations can result from different histories and require different actions.
  • The benchmark evaluates temporal, spatial, object, and procedural memory, corresponding to when, where, what, and how information must be retained.
  • Counting, Permanence, Reference, and Imitation provide four task suites, each emphasizing one primary memory type for controlled evaluation.
  • The dataset contains 16 tasks with 100 episodes each, totaling 1,600 demonstrations and 770k timesteps after retaining successful planner rollouts.
  • RoboMME systematically covers four memory types while adding environmental complexity and video-conditioned tasks beyond prior benchmarks.

4. Memory-Augmented Manipulation Policies

The MME-VLA suite studies symbolic, perceptual, and recurrent memory representations with three integration mechanisms on the π0.5 backbone. The framework separates how history is represented from how it conditions action prediction.

  • The MME-VLA suite is a family of memory-augmented VLA models built on the π0.5 backbone for controlled comparisons of memory designs.
  • Memory Representations: Symbolic memory summarizes history as language subgoals, while perceptual memory retains selected visual tokens and recurrent memory compresses history into fixed-size latent states.
  • Memory Representations: Perceptual memory uses token dropping or frame sampling to balance historical visual information against context-length growth.
  • Integration Mechanisms: The framework compares three memory representations across multiple integration strategies, including designs that preserve current-observation processing while injecting historical information into action prediction.
  • Integration Mechanisms: Memory-as-Context concatenates memory with inputs, memory-as-Modulator conditions the action expert through adaptive LayerNorm, and memory-as-Expert uses a dedicated memory pathway.

5. Experiments

The experiments compare 14 memory-augmented π0.5 VLA policies and prior methods under controlled evaluation across RoboMME’s tasks. Results show that memory effectiveness depends strongly on task characteristics, with perceptual memory and memory-as-modulator performing especially well overall while symbolic memory remains advantageous for some tasks.

  • Experiment Setup: 14 VLA policies and 4 prior methods are compared under multi-task training and controlled evaluation protocols.Models are evaluated across all 16 tasks with fixed memory budgets, 50 episodes per task, and maximum horizons of 1,300 steps.
  • Experiment Setup: The MME-VLA suite spans symbolic, perceptual, and recurrent memory integrated through context, modulator, or expert mechanisms.Symbolic variants use SimpleSG or GroundSG, while perceptual and recurrent variants use TokenDrop or FrameSamp, and TTT or RMT, respectively.
  • Main Results: No single memory representation or integration strategy dominates across all tasks.Symbolic memory performs well on counting tasks, whereas perceptual memory is crucial for time-sensitive behaviors and motion imitation.
  • Main Results: 44.51% overall success is achieved by FrameSamp+Modul, the strongest non-oracle variant across MME-VLA models.Most perceptual variants outperform the best symbolic and recurrent counterparts, while memory-as-modulator performs best among the tested integration mechanisms.
  • Main Results: High-level symbolic reasoning alone remains insufficient for manipulation-intensive and cluttered-scene tasks.GroundSG+Oracle still degrades on StopCube and InsertPeg and may select incorrect objects or cause collisions in BinFill and PickHighlight.
  • Main Results: Symbolic memory performs best on counting tasks such as PutFruits, while perceptual memory excels on motion-centric tasks such as DrawPattern.The remaining two real-world tasks show comparable performance between the approaches, leaving room for improvement.

6. Conclusion and Future Work

RoboMME provides a unified benchmark and controlled VLA comparison for memory-augmented robotic manipulation. Its results show that memory designs are complementary, while the benchmark remains scoped mainly to tabletop manipulation and the π0.5 backbone.

  • Benchmark and contributions: RoboMME evaluates memory-augmented robotic manipulation across temporal, spatial, object, and procedural memory dimensions.The benchmark is presented as a unified framework for systematic evaluation.
  • Benchmark and contributions: The study compares symbolic, perceptual, and recurrent memory representations using multiple integration mechanisms in a family of VLA models.The MME-VLA Suite is built primarily around controlled comparisons on the π0.5 backbone.
  • Findings: No single memory design consistently dominates: symbolic memory suits counting and short-horizon reasoning, whereas perceptual memory supports motion-centric and time-sensitive behaviors.The reported conclusion emphasizes task-dependent strengths rather than a universal winner.
  • Limitations and future work: RoboMME focuses on tabletop manipulation with a fixed asset set and mainly evaluates one pretrained backbone, π0.5.Mobile manipulation, alternative architectures, memory-bank methods, and additional VLA backbones remain outside the main evaluation scope.
  • Limitations and future work: The authors suggest that complementary memory representations motivate unified frameworks combining multiple forms of memory.This direction follows the reported observation that memory representations are complementary rather than exclusive.

A. Model Architectures in the MME-VLA Suite

The MME-VLA suite uses a shared π0.5-based VLA architecture with separate vision-language and action experts, then augments it with symbolic, perceptual, or recurrent memory under multiple integration strategies.

  • Base Architecture: The suite builds VLA policies on a shared π0.5 backbone with a VLM expert for joint visual-language features and an action expert for action prediction.Both experts use transformer layers with blockwise causal attention and MLPs.
  • Base Architecture: At each timestep, language instruction tokens and current multi-view image tokens are processed to produce features for action prediction.The VLM is used as a feature extractor rather than directly generating low-level commands.
  • Memory Representations: Memory states summarize past observations and actions, with every memory-based model constrained to a fixed token budget B for fair comparison.Sequences shorter than B tokens are zero-padded so all variants share the same interface.
  • Memory Representations: Symbolic memory converts trajectory history into compact language subgoals, while perceptual memory retains selected visual tokens for differentiable end-to-end processing.Perceptual memory uses frame sampling or token dropping to select visual history under the memory budget.
  • Memory Representations: Recurrent memory compresses history into bounded latent states using recurrent memory slots or online-updated fast weights.RMT stores history in persistent token slots, whereas TTT stores it in parameter space.
  • Memory Integration: Memory enters the policy as concatenated context, feature-wise modulation, or a dedicated memory-expert pathway.These mechanisms respectively process memory jointly with inputs, condition action features through AdaLN, or isolate memory processing before action attention.

B.3. Training Details for SAM2Act+

SAM2Act+ is adapted with a two-stage training pipeline and video-based memory initialization, but its discrete waypoint action space limits precision on continuous-control tasks.

  • Training Pipeline: SAM2Act+ is trained in two stages: SAM2 pretraining without temporal connections followed by fine-tuning of the memory attention module.Each stage uses 40k steps on the RoboMME dataset.
  • Action Space: Discrete waypoint actions degrade performance on precise continuous-control tasks such as StopCube and InsertPeg.Waypoint predictions are executed by an internal ManiSkill motion planner, while discrete actions are sufficient for the remaining tasks.
  • Video Conditioning: At test time, the demonstration video is rolled out first to prefill the memory bank before simulator interaction begins.The agent then executes using the current observation after memory initialization.

B.4. Training Details for OpenVLA-OFT

OpenVLA-OFT uses architecture-compatible memory-as-context variants, while MemoryVLA and Diffusion Policy provide adapted and non-VLA reference points alongside oracle and human evaluations.

  • OpenVLA-OFT: OpenVLA-OFT uses a single dense transformer, making dedicated action-expert integration mechanisms less suitable than memory-as-context.Vision, language, proprioception, and action tokens share one LLaMA backbone.
  • OpenVLA-OFT: The OpenVLA-OFT evaluation covers symbolic memory and perceptual memory-as-context using FrameSamp and TokenDrop from the OpenVLA-7B starting point.LoRA fine-tunes the LLaMA backbone while the action head and memory projections are fully trained.
  • MemoryVLA: MemoryVLA is adapted to RoboMME while retaining its native episodic-memory modules rather than using the proposed integration mechanisms.Its backbone is a Prismatic VLM with a Diffusion Transformer action head.
  • Diffusion Policy: Diffusion Policy serves as a non-VLA baseline without large-scale pretraining or explicit memory, using only a short observation window.It combines visual encoders, robot state, and projected language features for action prediction.
  • Oracle and Human Evaluation: Oracle evaluations use ground-truth subgoals or incremental VideoQA-style decisions executed by an oracle planner.The study also evaluates proprietary foundation models and 18 human participants across 800 episodes.

C.1. Results Using the Oracle Planner

Oracle-planner results show that humans perform well overall but still struggle on tasks requiring sustained, long-horizon memory. Across model evaluations, no single method fully solves an entire task suite, motivating functional rather than one-to-one memory comparisons.

  • Human and foundation-model results: Human participants perform well on most tasks, but Permanence, DrawPattern, RouteStick, SwingXTimes, and StopCube expose failures under strong memory demands.Failures involve sustained attention, forgotten trajectory details or intermediate goals, and losing track of action counts during incremental video revelation.
  • Human and foundation-model results: RoboMME imposes substantial long-term memory demands even for humans.
  • Task-suite comparisons: The benchmark compares MME-VLA variants and prior methods across task suites, with results regrouped by functional characteristics.Table 12 reports per-task-suite comparisons, while Tables 13 and 14 categorize and average results by functional requirements.
  • Task-suite comparisons: No single method fully solves an entire task suite, so task demands do not map one-to-one onto cognitive memory types and model memory representations.

C.4. Full Results on All Tasks

Full-task results show substantial variation across backbones and methods. OpenVLA-OFT is less effective than π0.5 in this setting, while perceptual and symbolic memory each retain task-dependent strengths.

  • Full-task comparisons: Below 10% overall success rate is achieved by both U-Net-based Diffusion Policy and MemoryVLA.
  • Full-task comparisons: The full-results evaluation provides per-task comparisons across MME-VLA models and additional baselines.The results are presented in Table 15; the leaderboard is updated as models improve.
  • Backbone comparison: 21.6% is achieved by OpenVLA-OFT with SimpleSG+QwenVL, compared with 19.5% for GroundSG+QwenVL, 9.1% for TokenDrop+Context, and 9.6% for FrameSamp+Context.
  • Backbone comparison: OpenVLA-OFT is less effective than π0.5 in this setting, particularly for leveraging grounded symbolic information and perceptual memory.The authors attribute the symbolic-memory difference partly to π0.5's visual-grounding co-training and report limited perceptual-memory gains for OpenVLA-OFT.

C.5. Discussion on Memory Token Contribution

Additional analyses show that historical information improves performance, is used selectively in attention-based designs, and can directly condition action features through modulation. These findings support perceptual memory, especially FrameSamp+Modul, as the strongest overall design in the reported experiments.

  • Effect of memory budget: 45.87% is achieved by FrameSamp+Modul with 1024 memory tokens, up from 30.42% with 64 tokens; TokenDrop+Modul rises from 18.11% to 40.17%.Performance gains become smaller beyond 512 tokens, motivating 512 tokens as the main-experiment trade-off between performance and computational cost.
  • Attention allocation: Memory tokens receive smaller average attention mass than observation and language tokens, but stronger models often assign higher peak attention to memory or language.FrameSamp+Expert has the best success rate among the analyzed variants and the highest maximum memory attention.
  • Effect of memory modulation: Memory-as-modulator predicts feature-wise scale and bias terms that directly modulate the action expert instead of adding context tokens.
  • Effect of memory modulation: FrameSamp+Modul achieves higher success with smaller scale and bias variance, suggesting that stable modulation benefits action prediction.The analysis reports non-trivial modulation signals for both TokenDrop+Modul and FrameSamp+Modul.
  • Overall interpretation: Memory contributes through complementary mechanisms: larger budgets preserve useful history, attention uses memory sparsely, and modulation directly conditions the action expert.

D. Real Robot Experiment

The real-robot evaluation mirrors four simulation tasks with four history-dependent manipulation tasks and uses demonstrations collected on a Franka Panda platform. Results show the same task-dependent pattern as simulation: perceptual memory is strongest for motion-centric behavior, while symbolic memory helps event counting.

  • Task design and platform: Four real-world tasks—PutFruits, TrackCube, RepickBlock, and DrawPattern—mirror four simulation tasks, respectively.
  • Task design and platform: 350 demonstrations and 78,400 timesteps are collected using Oculus Quest 2 teleoperation at 15 Hz.PutFruits has 50 demonstrations, while each of the other three tasks has 100.
  • Task design and platform: The experiments use a 7-DoF Franka Emika Panda with UMI fin-ray fingers in a tabletop environment with RGB-D cameras.
  • Results: Perceptual memory performs best on DrawPattern, symbolic memory performs better on PutFruits, and both achieve moderate performance on RepickBlock.
  • Results: Perceptual memory performs slightly better on TrackCube because the VLM subgoal predictor often fails to ground dynamic swapping accurately.

E. Task Description

RoboMME organizes 16 manipulation tasks into four memory-focused suites, using varied objects, receptacles, instructions, and dynamic conditions to test history-dependent behavior. The tasks require exact counting, persistent tracking, referential resolution, or reproduction of demonstrated procedures.

  • Benchmark organization: RoboMME maps temporal, spatial, object, and procedural memory to the Counting, Permanence, Reference, and Imitation suites, respectively.Each suite contains four tasks.
  • Environment and objects: The benchmark uses tabletop cubes, pegs, sticks, highlighted areas, colored targets, buttons, and specialized receptacles such as bins and containers.These objects support pick-and-place, insertion, pulling, navigation, temporary visual cues, state transitions, termination, and occlusion-based memory demands.
  • Counting tasks: BinFill requires placing specified colored cubes into an opaque bin, with static or incrementally streaming cubes, followed by exact-count termination.Language goals range from one specified color to combinations of two or three colors; over- or under-counting causes failure, and exceeding a required count can terminate the episode immediately.
  • Repetitive manipulation tasks: PickXTimes and SwingXtimes require exact repetitions of pick-and-place or right-to-left transport cycles before pressing the termination button.Both tasks define single- and multiple-cycle language goals, evaluate successful manipulation using geometric conditions, and terminate early for wrong objects, premature stopping, or excessive repetitions.
  • Dynamic timing task: StopCube combines continuous visual tracking with event counting by requiring a button press when an oscillating cube reaches a target on a specified visit.Success requires hovering over the button, synchronizing the press with the target-overlap window, and avoiding early or late termination.
Loading 2603.04639v3…