PaperlayerView exact revision on arXiv

Source-linked AI summary

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, Zhou Zhao

arXiv:2609.07047v1cs.ROcs.AI

TL;DR

Robotic manipulation often depends on information no longer visible, while existing memory benchmarks rely mainly on final task success and conflate forgetting with manipulation failure. MEMOBench supplies executable, operation-labeled checkpoints to evaluate memory alongside task success. Policies still struggle with updating and compressing memory, and the benchmark also supports memory-alignment supervision.

  • Problem

    Robotic manipulation can depend on information absent from the current observation, while existing memory benchmarks rely mainly on final task success and conflate memory failure with manipulation failure.

    Source: §1 · p. 1

  • Method

    MEMOBench provides history-dependent tasks and executable, operation-labeled checkpoints that measure memory fidelity alongside task success without reading model internals.

    Source: §1 · pp. 1–2

  • Results

    The strongest memory module baseline reaches 31.9% average success rate, and high storage often coexists with weak update and compression.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MEMOBench serves as both a diagnostic evaluation suite and a source of checkpoint-language supervision for memory alignment.

    Source: §6 · p. 9

    The benchmark cannot fully capture real-world sensor noise, contact dynamics, or visual domain shift, and transfer of its findings and metrics to physical robots remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present \textbf{MEMOBench}, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9\% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.

1 Introduction

Robotic manipulation can require acting on information absent from the current observation, while existing memory benchmarks make it difficult to distinguish memory failures from manipulation failures. MEMOBench evaluates memory operations separately from task success, and experiments find persistent weaknesses in updating and compressing memory.

1 Introduction

  • Robotic manipulation may depend on stored references, changed states, or accumulated events that are absent from the current observation.The paper notes that these settings break the Markov assumption.Source: §1 · p. 1
  • Prior memory benchmarks rely mainly on final task success, making it difficult to distinguish forgetting from manipulation failure or identify which memory operation failed.Relevant failures include storing information, updating stale state, compressing history, or acting correctly after remembering.Source: §1 · p. 1
  • MEMOBench provides 30 history-dependent tasks, 1,500 expert demonstrations, and 4,200 executable checkpoint instances with language, simulator predicates, and Storage, Update, or Compression labels.The benchmark defines Memory Storage Rate, Memory Update Rate, and Memory Compression Rate to measure memory fidelity alongside task success, without reading model internals.Source: §1 · pp. 1–2
  • The strongest memory module baseline reaches 31.9% average success rate under its best replan setting, while high storage often coexists with weak update and compression.Checkpoint-language alignment yields modest gains across different memory operations.Source: §1 · p. 2

2 Related Work

General manipulation benchmarks typically evaluate tasks using the current observation and instruction, while newer benchmarks require reasoning over history. MEMOBench targets the remaining difficulty of diagnosing memory operations separately from task completion.

2 Related Work

  • General manipulation benchmarks cover diverse skills and platforms, but usually assume current observations and instructions suffice and score final completion rather than stored information used later.Examples include RLBench, CALVIN, LIBERO, ManiSkill3 (Tao et al., 2025), and BridgeData V2.Source: §2 · p. 2
  • History-dependent benchmarks expand memory evaluation, but commonly use subtask or keyframe annotations and final or stage success rates that mix memory fidelity with motor execution.Examples include MemoryBench, MIKASA-Robo (Cherepanov et al., 2026), LIBERO-Mem, RoboMME (Dai et al., 2026), RMBench (Chen et al., 2026), RoboCerebra (Han et al., 2025), and RoboMemArena (Lei et al., 2026).Source: §2 · p. 3
  • MEMOBench addresses this gap with household-object scenarios, coarse-to-fine checkpoint language, executable predicates, and process-level Storage, Update, and Compression metrics.It evaluates memory alongside manipulation success.Source: §2 · p. 3

3 MEMOBench Benchmark

MEMOBench organizes history-dependent manipulation tasks by memory type and operation, then annotates executable checkpoints for evaluating memory processes. It also uses checkpoint language and frame-level supervision to train memory-alignment objectives.

3.1 Memory Taxonomy

  • The benchmark's two-layer taxonomy distinguishes task memory types from checkpoint operations; the operations are mutually exclusive, and blind relabeling achieved Fleiss’ κ = 0.799.Task types cover temporal, spatial, object, and procedural memory; operations are Storage, Update, and Compression, with Update including deletion and Retrieval left unannotated.Source: §3.1 · p. 3

3.2 Task Suite

  • The task suite contains 30 BDDL manipulation tasks and 1,500 expert demonstrations across 29 object classes and five fixture classes.Temporal, spatial, object, and procedural task families have different operation profiles, interpreted through memory type and executable checkpoints.Source: §3.2 · p. 4

3.3 Memory Checkpoint Annotations

  • Each checkpoint pairs coarse-to-fine language with an operation label and executable subgoal predicate; 84 templates yield 4,200 demonstration-level instances.At default granularity, the suite includes 1,750 Storage, 1,600 Update, and 850 Compression instances; sequence-aware analyses count progress only when preceding checkpoints are completed.Source: §3.3 · p. 4

3.4 Memory Alignment from Annotations

  • Checkpoint annotations supervise Semantic, Contrastive, and Framewise Memory Alignment objectives using subgoal text and temporally aligned history frames.Alignment losses are training-only regularizers added to the memory module objective.Source: §3.4 · pp. 4–5, Figure 2 · p. 5

4 Evaluation Metrics

MEMOBench pairs final task success with checkpoint-based process metrics for Storage, Update, and Compression. The metrics evaluate whether episodes satisfy each checkpoint predicate and average scores only across tasks with that operation.

4 Evaluation Metrics

  • Task Success Rate (SR) requires every goal predicate to be satisfied at termination, measuring end-to-end manipulation performance.It does not localize memory failures.Source: §4 · p. 5
  • Process metrics evaluate simulator-state predicates at executable checkpoints labeled Storage, Update, or Compression.Source: §4 · p. 5
  • Memory Storage Rate (MSR), Memory Update Rate (MUR), and Memory Compression Rate (MCR) average checkpoint satisfaction, excluding tasks without the relevant operation.A checkpoint score is 1 when an episode satisfies its predicate and 0 otherwise.Source: §4 · p. 5

5 Experiments

The experiments evaluate VLA policies on MEMOBench, diagnose failures at memory checkpoints, and test whether checkpoint annotations can supervise memory alignment. Across settings, policies more reliably store information than update or compress it, while annotation-based alignment yields modest gains.

5.1 Experimental Setup

  • Policies are evaluated on all 30 tasks with 50 demonstrations and 50 rollout episodes per task, using replan intervals of 5, 20, 35, and 50 actions.The reported measures are SR, MSR, MUR, and MCR, grouped by memory type with macro averages over applicable types.Source: §5.1 · pp. 5–6

5.2 Main Results

  • 31.9% average SR is the strongest pre-alignment setting (π0.5 memory module, step 5); MemoryVLA leads average MUR, while HiF-VLA’s highest MCR accompanies 3.5% average SR.The π0.5 memory module has higher average MSR than MUR and MCR across replanning settings; its average SR falls from 31.9% at step 5 to 21.8% at step 50.Source: Table 2 · p. 6, §5.2 · pp. 6–7

5.6 Memory Alignment Results

  • Across replan settings, Contrastive and Framewise alignment average 29.3% SR versus 27.9% for the unaligned module; Framewise leads mean MSR, Semantic MUR, and Contrastive MCR.At step 5, Contrastive alignment raises average SR from 31.9% to 34.7%.Source: Table 3 · p. 7, §5.6 · p. 8

5.3 Error Analysis with Failure Mode Taxonomy

  • In 150 stove-task rollouts, 10 succeed; of 140 failures, 36 occur at initial storage, 63 at the first stale-memory update check, and 41 at a later stale-memory checkpoint.The taxonomy therefore localizes many temporal failures to updating and using changed state, rather than only missing the initial observation.Source: Figure 4 · p. 7, §5.3 · p. 7

5.4 Oracle Memory Decomposition

  • With ground-truth memory text, temporal and spatial MUR rise from 18.4% to 54.2% and 0.3% to 84.1%, while MCR rises from 2.0% to 53.0% and 6.4% to 60.7%.Oracle memory closes 45.5% of the MUR gap and 54.9% of the MCR gap on average, but only 20.2% of the MSR gap.Source: Table 4 · p. 7, §5.4 · p. 8

5.5 Shuffled Memory Control

  • Shuffled subgoals complete 90% of first-storage checkpoints but only 24% after the stove-state update and 2% final rollouts, versus 76%, 52%, and 8% with normal order.The control retains the initial action but loses advantage when the final action depends on the updated state.Source: §5.5 · p. 8

5.7 Annotation Granularity Ablation

  • Coarse checkpoint text yields 4/50 final successes, compared with 3/50 for intermediate and fine-grained text; intermediate text has 80% initial storage but drops at the update checkpoint.Fine-grained text recovers some update performance relative to intermediate text without improving final success.Source: §5.7 · pp. 8–9

5.8 Takeaways

  • The experiments find that initial storage is often easier than update and compression, and that checkpoint annotations can both diagnose memory operations and supervise alignment losses.The oracle decomposition attributes much of the remaining update and compression failure to memory; the paper identifies maintaining and consolidating relevant history as the central challenge.Source: §5.8 · p. 9

6 Conclusion

MEMOBench evaluates memory-grounded manipulation through task outcomes and process-level memory operations. The benchmark’s experiments show that policies store initial information more reliably than they update stale state or compress history for action.

6 Conclusion

  • MEMOBench pairs 30 history-dependent tasks and 1,500 demonstrations with checkpoint language and executable predicates for Storage, Update, and Compression.It evaluates both task outcomes and process-level memory operations.Source: §6 · p. 9
  • Across standard and memory-augmented VLA policies, initial information is stored more reliably than stale state is updated or history compressed for action.The conclusion frames memory fidelity as a complement to final task success.Source: §6 · p. 9
  • Checkpoint language also supplies training supervision for memory alignment, enabling comparison by which operations policies improve rather than only by final success.Source: §6 · p. 9

Limitations

The stated limitations concern simulation-to-robot transfer, model-family coverage, and human judgment in checkpoint annotation.

Simulation Evaluation

  • MEMOBench is built entirely in simulation, so sensor noise, contact dynamics, and visual domain shift remain outside its evaluation.Whether the findings and process-level metrics transfer to physical robots remains open.Source: Limitations · p. 9

Model Coverage

  • The main comparison covers six models, but alignment experiments use only π0.5 memory-module variants and omit other families such as state space models.The authors say broader model-family evaluation would strengthen generalizability.Source: Limitations · p. 9

Annotation Subjectivity

  • Checkpoint operation labels require human judgment, with some subjectivity remaining despite guidelines, quality checks, and substantial relabeling agreement.Disagreements concentrate on the boundary between Update and Compression.Source: Limitations · p. 9

Ethics Statement

The ethics statement describes the benchmark's intended diagnostic use, planned artifact licensing, simulated data contents, annotation process, and use of AI assistance.

Potential Risks

  • MEMOBench is intended for diagnostic evaluation and training analysis, not direct deployment in safety-critical settings.The authors identify no additional risks beyond standard research use of simulated robot benchmarks.Source: Ethics Statement · p. 9

Artifacts and Licenses

  • The authors plan Apache-2.0 licensing for code and CC BY 4.0 licensing for the dataset, task specifications, and annotations.The work builds on existing robotics and VLA artifacts used compatibly with their terms.Source: Ethics Statement · p. 9

Data Content

  • The simulated dataset contains task specifications, trajectories, simulator states, rendered observations, and checkpoint annotations, but no personal data or real household videos.It also excludes faces, speech, and offensive text.Source: Ethics Statement · p. 10

Human Annotation

  • Authors created and reviewed the annotations; internal lab members unaware of the study and not involved in labeling conducted the blind relabeling.No external annotators or human subjects were recruited, and no payment, consent procedure, or ethics-board review was involved.Source: Ethics Statement · p. 10

AI Assistance

  • AI assistants were used for sentence polishing, not to generate results, create annotations, or make scientific decisions.Source: Ethics Statement · p. 10

A Task Suite Details

The task suite expands 30 tasks into 84 executable checkpoint templates and 4,200 demonstration-level instances, with uneven operation counts and varied episode lengths.

A Task Suite Details

  • 84 templates yield 4,200 checkpoint instances: 1,750 Storage, 1,600 Update, and 850 Compression, with uneven counts reflecting task-family demands.The suite covers all four memory types; object-reference tasks mainly test Storage, while spatial drawer tasks contribute all three operations.Source: A Task Suite Details · p. 11
  • The dataset contains 927,444 action steps in 340.56 GiB; procedural demonstrations average 1,000.82 steps and reach 2,012 steps.Each task has 50 demonstrations, so family differences mainly reflect horizon length; spatial tasks occupy 131.58 GiB across 10 tasks and 500 episodes.Source: A Task Suite Details · p. 11
  • Episode lengths span compact and long-horizon cases: 687 demonstrations contain 400–599 steps, 456 have at least 800, and 25 have at least 1,500.The shortest episodes are temporal top-drawer checks, while the longest come from procedural buffered object swaps.Source: A Task Suite Details · p. 11

B Data Collection and Quality Control

MEMOBench builds task and checkpoint specifications on simulator predicates, collects expert demonstrations, and checks annotations and execution for consistency.

B Data Collection and Quality Control

  • Across 1,500 expert demonstration episodes, MEMOBench contains 927,444 control steps, with a mean episode length of 618.30 steps.Tasks are instantiated from BDDL specifications on the LIBERO and robosuite stack.Source: B Data Collection and Quality Control · p. 12
  • Evaluation randomizes initial states within BDDL regions, stabilizes for 10 steps, then gives policies a 1,200-step horizon and RGB, proprioceptive, and language inputs.The task definitions require preserving earlier references, state changes, or demonstrated routines across decisions.Source: B Data Collection and Quality Control · p. 12
  • Each HDF5 episode stores actions, states, and third-person and wrist RGB frames; actions have shape (T, 7), and images have shape (T, 256, 256, 3).State tensors have shape (T, D), with task-dependent D; all action, state, and image streams share episode length T.Source: B Data Collection and Quality Control · p. 12
  • Checkpoint predicates are evaluated after every policy step, while final task success uses the BDDL goal; sequence-aware summaries apply an ordered prefix gate.The logger records whether each predicate has ever been satisfied, making checkpoint completion a process measure rather than a direct probe of hidden memory states.Source: B Data Collection and Quality Control · p. 13
  • Quality control checks consistency among BDDL goals, checkpoint predicates, and operation labels, which are checked against the Storage, Update, and Compression taxonomy.Simulator execution surfaces invalid objects, fixtures, regions, or predicates during instantiation or rollout.Source: B Data Collection and Quality Control · p. 13

C Blind Annotator Agreement Study

The supplementary sections assess operation-label agreement, detail the process metrics and alignment objectives, and report radar and privileged-supervision results. The findings show operation-specific performance differences and persistent room for improvement.

C Blind Annotator Agreement Study

  • Fleiss’ κ = 0.799, and majority vote matched released labels on 84.5% of templates.All three raters agreed on 70 of 84 templates; every disagreement was a one-versus-two split.Source: C Blind Annotator Agreement Study · p. 13
  • The main disagreement involved Compression versus Update when a checkpoint combined a physical state change with completion of a multi-step sequence.The adjudication rule reserves Compression for predicates or language requiring an aggregate over prior events or facts, Update for a later state change of a tracked entity, and Storage for a new record about a distinct entity.Source: C Blind Annotator Agreement Study · p. 13, Table 9 · p. 13

D Metric Formulation Details

  • Memory Storage Rate, Memory Update Rate, and Memory Compression Rate track checkpoint completion by operation, distinguishing process completion from final task success.A selective drop in one process metric localizes failure to that operation’s checkpoint predicates; tasks without that operation are excluded from its macro average.Source: D Metric Formulation Details · pp. 13–14

E Detailed Memory Alignment Objectives

  • Semantic Memory Alignment projects pooled memory toward an annotated language target as a lightweight regularizer without batch negatives.The text target is the annotated simple_subgoal, encoded offline with a frozen SigLIP text encoder.Source: E Detailed Memory Alignment Objectives · p. 14
  • Contrastive Memory Alignment makes pooled memory discriminative across subgoals, treating repeated subgoals within a batch as positives.Each memory vector retrieves its matching subgoal from batch candidates while avoiding false-negative gradients for repeated labels.Source: E Detailed Memory Alignment Objectives · p. 14
  • Framewise Memory Alignment preserves the temporal layout of sampled history by aligning frame representations with annotated subgoals.Valid frames use the annotation at the same episode step or the next available annotated step; frames beyond the last annotation use the final one, and the reported variant applies linear recency weighting.Source: E Detailed Memory Alignment Objectives · p. 15

G MemER with Privileged Supervision

F Radar Summaries

  • Under the best-over-replan radar rule, the π0.5 memory module leads SR and MSR, MemoryVLA leads MUR, and HiF-VLA leads MCR.Contrastive Memory Alignment leads SR, MUR, and MCR in the alignment comparison, while Semantic Memory Alignment leads MSR; the radar normalizes each axis by its maximum.Source: F Radar Summaries · p. 15, Figure 7 · p. 15

H Reproducibility Settings

The section points to a table of reproducibility settings for MEMOBench evaluation and controlled π0.5 alignment runs, which share the RoboMME memory module backbone.

H Reproducibility Settings

Related papers

Loading 2609.07047v1…