Source-linked AI summary
RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, Hongcheng Wang, Junjie Wang, Tianhang Yang, Renjing Xu, Ruihai Wu, Yao Mu, Yaodong Yang, Hao Dong, Ping Luo
TL;DR
Existing robotic manipulation policies and benchmarks provide limited systematic support for tasks requiring long-term retention and use of historical information. The paper introduces RMBench, a nine-task benchmark organized by memory complexity, and Mem-0, a modular policy with explicit memory components. Evaluations show that existing policies underperform on memory-dependent tasks, while Mem-0 improves success rates across task categories but remains limited by semantic understanding, placement accuracy, and transition detection.
Problem
Existing policies rely heavily on short observation histories, while systematic evaluation of memory-dependent manipulation and architectural memory mechanisms remains limited.
Method
The paper introduces Task Memory Complexity and RMBench's nine-task simulation benchmark, together with Mem-0's modular dual-system policy and task-phase classifier.
Results
Mem-0 improves average success rates by 38.4% on M(1) tasks and 21.2% on M(n) tasks relative to baselines, while existing baselines consistently underperform on memory-dependent tasks.
Takeaways & Limitations
The experiments provide empirical insights into how memory modules and architectural choices affect memory performance in robotic manipulation.
Takeaways & Limitations
Mem-0 remains limited on semantically demanding and fine-grained tasks, and its simple subtask-transition classifier can cause termination-detection errors.
Abstract
from arXiv · showhide
Robotic manipulation policies have made rapid progress in recent years, yet most existing approaches give limited consideration to memory capabilities. Consequently, they struggle to solve tasks that require reasoning over historical observations and maintaining task-relevant information over time, which are common requirements in real-world manipulation scenarios. Although several memory-aware policies have been proposed, systematic evaluation of memory-dependent manipulation remains underexplored, and the relationship between architectural design choices and memory performance is still not well understood. To address this gap, we introduce RMBench, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities. We further propose Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies. Through extensive simulation and real-world experiments, we identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance. The website is available at https://rmbench.github.io/.
1. Introduction
Existing robotic manipulation policies often assume short-horizon, approximately Markovian decisions, limiting performance on tasks requiring historical information. The paper addresses this gap with RMBench and Mem-0, enabling systematic evaluation and analysis of memory mechanisms.
- Motivation: Most existing policies rely on fixed-length recent observations and implicitly assume approximately Markovian decision processes.This limits their ability to handle long-horizon, memory-dependent manipulation.
- Motivation: Memory-dependent tasks require retaining, retrieving, and using information from past observations and actions over extended horizons.Examples include remembering object locations and reasoning across repeated attempts.
- Research gap: Existing memory-aware policies have emerged, but the field lacks a systematically designed platform for evaluating long-term memory requirements and mechanisms.Prior benchmarks provide incomplete coverage or limited task-design guidance.
- Contributions: Task Memory Complexity provides a principled metric for characterizing memory requirements and guides RMBench's design of nine dual-arm tasks spanning multiple complexity levels.The benchmark supports controlled studies of memory retention and utilization.
- Contributions: Mem-0 uses a modular dual-system architecture with a task-phase classifier to support structured memory usage across long horizons.Its replaceable components enable controlled ablation studies of policy design mechanisms.
- Contributions: Comprehensive evaluations and Mem-0 ablations reveal memory-related limitations in existing policies and identify beneficial design mechanisms.The study analyzes which components most support memory in robotic manipulation.
2. Related Work
Simulation benchmarks and manipulation policies have advanced substantially, but existing resources and architectures provide incomplete support for systematic evaluation of memory-dependent behavior. Mem-0 builds on recent memory-aware policy work to enable controlled analysis of memory components.
- Benchmarks: Many manipulation simulators emphasize short-horizon interactions or tasks solvable without historical observations.This limits their suitability for evaluating memory-dependent manipulation.
- Benchmarks: MemoryBench offers seven memory-related tasks but has limited simulation reproducibility, while MIKASA is primarily tailored to reinforcement learning.LIBERO-Long and RoboCerebra include long-horizon tasks whose relevant information remains observable.
- Policies: Many modern policies use fixed-length observation histories, limiting selective retention of task-relevant information over long horizons.Recent memory-aware approaches address this limitation by explicitly incorporating memory mechanisms.
- Policies: Mem-0 extends memory-aware policy research with a modular design intended to facilitate systematic ablation and analysis of memory components.Its design directly supports studying architectural choices for memory-dependent manipulation.
3. RMBench
RMBench defines task memory requirements through Task Memory Complexity and organizes nine simulation tasks across different levels of historical dependence. Its formulation and annotations support controlled, reproducible evaluation of memory retention and retrieval.
- Task Memory Complexity: Task Memory Complexity characterizes the minimum number of task-relevant past observations an optimal policy must retain.The notation M(0), M(1), and M(n) indicates progressively greater memory requirements.
- Benchmark design: RMBench comprises nine manipulation tasks with varying memory demands, enabling controlled evaluation across Task Memory Complexity levels.The benchmark is developed within RoboTwin 2.0 and supports integrated policy evaluation.
- Benchmark design: The benchmark addresses partial observability in which current observations may not determine task progress or the next action without historical information.Relevant observations may occur at arbitrary time steps rather than in a contiguous recent window.
- Task Memory Complexity: M(0) tasks are memory-free, M(1) tasks require one past observation, and M(n) tasks require n task-relevant observations.These categories capture nonlocal and multi-step temporal dependencies.
- Data and evaluation: Fine-grained language annotations align with action–observation pairs to provide structured supervision for high-level reasoning or memory modules.These annotations describe low-level interactions and state transitions.
- Task families: The nine tasks comprise five M(1) tasks and four M(n) tasks with distinct memory demands.M(1) tasks require limited historical frames, whereas M(n) tasks require repeated exploration, trial-and-error, or attempt tracking.
4. Mem-0 Policy
Mem-0 is a modular memory-oriented policy that separates subtask-level planning from low-level execution and connects them through completion detection. Its structured memories support iterative planning, memory-conditioned action generation, and closed-loop control across long-horizon tasks.
- Architecture: Mem-0 combines a key-memory Planning Module, a memory-conditioned Execution Module, and a Subtask End Classifier for closed-loop manipulation.The modular design enables fine-grained analysis of different memory components.
- Planning Module: The Planning Module uses the initial observation, task goal, and finished-task memory to predict the next subtask.Finished-task memory aggregates previously completed subtasks and their termination observations.
- Planning Module: Conditioning on completed subtasks and visual outcomes enables subtask inference beyond planning from a single observation.This design targets tasks where accurate subtask inference requires reasoning over multiple previously completed subtasks.
- Planning Module: Planning occurs at subtask termination rather than every observation frame, reducing invocations from O(T) to O(N) when N ≪ T.The Execution Module can therefore operate at high control frequency within each subtask.
- Execution Module: The Execution Module generates diffusion-policy actions from current perception, the active subtask, and fused anchor and sliding memories.The predicted action sequence has horizon H = 30, and a prefix is executed before replanning.
- Subtask End Classifier: A temporal consistency rule requires L = 8 consecutive termination predictions before ending a subtask and passing its final observation to planning.This criterion is intended to improve robustness against premature termination caused by transient noise.
5. Experiment
The experiments evaluate policies on RMBench, analyze Mem-0's memory components through ablations, and test real-world generalization. Mem-0 improves memory-dependent manipulation performance but remains limited by semantic understanding, fine-grained placement, and subtask termination.
- 5. Experiment: Experiments evaluate existing policies and Mem-0 on RMBench, ablate Mem-0 modules, and assess real-world performance beyond simulation.RMBench evaluations use multiple policy types, while real-world tests compare Mem-0 with ACT and Pi0.5.
- 5.1. Evaluation of Policies on RMBench: Baseline policies underperform because their Markovian design cannot infer correct actions without task-relevant historical information.Representative baseline failures are illustrated in Figure 3.
- 5.1. Evaluation of Policies on RMBench: 38.4% and 21.2% are Mem-0's average success-rate improvements on M(1) and M(n) tasks, respectively, relative to baselines.These results support explicit memory mechanisms for memory-dependent manipulation in RMBench.
- 5.1. Evaluation of Policies on RMBench: Mem-0 is disadvantaged on semantically demanding tasks, while Swap T exposes limited placement accuracy and Press Button exposes unreliable termination detection.Pretrained models retain an advantage on Observe and Pick Up; Press Button failures include repeated presses and missed contacts.
- 5.2. Analysis on Memory-Related Module: Removing anchor or sliding memory generally degrades performance, while removing sliding memory can improve Swap T by reducing interference from transient motion cues.Anchor memory preserves task-critical information, whereas sliding memory captures short-term motion trends and can either help or interfere.
- 5.2. Analysis on Memory-Related Module: Key memory supports long-term subtask inference, while inaccurate subtask-transition timing can undermine planning despite the benefit of decomposition.Ground-truth classifier results indicate that subtask decomposition can outperform fully end-to-end approaches, but the current classifier lacks precision.
- 5.3. Real World Experiment: Mem-0 outperforms ACT and Pi0.5 in real-world experiments, but most failures arise from imprecise block manipulation rather than high-level planning.The authors associate these failures with variable human demonstrations and the absence of dedicated robotic-manipulation pretraining.
6. Conclusion
RMBench and Mem-0 provide a systematic framework for evaluating memory in robotic manipulation and examining architectural choices. The paper identifies explicit memory modeling as beneficial while pointing to improved representations, termination criteria, and pretraining as future directions.
- 6. Conclusion: RMBench and Mem-0 systematically evaluate memory in robotic manipulation and reveal how anchor, sliding, and key memory affect performance.The framework offers preliminary insights into integrating memory mechanisms for memory-dependent manipulation.
- 6. Conclusion: Future work includes improved memory representation and fusion, more robust subtask termination criteria, and pretraining for semantic understanding and generalization.These directions are presented as ways to address current design challenges.
A. RMBench Tasks Description
Table 4 presents the task descriptions for the RMBench benchmark.
- Table 4 provides task descriptions for RMBench.
- The table organizes descriptions around RMBench benchmark tasks.
- RMBench task descriptions are presented in tabular form.
B. Training Details
Mem-0 uses separate Planning and Execution Modules with specialized training procedures, hardware, precision, and memory-sequence handling.
- Planning Module: The Planning Module fine-tunes Qwen3-VL-8B-Instruct with LoRA for reasoning over key memories.Training uses LLaMAFactory on 8 NVIDIA A800 GPUs, and single-task training takes approximately half an hour.
- Execution Module: The Execution Module trains from scratch separately for each task over 30K iterations with global batch size 448.Training uses 8 NVIDIA A800 GPUs and takes approximately 18 hours per task.
- Forward Pass Strategy: VLM token generation and DiT action-chunk generation run in parallel, while memory fusion is processed serially to preserve episode order.A global data structure stores episode memory information for cross-batch token utilization.
- Dataloader Implementation: The custom dataloader distributes episodes across GPUs and independently manages resets.Frames can become temporally desynchronized across a global batch as training progresses.
- Training Configuration: Training uses grouped learning rates, cosine scheduling with linear warm-up, and mixed bfloat16 and float32 precision.Images are resized to 224 × 224 with mild frame-independent ColorJitter augmentation.
C. Additional Visualizations and Analysis of Failure Cases in Mem-0
The additional visualizations document representative Mem-0 failures across several tasks, while the authors note that the architecture remains open to refinement.
- Mem-0 exhibits suboptimal performance despite substantial improvements over baselines.The authors present representative failure cases to inform future research.
- Swap Blocks: Swap Blocks failures include premature termination after one or two subtasks and delayed termination that starts a redundant subtask.
- Observe and Pick Up: Observe and Pick Up failures involve confusion among objects with similar colors, shapes, or identical morphologies.Another failure is inaccurate target identification, causing grasps at mean or unintended positions.
- Rearrange Blocks: Rearrange Blocks can fail when Mem-0 presses the button redundantly.
C.1. Failures Analysis for M(1) Tasks
Mem-0’s M(1) failures expose challenges in target identification, sequence termination, and button pressing, while ablations clarify the roles of Anchor and Sliding Memory.
- Observe and Pick Up; Swap Blocks: Mem-0 misidentifies targets in Observe and Pick Up and mistimes termination in Swap Blocks.These errors can produce unintended grasps or confirmation-button presses at inappropriate times.
- Anchor Memory: Anchor Memory continuously influences the task horizon, requiring attention and adaptive weighting during action prediction.
- Anchor Memory: Anchor Memory yields substantial ablation gains, particularly for Rearrange Blocks and Put Back Block.
- Sliding Memory: Omitting Sliding Memory significantly degrades Rearrange Blocks performance and increases redundant button presses.Video analysis identifies redundant pressing as a primary failure mode without Sliding Memory.
- Future Improvements: Richer fusion mechanisms and stronger visual processing are proposed as potential directions for improving memory utilization and target identification.The proposed directions address both Anchor and Sliding Memory integration and Observe and Pick Up performance.
C.2. Failures Analysis for M(n) Tasks
M(n) tasks remain limited by classifier robustness and fine-grained visual or manipulation challenges, despite Mem-0’s substantial gains. Failures arise from incorrect subtask transitions, button-press detection, and imprecise battery manipulation.
- Mem-0 still has significant room for improvement on M(n) tasks, with classifier performance and robustness identified as the primary challenge.
- Cover Blocks: The Classifier can misidentify subtask progress in Cover Blocks, preventing transitions and causing the policy to operate under an incorrect task context.
- Blocks Ranking Try: Button-pressing interferes with hybrid-task coordination in Blocks Ranking Try, where delayed transition leaves the dual arms following conflicting instructions.
- Blocks Ranking Try: A single execution error inevitably causes overall failure in Blocks Ranking Try, making the task especially sensitive to coordination mistakes.
- Press Button: In Press Button, subtle visual differences between pressed and unpressed states limit VLM token granularity, impairing reliable termination detection.
- Battery Try: Battery Try remains constrained by imprecise insertion and grasp strategies caused by subtle visual cues from the battery slot.
- The authors propose proprioceptive or tactile feedback and improved VLM token extraction and memory fusion as avenues for strengthening the Classifier.
- Mem-0 nevertheless shows substantial gains across various M(n) tasks, notably on Cover Blocks relative to the baseline.