PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning

Egor Cherepanov, Nikita Kachaev, Alexey K. Kovalev, Aleksandr I. Panov

arXiv:2502.10550v3cs.LGcs.AIcs.RO

TL;DR

Memory-enhanced agents lack a unified benchmark across diverse scenarios, particularly for tabletop manipulation. MIKASA combines a memory-task taxonomy with unified RL and robotic manipulation benchmarks. Across online, offline, and VLA evaluations, existing methods struggle as memory demands grow, motivating explicit long-horizon memory mechanisms.

  • Problem

    Memory evaluation lacks a universal benchmark across diverse scenarios, and tabletop robotic manipulation introduces memory demands that existing benchmarks do not fully capture.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    MIKASA combines a four-category memory-task taxonomy, MIKASA-Base, MIKASA-Robo, and accompanying offline datasets for memory evaluation.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    Experiments with online, offline, and VLA agents consistently show that existing methods struggle when memory requirements become substantial.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The results highlight the need for architectures with explicit and robust long-horizon memory mechanisms.

    Source: §8 · p. 10

    The benchmark focuses on spatio-temporal dependencies within single episodes and does not capture higher-level processes such as meta-RL.

    Source: §7 · p. 10

Abstract

from arXiv · show

Memory is crucial for enabling agents to tackle complex tasks with temporal and spatial dependencies. While many reinforcement learning (RL) algorithms incorporate memory, the field lacks a universal benchmark to assess an agent's memory capabilities across diverse scenarios. This gap is particularly evident in tabletop robotic manipulation, where memory is essential for solving tasks with partial observability and ensuring robust performance, yet no standardized benchmarks exist. To address this, we introduce MIKASA (Memory-Intensive Skills Assessment Suite for Agents), a comprehensive benchmark for memory RL, with three key contributions: (1) we propose a comprehensive classification framework for memory-intensive RL tasks, (2) we collect MIKASA-Base -- a unified benchmark that enables systematic evaluation of memory-enhanced agents across diverse scenarios, and (3) we develop MIKASA-Robo (pip install mikasa-robo-suite) -- a novel benchmark of 32 carefully designed memory-intensive tasks that assess memory capabilities in tabletop robotic manipulation. Our work introduces a unified framework to advance memory RL research, enabling more robust systems for real-world use. MIKASA is available at https://tinyurl.com/membenchrobots.

1 INTRODUCTION

The introduction argues that memory evaluation is fragmented and especially incomplete for robotic manipulation. MIKASA addresses this with a task classification, a unified RL benchmark, 32 robotic tasks, and expert datasets.

1 INTRODUCTION

  • Custom environments and narrow evaluation scopes can obscure agents’ varying strengths across memory tasks, while robotic tasks add occlusion, object recall, and multi-step dependencies.The introduction illustrates these demands with remembering a hidden plate and tracking whether a microwave door has been wiped.Source: §1 · p. 2
  • MIKASA classifies memory-intensive tasks into four categories to guide systematic evaluation and environment selection.Figure 1 presents the classification framework.Source: Figure 1 · p. 1, §1 · p. 2
  • MIKASA-Base is a Gymnasium-based framework for evaluating memory-enhanced RL agents.Source: §1 · p. 2
  • MIKASA-Robo comprises 32 robotic manipulation tasks targeting specific memory-dependent skills in realistic settings.The benchmark is open source and evaluated with online RL baselines and VLA models.Source: §1 · p. 2
  • Expert-quality trajectories for all 32 MIKASA-Robo tasks are released to support offline RL research.The paper also reports evaluations using offline RL baselines.Source: §1 · p. 2

2 RELATED WORKS

Prior memory benchmarks cover varied but often incomparable settings, and existing classifications do not fully address physical interaction in robotics. MIKASA-Robo exemplifies manipulation tasks spanning different memory demands.

2 RELATED WORKS

  • Existing benchmarks often focus on abstract puzzles or navigation and differ enough to make direct comparison difficult, while robotic manipulation adds physical interaction and multi-step demands.Source: §2 · p. 3
  • Earlier classifications describe temporal horizons, memory dependencies, agent memory types, or cognitive tasks, but leave robotics’ physical dimensions underexplored.This gap motivates a framework for spatio-temporal memory in real-world tasks.Source: §2 · p. 3
  • MemoryBench covers three manipulation tasks focused on spatial memory and is based on RLBench, which the authors say does not allow efficient training parallelization.Source: §2 · p. 3
  • MIKASA-Robo’s examples span recalling a hidden ball location, remembering a cube’s color, and tracking a peg’s previous rotations.Source: Figure 2 · p. 4

3 BACKGROUND

The background frames memory evaluation around partial observability and delayed recall, then identifies robotic tabletop manipulation as a setting where existing MDP-oriented simulators omit spatio-temporal dependencies.

3.1 PARTIALLY OBSERVABLE MARKOV DECISION PROCESS

  • A POMDP models settings where agents observe noisy or incomplete information about the true state and choose actions based on observation history.The objective is to maximize expected discounted cumulative reward; partial observations make POMDPs harder to solve than MDPs.Source: §3.1 · p. 4

3.2 MEMORY-INTENSIVE ENVIRONMENTS

  • A memory-intensive task is a POMDP with correlation horizon ξ > 1, meaning decision-critical information must be recalled after at least one intervening timestep.Agents may address such environments by augmenting their policies with memory mechanisms.Source: §3.2 · p. 4

3.3 ROBOTIC TABLETOP MANIPULATION

  • Robotic tabletop manipulation involves actions such as grasping, pushing, and picking, but most simulators model these tasks as MDPs without real-world spatio-temporal dependencies.The paper says this limits development of memory-enhanced agents for practical applications.Source: §3.3 · p. 4

4 CLASSIFICATION OF MEMORY-INTENSIVE TASKS

The section proposes a balanced framework for classifying memory-intensive RL tasks, drawing on cognitive-science concepts while addressing fragmented evaluation. Its four categories organize distinct memory demands and support systematic environment selection.

4 CLASSIFICATION OF MEMORY-INTENSIVE TASKS

  • Existing evaluations use different, minimally overlapping environments and can test similar memory demands while neglecting others.Narrow evaluations may miss differences between architectures that perform well on object-property recall but struggle with sequential memory.Source: §4 · p. 4

4.1 MEMORY: FROM COGNITIVE SCIENCE TO RL

  • The classification draws on cognitive-science memory concepts to balance coverage of important agent memory needs with practical simplicity.The framework aims to cover memory capabilities relevant to real-world applications without requiring the full spectrum of human memory.Source: §4 · p. 5, §4.1 · p. 5, Figure 3 · p. 5

4.2 TAXONOMY OF MEMORY TASKS

  • The framework distinguishes object memory, spatial memory, sequential memory, and memory capacity as four types of memory-intensive RL tasks.They cover retaining object information, remembering locations, using temporally ordered information, and managing multiple pieces of information.Source: §4.2 · p. 5
  • The framework supports systematic evaluation across diverse scenarios and provides a basis for selecting benchmark tasks spanning varied memory requirements.The authors present a curated task set based on this classification as the basis of MIKASA.Source: §4.2 · pp. 5–6

5 MIKASA-BASE

MIKASA-Base organizes memory evaluation into diagnostic vector-based tasks and more complex image-based tasks. It unifies representative environments under a common API to support comparisons and attribution of memory-related failures.

5 MIKASA-BASE

  • MIKASA-Base progresses from vector-based diagnostic environments isolating memory mechanisms to image-based tasks that add realistic perception challenges.This tiered design supports incremental validation from atomic reasoning to high-dimensional sensory input.Source: §5 · p. 6
  • The benchmark selects representative environments spanning four memory-task types and unifies them under a consistent API.The selection ranges from object permanence to sequential planning and supports architectural ablations, direct method comparisons, and simplified evaluation.Source: §5 · p. 6
  • MIKASA-Base standardizes memory-RL evaluation and its structured progression supports precise attribution of memory-related agent failures.It covers a broad spectrum of memory challenges in a common framework for reproducible research.Source: §5 · p. 6

6 MIKASA-ROBO

MIKASA-Robo targets memory-intensive robotic manipulation with fine-grained tasks that test memory under meaningful partial observability. Its evaluations span online, offline, and VLA agents, which struggle as memory demands increase.

6 MIKASA-ROBO

  • MIKASA-Robo addresses manipulation benchmarks that often assume full observability or use complex tasks that obscure specific memory mechanisms.Its fine-grained tasks are designed to evaluate diverse memory skills in robotic manipulation.Source: §6 · p. 7
  • Compared with concurrent MemoryBench, MIKASA-Robo covers four memory categories and 32 tasks rather than three tasks focused on spatial memory.The benchmark aims to enable broader, more systematic evaluation of memory mechanisms.Source: §6 · p. 7

6.1 MIKASA-ROBO MANIFESTATION

  • The 32 tasks span 12 categories and four memory types with varied difficulty; RGB+joints is the standard memory-test configuration, and rewards can be dense or sparse.Dense rewards provide progress feedback, whereas sparse rewards signal task completion; the benchmark also offers state, RGB, joints, oracle, and prompt modes.Source: §6.1 · pp. 7–8

6.3 OFFLINE RL BASELINES

  • No evaluated offline model solves most of the 32 tasks, and none solves tasks requiring high memory capacity or sequential memory.Training used datasets of 1,000 successful trajectories per task, generated with oracle-level environment information.Source: Figure 6 · p. 8, §6.3 · p. 9

6.2 ONLINE RL BASELINES

  • Under RGB+joints, PPO-LSTM outperforms PPO-MLP on RememberColor-v0 with three colors, but both approach zero at five or nine colors and fail at three with sparse rewards.The comparison uses dense rewards for the three-, five-, and nine-color variants.Source: §6.2 · p. 8

6.4 VLA BASELINES

  • Octo and OpenVLA show some gains over random on simpler tasks, but performance degrades with complexity; real-world π0.5 results also sharply decline under long occlusion.OpenVLA exceeds random on selected tasks at K=8, while K=4 runs for OpenVLA, SpatialVLA, and π0 are random-like; larger chunks offer only limited memory compensation.Source: §6.4 · p. 9

7 LIMITATIONS AND FUTURE WORK

The authors identify scope gaps in MIKASA and propose broader evaluations of memory phenomena and metrics. The current framework focuses on spatio-temporal dependencies within single episodes.

7 LIMITATIONS AND FUTURE WORK

  • The benchmark focuses on within-episode spatio-temporal dependencies, leaving meta-RL over task distributions outside its current scope.The authors also identify spurious correlations, interference, memory overwriting or forgetting, and richer evaluation metrics as directions for future work.Source: §7 · p. 10
  • The authors call for more extensive adaptation of large VLA models to clarify their memory capabilities in MIKASA.Source: §7 · p. 10
  • The current evaluation relies on success rates or returns; the authors suggest additional metrics for more precise assessment of memory mechanisms.Source: §7 · p. 10

8 CONCLUSION

The paper presents MIKASA as a unified suite for evaluating memory in RL, including a taxonomy, MIKASA-Base, MIKASA-Robo, and offline datasets. Evaluations across agent types find that substantial memory demands remain challenging.

8 CONCLUSION

  • MIKASA combines a taxonomy of four memory types, standardized MIKASA-Base tasks, 32 memory-focused robotic manipulation tasks, and accompanying offline datasets.Source: §8 · p. 10
  • Online, offline, and VLA agents struggle when memory requirements become substantial, and real-world robot experiments show a similar decline under long-horizon occlusion.The conclusion attributes this pattern primarily to memory rather than embodiment or perception noise.Source: §8 · p. 10
  • The authors intend MIKASA to guide and accelerate progress in memory-intensive RL for real-world applications.Source: §8 · p. 10

REPRODUCIBILITY STATEMENT

The paper describes reproducibility materials and a practical MIKASA-Robo workflow, including released code, protocols, datasets, and debugging wrappers.

REPRODUCIBILITY STATEMENT

  • MIKASA-Base and MIKASA-Robo tasks are released under MIT, with code, installation instructions, environment guides, experiment protocols, and implementation details provided or planned for supplementary materials.Processed offline datasets, collection scripts, and pretrained models are also described; the datasets are to be published after the rebuttal period.Source: REPRODUCIBILITY STATEMENT · p. 10

A MIKASA-ROBO IMPLEMENTATION DETAILS

  • The MIKASA-Robo example creates and runs an environment, while wrappers can overlay step, reward, and task-specific information to support debugging.Examples include displaying the target color in RememberColor and the ball’s actual location in ShellGame.Source: A MIKASA-ROBO IMPLEMENTATION DETAILS · p. 21

B MIKASA-ROBO TASKS CUSTOMIZATION

MIKASA-Robo exposes task parameters for controlled customization, allowing difficulty to scale while retaining the tasks’ memory structure. Its atomic settings isolate memory, with optional extensions adding longer delays, richer objects, clutter, or alternative robot embodiments.

B MIKASA-ROBO TASKS CUSTOMIZATION

  • Researchers can vary object counts, episode lengths, cue durations, and delays, while official configurations provide multiple difficulty levels and challenging memory variants.Examples include RememberColor configurations with different color counts and SeqOfColors configurations with adjustable sequence length and timing.Source: B MIKASA-ROBO TASKS CUSTOMIZATION · p. 22
  • The tasks isolate memory rather than long-horizon credit assignment: a single occlusion can make RememberColor non-Markovian, and memory-related parameters can extend its horizon and difficulty.The long-horizon example increases episode length and delay.Source: B MIKASA-ROBO TASKS CUSTOMIZATION · pp. 22–23
  • The same task logic and memory challenge can run on different robot embodiments; the embodiment changes kinematics and motion feasibility, not the task-induced memory load.The examples include Franka Panda and Unitree arms.Source: B MIKASA-ROBO TASKS CUSTOMIZATION · p. 23, Figure 7 · p. 24
  • Atomic tasks use simple objects to isolate memory, but can be lifted to realistic object distributions that add visual variability and manipulation demands while preserving the underlying memory requirements.RememberObject3 replaces colored cubes with diverse objects, increasing perceptual and control demands.Source: B MIKASA-ROBO TASKS CUSTOMIZATION · p. 23, Figure 8 · p. 24
  • Clutter is excluded from atomic tasks to separate memory evaluation from perception, planning, and control challenges, but can be added after a policy solves the base task.Clutter adds visual complexity and occlusions without changing the original task’s memory structure.Source: B MIKASA-ROBO TASKS CUSTOMIZATION · p. 23, Figure 9 · p. 25

C MIKASA-ROBO DATASETS FOR OFFLINE RL AND IMITATION LEARNING

The appendices describe datasets and training setups for offline RL and VLA baselines, alongside MIKASA-Base usage and a taxonomy of RL memory mechanisms.

C MIKASA-ROBO DATASETS FOR OFFLINE RL AND IMITATION LEARNING

  • For each of 32 MIKASA-Robo tasks, the offline dataset contains 1,000 successful trajectories collected by a PPO-MLP trained to 100% success in state mode with sparse rewards.Trajectories record images, joints, actions, dense rewards, sparse success flags, and done flags; datasets, agent weights, and collection scripts are available.Source: C MIKASA-ROBO DATASETS FOR OFFLINE RL… · pp. 25–26

D MIKASA-ROBO SETUP FOR VLA BASELINES

  • For a representative spatial- and object-memory subset, VLA datasets contain 250 oracle-policy episodes per task with synchronized camera frames, end-effector actions, and language instructions.The baselines are trained for 50,000 iterations and evaluated independently on each task.Source: D MIKASA-ROBO SETUP FOR VLA BASELINES · p. 26

E MIKASA-BASE IMPLEMENTATION DETAILS

  • MIKASA-Base supports the Gymnasium API, its wrappers, parallelization, predefined difficulty levels, and custom environment parameters.The example uses a MemoryLength environment with custom memory length and bit-count parameters.Source: E MIKASA-BASE IMPLEMENTATION DETAILS · p. 26

F MEMORY MECHANISMS IN RL

  • RL memory approaches include recurrent states, state-space models, transformer attention, graph networks, temporal convolutions, world models, and external read-only or read-write memory.Memory may also be implicit in action patterns or externalized through persistent environment traces; different works operationalize memory differently, contributing to evaluation heterogeneity.Source: F MEMORY MECHANISMS IN RL · p. 27

G CLASSIC BASELINES PERFORMANCE ON THE MIKASA-ROBO

The baseline experiments show that MIKASA-Robo tasks are solvable with full state information but challenge memory-based agents under partial observations, especially with sparse rewards. Additional offline comparisons find that DMamba and GTrXL do not match RATE on more demanding memory tasks.

G CLASSIC BASELINES PERFORMANCE ON THE MIKASA-ROBO

  • 25% success rate for PPO-MLP versus 100% for PPO-LSTM on RememberColor with three cubes; PPO-LSTM performance degrades as the task grows to five or nine cubes.The authors report that these results distinguish memory-less from memory-enhanced agents and show that increased task complexity challenges conventional memory mechanisms.Source: G CLASSIC BASELINES PERFORMANCE ON THE… · p. 28

H ADDITIONAL OFFLINE RL RESULTS WITH DENSE REWARDS

I EXPERIMENTS REPRODUCING AND COMPUTE RESOURCES

  • Offline experiments used one NVIDIA A100 GPU, three training runs per task, and 100 evaluation episodes per run, with results reported as mean ± SEM across run-level means.Evaluation used environment and agent random seeds from 1 to 100.Source: I EXPERIMENTS REPRODUCING AND COMPUTE… · p. 28

G CLASSIC BASELINES PERFORMANCE ON THE MIKASA-ROBO

J ADDITIONAL BASELINES: SSMS AND MEMORY-ENHANCED TRANSFORMERS

  • GTrXL improves over standard DT and BC baselines, but GTrXL and DMamba still trail RATE, especially on RememberColor5/9; the authors argue that increasing sequence complexity stresses continual retention and rewriting.The additional SSM/Transformer baseline comparison is presented in Table 7.Source: J ADDITIONAL BASELINES: SSMS AND… · pp. 29–30, Table 7 · p. 31

G CLASSIC BASELINES PERFORMANCE ON THE MIKASA-ROBO

K MIKASA-ROBO DETAILED TASKS DESCRIPTION

This section describes MIKASA-Robo’s 32 memory-intensive tabletop manipulation tasks, their memory demands, configurations, observation and reward structures, success criteria, and evaluation setup. It also covers related MIKASA-Base environments and customization guides, including tasks involving changing physical states, hidden positions, and colored-object sets or sequences.

K.1 SHELLGAME-V0

  • ShellGame tests whether an agent can remember a ball’s location after mugs occlude it, with Touch, Push, and Pick variants requiring progressively more demanding interactions.The task also offers sparse binary or dense rewards.Source: §K.1 · p. 33

K.2 REMEMBERCOLOR-V0

  • RememberColor-v0 tests remembering a briefly displayed cube’s color across a delay, then identifying and interacting with the matching cube among 3, 5, or 9 differently colored candidates.Source: §K.2 · p. 34

K.3 REMEMBERSHAPE-V0

  • RememberShape-v0 evaluates an agent’s ability to remember and identify objects by geometric properties: after a shape is displayed and disappears, it selects matching geometry from 3, 5, or 9 options.Source: §K.3 · p. 35

K.4 REMEMBERSHAPEANDCOLOR-V0

  • RememberShapeAndColor-v0 tests whether an agent can memorize an object's shape and color, then after a delay identify and interact with the matching object among 6 easy or 9 medium objects.Source: §K.4 · p. 36

K.5 INTERCEPT-V0

  • Intercept evaluates prediction of a moving ball’s trajectory and interception, targeting motion prediction and spatial-temporal reasoning for dynamic manipulation.Its environment includes a red ball moving across a table and a target zone.Source: §K.5 · p. 37

3. Guide the ball to reach a designated target zone

  • The task has slow, medium, and fast ball-velocity modes of 0.25–0.5, 0.5–0.75, and 0.75–1.0 m/s; success requires guiding the ball into the target zone and stopping within its 0.1 m-radius area.Source: §3 · p. 37

K.6 INTERCEPTGRAB-V0

  • InterceptGrab extends interception by requiring the agent to predict a moving object’s trajectory and grasp it while in motion, combining prediction with precise manipulation timing.The task simulates catching or intercepting moving objects; its environment uses a red ball moving across a table.Source: §K.6 · p. 38

5. Maintain a stable grasp while bringing the ball to rest

  • The moving-ball task requires grasping a ball traveling at 0.25–1.0 m/s and holding it until it rests while the robot remains static.It has slow, medium, and fast variants; success requires a stable grasp through rest.Source: §5 · p. 38

K.7 ROTATELENIENT-V0

  • RotateLenient-v0 tests whether the agent remembers how far a peg has rotated and turns it to a prompted target within ±0.1 radians, then holds it stable.The PosNeg variant uses target angles between −π/4 and π/4; the peg center may shift.Source: §K.7 · p. 39, §1 · p. 39

K.8 ROTATESTRICT-V0

  • RotateStrict-v0 adds positional control: the peg must reach its prompted angle within ±0.1 radians while staying within 5 cm of its initial XY position.Success also requires a stable, static final pose without significant deviation on other rotation axes.Source: §K.8 · p. 40

K.9 TAKEITBACK-V0

  • TakeItBack-v0 requires moving a cube to a goal, then returning it to its invisible initial region after the goal changes color, in that order.The agent must remember the starting position because it is not visible during the return phase.Source: §K.9 · p. 41

K.10 SEQOFCOLORS-V0

  • SeqOfColors-v0 tests memory by showing N colored cubes one at a time, then presenting a larger set and requiring the agent to identify and touch all previously shown cubes in any order.Source: §K.10 · p. 42, §2 · p. 42

K.11 BUNCHOFCOLORS-V0

  • BunchOfColors-v0 tests memory capacity by requiring an agent to remember multiple objects simultaneously; it presents colored cubes together and requires touching all previously shown cubes in any order.The hard level requires remembering 7 colors demonstrated simultaneously.Source: §K.11 · p. 43, §2 · p. 43

K.12 CHAINOFCOLORS-V0

  • ChainOfColors-v0 evaluates ordered-information storage and retrieval: the agent sees an ordered sequence of N colored cubes one at a time, then identifies and touches previously shown cubes in the demonstrated order.At hard difficulty, the agent remembers 7 colors demonstrated sequentially.Source: §K.12 · p. 44, §2 · p. 44

2. Delay Phase (steps (5N)-(5N + 4)): All cubes disappear

  • During the delay phase, all cubes disappear; the demonstrated set must be touched without mistakes, with sparse binary or dense multi-component rewards.Dense reward components include target distance, stable contact, correctly touched cubes, motion smoothness, and task completion.Source: §2 · p. 44

L MIKASA-BASE BENCHMARK TASKS DESCRIPTION

  • MIKASA-Base includes card matching, ordered tile sequences, delayed sequence-value prediction, and object-location memory tasks.Memory Cards hides five shuffled pairs; Numpad resets progress after an incorrect tile, while MemoryLength rewards predicting a specified earlier observation at the final step.Source: L MIKASA-BASE BENCHMARK TASKS… · p. 45, §L.1 · p. 45, §L.2 · p. 45

L.4 MINIGRID-MEMORY–L.7 VIZDOOM-TWO-COLORS

  • Other MIKASA-Base tasks test remembering a visual cue for maze navigation, identifying dancers from egocentric views, and using an initial goal cue at a T-maze junction.ViZDoom-Two-Colors has a harder variant where the beneficial-color column disappears after 45 steps; Minigrid-Memory rewards a correct junction choice with Rt = 1 − 0.9 × t/T.Source: §L.4 · p. 46, §L.5 · p. 46, §L.6 · p. 46

L.8 MEMORY MAZE–L.10 MEMORYGYM MYSTERY PATH

  • Memory Maze, Mortar Mayhem, and Mystery Path require recalling object locations, command sequences, or an invisible route, with failure or missteps affecting progress.Memory Maze rewards reaching the prompted target object; Mortar Mayhem gives +0.1 per correctly executed command, and Mystery Path gives +0.1 for each previously unvisited tile reached.Source: §L.8 · p. 46, §L.9 · p. 47, §L.10 · p. 47

L.11 POPGYM ENVIRONMENTS–L.11.4 POPGYM REPEAT PREVIOUS

  • POPGym includes sequence reconstruction, card matching, first-value recall, and retrieval of a value from a specified number of steps earlier.Autoencode requires reproducing a watched card sequence after the indicator disappears; Concentration requires remembering previously revealed cards to find pairs.Source: §L.11 · p. 47, §L.11.1 · p. 47, §L.11.2 · p. 47

L.11.5 POPGYM STATELESS CARTPOLE–L.11.10 POPGYM HIGHER LOWER

  • POPGym also tests inferring hidden position from velocity, filtering noisy observations, adapting exploration and exploitation, and estimating card ranks.The stateless and noisy Cartpole and Pendulum variants require position inference; Multiarmed Bandit resets reward probabilities each episode, while Higher Lower permits card-counting strategies.Source: §L.11.5 · p. 47, §L.11.6 · p. 48, §L.11.7 · p. 48

L.11.11 POPGYM COUNT RECALL–L.11.15 POPGYM LABYRINTH ESCAPE

  • Further POPGym tasks require counting past occurrences, tracking hits and misses or mine clues, and exploring restricted-view mazes or finding their exits.Battleship penalizes repeated shots; Mine Sweeper ends after a mine is clicked, while Labyrinth Escape rewards only reaching the exit.Source: §L.11.11 · p. 48, §L.11.12 · p. 48, §L.11.13 · p. 48

M MIKASA-ROBO CUSTOMIZATION GUIDES–M.7 BUNCHOFCOLORS

  • MIKASA-Robo customization guides expose task-specific controls for shell games, interception, rotation, object return, and remembering visual cues or sequences.The supplied examples configure ball speed and goal tolerance, rotation mode and angle threshold, goal-region and object size, cue counts and visibility timing, and sequence length and presentation timing.Source: M MIKASA-ROBO CUSTOMIZATION GUIDES · p. 49, §M.1 · p. 50, §M.2 · p. 51

M.8 SEQOFCOLORS

  • SeqOfColors uses 6 colors, a 15-cube sequence, 15 steps per cube, a 5-step empty-table interval, and a 100-second empty-table duration.Source: §M.8 · p. 53

M.9 CHAINOFCOLORS

  • ChainOfColors is described as chained sub-episodes with a controlled pace.Source: §M.9 · p. 53
  • The real-world setup uses a SO-101 arm (Knight et al., 2024), wrist-mounted and overhead RGB cameras, and three colored cubes for RememberColor3 tasks.The setup matches the sensing and embodiment used for fine-tuning and evaluation.Source: Figure 26 · p. 54
  • Real-world evaluation tests fine-tuned π0.5 on three tasks over 30 episodes per task, reporting correct-cube touches and successful picks by color and task.Source: Table 10 · p. 54

N REAL-WORLD EXPERIMENTS

Real-world experiments test whether a fine-tuned VLA can handle tabletop tasks with increasing memory demands. Performance degrades most sharply when the target is occluded and shuffled before the robot must identify it.

Experimental setup

  • Experiments used a fine-tuned pretrained π0.5 with wrist-mounted and overhead RGB cameras, trained on 300 demonstrations per task.Each task had 100 trajectories per target color; the model was fine-tuned for 100,000 gradient steps.Source: N REAL-WORLD EXPERIMENTS · pp. 54–55

Evaluation protocol

  • The three-stage protocol separates fully observed MDP competence, dynamic scene changes without occlusion, and the full occluded long-horizon memory requirement.Task 3 requires recalling the initially observed color after the cubes are permuted under occlusion.Source: N REAL-WORLD EXPERIMENTS · pp. 54, 56, Figure 29 · p. 57

Task 3: Real World RememberColor3-v0

  • 0.10 touch success and 0.10 pick success in Task 3, below Task 2’s 0.63 touch and 0.43 pick, as Task 1 reaches 100% touch success across colors.The Task 3 result is reported for the occluded, long-horizon setting.Source: N REAL-WORLD EXPERIMENTS · pp. 55–56

Discussion and Conclusion

  • The Task 3 collapse aligns with simulation results and supports the conclusion that current VLA models lack temporal memory for simple real-world memory tasks.The comparison across tasks shows degradation from fully observed manipulation to dynamic variation and then long-horizon memory.Source: N REAL-WORLD EXPERIMENTS · p. 56

Open-Source Datasets

  • The authors publicly release the real-world datasets, with 300 trajectories per task balanced across red, green, and blue cube episodes.Dataset splits vary for hosting convenience while preserving the color balance and total count.Source: N REAL-WORLD EXPERIMENTS · p. 56
Loading 2502.10550v3…