RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

• History-dependent manipulation requires retaining and reusing information across time because immediate perception alone cannot support tasks involving accumulation, occlusion, or demonstrations.Examples include returning books to original positions, performing a specified number of cleaning passes, and folding laundry after observing a human demonstration.Source: §1 · p. 1
• Existing memory methods and benchmarks use inconsistent protocols and limited task coverage, preventing clear conclusions about generalization across memory designs.MemoryBench [17] covers only three near-solved spatial-memory tasks, while MIKASA-Robo [11] is short-horizon and lacks sufficient high-quality demonstrations for VLA imitation learning.Source: §1 · pp. 1–2
• RoboMME organizes memory evaluation into temporal, spatial, object, and procedural dimensions across 16 long-horizon tasks, 1,600 demonstrations, and 770k high-quality timesteps.The four task suites are Counting, Permanence, Reference, and Imitation, each containing four tasks.Source: §1 · p. 2
• The study develops 14 π0.5-based memory-augmented VLA variants spanning symbolic, perceptual, and recurrent representations integrated as context, modulator, or expert.The memory-as-modulator design conditions the action expert through adaptive LayerNorm [36], while memory-as-expert uses block-wise causal attention.Source: §1 · p. 2
• No memory design performs consistently across tasks: symbolic memory excels at counting and short-horizon reasoning, perceptual memory supports time-sensitive and motion-centric behaviors, and perceptual modulation offers the best performance-efficiency balance.These findings indicate that conclusions from prior work may not generalize across tasks.Source: §1 · p. 2

Memory-based Robotic Manipulation Benchmarks

• Most manipulation benchmarks [29] [31] [34] [53] rarely require temporal memory, while MemoryBench [17] and MIKASA-Robo [11] remain simple, short-horizon, or demonstration-limited.Current-observation policies can often succeed through local perception [16] [48], and recent memory policies [38] [41] are evaluated on narrow self-designed tasks.Source: §2 · p. 2

Benchmark and Model Comparison

• RoboMME addresses fragmented evaluation with a unified, challenging testbed spanning diverse memory types, while its same-backbone VLA family enables controlled comparison of representations and integration strategies.The benchmark provides explicit history dependence and sufficient demonstrations for large-scale imitation-based training and evaluation, addressing limitations in prior work.Source: §2 · pp. 2–3

Memory-based Robotic Manipulation Models

• Memory-based robotic models use symbolic representations, perceptual feature histories, or recurrent latent states, with distinct tradeoffs in optimization and long-horizon modeling.Symbolic methods include HistRISE [9], UniVLA, MemER [41], Gemini-Robotics-1.5 [43], and MEM [45]; perceptual and recurrent examples include ContextVLA [22], BC-RNN [33], MITL [52], and RoboMamba [32].Source: §2 · p. 3

3. RoboMME Benchmark

• RoboMME targets history-dependent manipulation by requiring models to reason over past observations that are no longer visible and disambiguate identical current observations.Memory is essential because different histories can lead to the same observation but require different actions.Source: §3 · p. 3

3.1. Cognitively-Motivated Task Design

• The benchmark defines temporal, spatial, object, and procedural memory, corresponding respectively to when, where, what, and how.These types cover event counts and ordering, object locations, persistent identity, and reproduction of demonstrated motion patterns.Source: §3.1 · p. 3
• Its four suites evaluate distinct memory demands: Counting tests temporal repetition, Permanence spatial tracking, Reference identity resolution, and Imitation procedural reproduction.The suites use tasks involving pick-and-place, occlusion, referential cues, and demonstrated pushing, inserting, or sequential motions.Source: §3.1 · p. 3

3.2. Benchmark Construction

• RoboMME comprises 16 tasks with 100 episodes each, yielding 1,600 demonstrations and 770k timesteps across episodes lasting hundreds to over one thousand steps.Tasks are stratified by scene clutter, horizon length, and environmental dynamics, with controlled waypoint perturbations supporting behavioral diversity and failure recovery.Source: §3.2 · pp. 4–5
• Compared with prior benchmarks [19] [34] [50], RoboMME explicitly enforces four memory types and adds greater environmental complexity and video-conditioned tasks.Prior benchmarks generally emphasize long-horizon planning and task complexity but can often be solved from immediate observations alone.Source: Table 2 · p. 5, §3.2 · p. 5

4. Memory-Augmented Manipulation Policies

• MME-VLA systematically compares memory representations and integration mechanisms on the π0.5 backbone under controlled settings.The suite is introduced as a family of memory-augmented VLA models, with detailed formulations provided in Appendix A.Source: §4 · p. 5

4.1. Memory Representations

• Three memory representations are studied: symbolic language subgoals, perceptual visual tokens, and recurrent fixed-size latent states.Symbolic memory uses discrete language tokens, while perceptual and recurrent memory provide differentiable neural features.Source: §4.1 · pp. 5–6
• Symbolic memory generates interpretable subgoals from the current image and previous subgoals, with grounded variants adding image coordinates for spatial reasoning.The formulation uses an auxiliary VLM and typically requires additional subgoal annotations for training.Source: §4.1 · p. 5
• Perceptual memory selects visual tokens from past images through token dropping or uniform frame sampling, trading local visual changes against global temporal coverage.Visual tokens preserve object appearance, spatial layout, and motion cues, and are sufficient without explicitly adding proprioceptive states.Source: Figure 2 · p. 6, §4.1 · p. 6

4.2. Memory Integration Mechanisms

• The suite evaluates Memory-as-Context, Memory-as-Modulator, and Memory-as-Expert integration strategies for connecting historical information to action prediction.Context concatenates memory with inputs; modulation uses cross-attention and AdaLN in the action expert; the expert pathway uses asymmetric blockwise causal attention to limit interference with the VLM.Source: Figure 2 · p. 6, §4.2 · pp. 6–7

4.1. Memory Representations

• Recurrent memory uses TTT fast weights or recurrent memory transformers to compress visual histories into bounded-size latent states for long-horizon action prediction.Unlike perceptual memory, its representation size does not grow with episode length.Source: §4.1 · p. 6

5.1. Experiment Setup

• The experiments evaluate 14 MME-VLA policies and four prior methods under a 512-token memory budget across 16 tasks, 800 episodes, and nine runs per MME-VLA variant.Models use 20-step action chunks during training, execute 16 steps during evaluation, and are trained for 80k steps; recurrent variants use smaller batches.Source: §5.1 · pp. 7–8

5.2. Main Results and Analysis

• 44.51% overall success: FrameSamp+Modul achieves the highest rate among MME-VLA variants, with perceptual methods generally outperforming symbolic and recurrent counterparts.TokenDrop is less effective than FrameSamp, likely because aggressive token pruning removes global spatial context; memory-as-modulator performs best for both perceptual methods.Source: Table 3 · p. 8, §5.2 · p. 9
• No single memory representation or integration strategy dominates all tasks: symbolic memory excels at counting and grounding, whereas perceptual memory supports time-sensitive behavior and motion imitation.Symbolic memory performs well on Counting tasks such as BinFill and PickXTimes, while perceptual memory excels on Imitation tasks including PatternLock and RouteStick.Source: §5.2 · pp. 9–10
• GroundSG reaches up to 32.70% success and usually outperforms SimpleSG with QwenVL, but grounding errors can make SimpleSG better on simpler tasks such as PickXTimes.Recurrent methods perform worst, likely because shallow recurrent layers destabilize fine-tuning; prompt-only Gemini also suffers from domain shift and incorrect labels.Source: §5.2 · p. 9
• MemER achieves 42.38% success and is the strongest compared method, while SAM2Act+ performs poorly on average but leads on ButtonUnmaskSwap.MemER benefits from combining stored keyframes for long visual histories with symbolic subgoals; SAM2Act+ benefits from reliable planner-executed discrete actions.Source: §5.2 · p. 9
• GroundSG+Oracle confirms that language supports high-level reasoning, yet performance degrades when precise visuomotor control or clutter handling is the primary bottleneck.These limitations appear in StopCube, InsertPeg, BinFill, and PickHighlight, where policies may manipulate incorrect objects or cause collisions despite clear subgoals.Source: §5.2 · p. 9
• Humans achieve 90.5% overall success but still fail on long-horizon PatternLock and time-sensitive StopCube, underscoring RoboMME’s difficulty.The human study reformulates tasks as incremental VideoQA with oracle low-level control.Source: §5.2 · p. 9
• Real-world evaluation reproduces the simulated pattern: symbolic memory performs best on counting in PutFruits, perceptual memory excels on motion-centric DrawPattern, and the remaining tasks are comparable.The transfer study uses four physical tasks and 350 demonstrations, evaluating GroundSG+QwenVL and FrameSamp+Modul.Source: Table 4 · p. 10, §5.2 · pp. 10–11

6. Conclusion and Future Work

• RoboMME evaluates memory-augmented robotic manipulation across temporal, spatial, object, and procedural memory, alongside controlled comparisons of symbolic, perceptual, and recurrent representations with multiple integration mechanisms.The benchmark is paired with a family of VLA models for systematic memory evaluation.Source: §6 · p. 11
• Symbolic memory excels at counting and short-horizon reasoning, whereas perceptual memory is crucial for motion-centric and time-sensitive behaviors, showing that no single design consistently dominates.The conclusion characterizes memory representations as task-dependent rather than universally superior.Source: §6 · p. 11
• RoboMME focuses on tabletop manipulation with fixed assets and mainly a single pretrained π0.5 backbone, leaving mobile manipulation, memory-bank methods, and additional VLA backbones for future study.The conclusion also motivates unified frameworks that combine complementary memory forms and positions RoboMME as a testbed for reliable memory-augmented robotic generalist agents.Source: §6 · p. 11

Appendix Outline

• The appendix details the MME-VLA Suite’s basic π0.5 architecture, memory representations, and memory integration mechanisms.These materials are organized under model architectures in the MME-VLA Suite.Source: Appendix Outline · p. 16
• The appendix documents RoboMME implementation details, including VLM subgoal prediction, prompts, training for evaluated models, the oracle planner, and the human study.Covered models include MME-VLA, SAM2Act+, OpenVLA-OFT, MemoryVLA, and Diffusion Policy.Source: Appendix Outline · p. 16
• The appendix provides full RoboMME results by oracle-planner setting, task suite, functional requirements, all tasks, and memory-token contribution, plus real-robot experiment details.It also includes complete descriptions of the benchmark’s 16 tasks across the Permanence, Reference, and Imitation suites.Source: Appendix Outline · p. 16

A. Model Architectures in the MME-VLA Suite

• The suite instantiates memory-augmented VLA policies from a shared π0.5 backbone, varying memory representations and their integration into the policy.The architecture description covers the base policy, memory representations, and integration mechanisms.Source: A. Model Architectures in the MME-VLA… · p. 17

A.1. Basic 𝜋0.5 Architecture

• The basic π0.5 architecture uses separate VLM and action transformer experts, with the VLM extracting joint visual-language features and the action expert predicting actions conditioned on those features.Both experts use standard transformer layers with blockwise causal attention and MLPs; unlike the original π0.5, the VLM is used solely as a feature extractor.Source: §A.1 · p. 17

A.2. Memory Representations

• Memory is represented as an explicit state summarizing past observations and actions, with every memory model constrained to the same fixed budget B and interface.The suite considers symbolic, perceptual, and recurrent memory classes, padding representations when they contain fewer than B tokens.Source: §A.2 · p. 18

A.2.1. Symbolic Memory

• Symbolic memory compresses trajectories into tokenized language subgoals for compact semantic descriptions, whereas perceptual memory retains selected differentiable visual tokens from past observations.Perceptual memory includes frame sampling and token dropping; historical proprioceptive states were excluded because they showed no consistent performance improvement.Source: §A.2.1 · p. 19, §A.2.2 · p. 19

A.2.3. Recurrent Memory

• Recurrent memory compresses history online either into persistent token-space slots with RMT or into parameter-space fast weights updated by local gradient steps with TTT.Both recurrent variants produce memory tokens for integration with the π0.5 backbone, while TTT implicitly stores history in its updated weights.Source: §A.2.3 · p. 20

A.3. Memory Integration Mechanisms

• Memory enters the backbone as appended context, feature-wise modulation inside the action expert, or a dedicated memory expert whose features condition action attention.The memory-as-expert design preserves explicit memory tokens while separating memory processing from the VLM and action pathways.Source: §A.3.1 · p. 21, §A.3.2 · p. 21, §A.3.3 · p. 21

B.1. VLM Subgoal Prediction and Prompts

• VLM-based subgoal predictors are fine-tuned with Qwen3-VL-4B-Instruct and LoRA, using separate simple and grounded prompts, while MemER is modified to predict grounded subgoals.Grounded subgoals consistently perform better in the reported experiments, and each VLM variant uses one prompt for multi-task training.Source: §B.1 · p. 22, Figure 6 · p. 24, Figure 7 · p. 25

B.2.1. Parameters for different memory representations.

• Implementations use B=512 memory tokens and generally one front-view camera; FrameSamp samples 32 pooled frames, while TokenDrop selects spatial tokens using K=8 and a 10^-4 RGB-difference threshold.Neural memories project SigLIP2 features from 2048 to the 1024-dimensional π0.5 width; recurrent variants process up to 64 frames and add under 10M parameters.Source: §B.2.1 · p. 26

B.2.3. Training Configuration

• All MME-VLA variants follow a largely unified recipe based on the released π0.5 LIBERO configuration, excluding proprioceptive inputs and freezing SigLIP2 while caching visual tokens as in [44].Training updates only the VLM expert, action expert, and parameters introduced by memory.Source: §B.2.3 · p. 27
• Models train for 80K steps, using batch size 64 for symbolic and perceptual memory and 16 for recurrent memory, with RMT variants requiring 2×H100 GPUs.Most models use 4×A40 GPUs; approximate training times range from 2–3 days for RMT models to 5–6 days for TTT-based recurrent models.Source: §B.2.3 · p. 27, Table 6 · p. 27
• The MME-VLA code is publicly available at the RoboMME GitHub repository.The repository is https://github.com/RoboMME/robomme_policy_learning.Source: §B.2.3 · p. 27

Training Pipeline

• SAM2Act+ pretrains the SAM2 backbone without temporal connections and then fine-tunes its memory attention module, with each stage trained for 40k steps on four NVIDIA A40 GPUs for approximately 2.5 days.The detailed training configurations and hyperparameters are provided in Table 7.Source: §B.3 · p. 28, Table 7 · p. 28

Discrete Actions

• Discrete waypoint actions are sufficient for most tasks, but performance degrades on precision-demanding tasks such as StopCube and InsertPeg because the action parameterization cannot provide precise continuous control.After waypoint prediction, the ManiSkill simulator uses an internal motion planner to execute the action.Source: §B.3 · p. 28

Video-based Observation Input

• For video-conditioned tasks, the demonstration video is rolled out through SAM2Act+ before interaction to prefill its memory bank, after which execution uses the current observation.This adapts the architecture despite its lack of a dedicated demonstration-video conditioning mechanism and follows the protocol adopted in [17].Source: §B.3 · p. 28

B.4. Training Details for OpenVLA-OFT

• OpenVLA-OFT evaluates memory-as-context because memory-as-expert is unsupported and memory-as-modulation can destabilize the shared backbone.Symbolic memory is appended as language tokens, while perceptual memory is projected into token embeddings and appended as context tokens; pilot modulation experiments produced loss spikes.Source: §B.4 · p. 29
• OpenVLA-OFT starts from OpenVLA-7B, uses LoRA on the LLaMA backbone, and fully trains the action head and memory projection layers.The evaluated variants use QwenVL subgoal predictors for symbolic memory and fused DINOv2-SigLIP features with FrameSamp or TokenDrop for perceptual memory.Source: §B.4 · p. 29

B.5. Training Details for MemoryVLA

• MemoryVLA is adapted to RoboMME while retaining its native CogACT-style diffusion policy, Prismatic VLM, DiT action head, and dedicated memory modules.The adaptation keeps LIBERO hyperparameters except for batch size 64 and 160k training steps, and uses the EEF action space.Source: §B.5 · p. 30

B.6. Training Details for Diffusion Policy

• The Diffusion Policy baseline is trained from scratch without explicit memory, conditioning on a short observation window of visual, robot-state, and frozen CLIP language features.It uses ResNet-18 and SpatialSoftmax visual encoders, a FiLM-modulated temporal U-Net with DDPM denoising, and min-max action normalization, unlike the quantile normalization used by other methods.Source: §B.6 · p. 31

B.7. Oracle Planner and Human Study

• Oracle evaluation estimates an approximate upper bound using ground-truth subgoals and an incremental VideoQA-style decision protocol followed by oracle low-level planning.The planner guarantees successful low-level motion whenever a valid high-level action is selected, such as picking a cube near a specified image coordinate.Source: §B.7 · p. 32
• The human-study protocol evaluates proprietary foundation models and 18 human participants across 800 episodes using the same VideoQA-style decision setting.Participants and models select high-level actions from a predefined set with grounding information while video clips are revealed incrementally.Source: §B.7 · p. 32

C.1. Results Using the Oracle Planner

• Prompting existing video foundation models yields limited performance, revealing difficulty translating visual understanding into reliable sequential decisions without task-specific adaptation or fine-tuning [41].The authors attribute this limitation to substantial domain shift between RoboMME’s environment and the models’ pretraining data, consistent with MemER [41].Source: §C.1 · p. 33
• Humans perform well on most oracle-planner tasks, but failures on demanding tasks demonstrate substantial long-term memory requirements.The reported oracle-planner results evaluate each episode with a single run, and Table 11 covers both foundation models and human participants.Source: §C.1 · p. 33, Table 11 · p. 33
• Permanence tasks cause human performance degradation because they require sustained attention over long horizons.This is identified as a prominent human failure mode in the oracle-planner evaluation.Source: §C.1 · p. 33
• DrawPattern and RouteStick show lower human success rates when participants forget earlier trajectory details or intermediate goals.The failures are attributed primarily to lost information from earlier in the trajectory or task procedure.Source: §C.1 · p. 33
• SwingXTimes and StopCube also yield low human performance because incremental video revelation makes participants lose track of completed actions.Participants often select several correct high-level actions before forgetting how many actions they have already taken.Source: §C.1 · p. 33

C.2. Results Based on Each Task Suite

• Table 12 compares all MME-VLA models with SAM2Act and MemER across task suites, using nine runs for MME-VLA and three for prior methods.The table marks the best result per section and the overall best among non-oracle models.Source: §C.2 · p. 34, Table 12 · p. 34

C.3. Results Based on Task Functional Requirements

• No single method fully solves an entire task suite, indicating that task demands do not map one-to-one onto cognitive memory types or model memory representations.The authors regroup tasks by functional characteristics and compare symbolic- and perceptual-memory MME-VLA variants with MemER using averaged category results in Table 14.Source: §C.3 · p. 35, Table 13 · p. 35, Table 14 · p. 36

Full results

• The U-Net-based Diffusion Policy and MemoryVLA [38] baselines each achieve below 10% overall success across the benchmark.Complete per-task results for MME-VLA models and additional baselines are reported in Table 15.Source: §C.4 · p. 37, Table 15 · p. 38
• OpenVLA-OFT variants reach 21.6%, 19.5%, 9.1%, and 9.6%, and are less effective than corresponding π0.5-based models in this setting.The reported variants are SimpleSG+QwenVL, GroundSG+QwenVL, TokenDrop+Context, and FrameSamp+Context, respectively.Source: §C.4 · p. 37, Table 15 · p. 38

Backbone analysis

• π0.5 gains more consistently from memory because it was co-trained on large-scale visual grounding data, whereas OpenVLA has greater difficulty using grounded symbolic information.Perceptual memory also provides limited gains for OpenVLA-OFT, while π0.5 shows more stable improvements across memory representations.Source: §C.4 · p. 37

Additional analyses

• Increasing the memory-token budget generally improves perceptual-memory performance, while attention allocation is measured across memory, observation, and language tokens.Attention statistics are reported as mean and maximum probability mass aggregated across layers.Source: Table 16 · p. 39, Table 17 · p. 39

Effect of memory budget

• Increasing memory tokens improves success rates: FrameSamp+Modul rises from 30.42% to 45.87%, while TokenDrop+Modul rises from 18.11% to 40.17% between 64 and 1024 tokens.Gains become smaller beyond 512 tokens, motivating 512 tokens as a performance–computational-cost trade-off; finer spatial tokens can further preserve visual detail.Source: §C.5 · p. 39

Attention allocation

• Memory receives less average attention than observation and language tokens, but stronger models assign higher peak memory or language attention at crucial history-dependent decisions.FrameSamp+Expert has the best success rate among these variants and the highest maximum memory attention, indicating sparse but influential memory use.Source: §C.5 · p. 39

Effect of memory modulation

• TokenDrop+Modul and FrameSamp+Modul produce non-trivial scale and bias signals, while FrameSamp+Modul combines higher success with lower modulation variance.This suggests that stable memory-conditioned feature modulation benefits action prediction, whereas excessive variation may perturb action features.Source: §C.5 · pp. 39–40, Table 18 · p. 40

Overall findings

• FrameSamp+Modul achieves the best overall performance, with memory improving MME-VLA through complementary capacity, sparse attention, and feature-modulation mechanisms.These analyses explain why perceptual memory, especially FrameSamp+Modul, performs best overall.Source: §C.5 · p. 40

Real Robot Experiment

• Four real-world tasks—PutFruits, TrackCube, RepickBlock, and DrawPattern—mirror BinFill, VideoUnmask(Swap), VideoRepick, and PatternLock, respectively.They test counting under intervention, tracking occluded and swapped objects, repeated block pickup, and trajectory reproduction.Source: D. Real Robot Experiment · p. 41
• The dataset contains 350 demonstrations and 78,400 timesteps, collected at 15 Hz with 50 PutFruits demonstrations and 100 for each other task.PutFruits receives fewer demonstrations because its episodes contain longer trajectories.Source: D. Real Robot Experiment · p. 41
• Experiments use a 7-DoF Franka Emika Panda with UMI fin-ray fingers and three RGB-D cameras in a tabletop environment, with grounded subgoals annotated except for PutFruits.Training compares FrameSamp+Modul, GroundSG + QwenVL, and the 𝜋0.5 baseline using memory budget B=2048 and 50k training steps.Source: D. Real Robot Experiment · p. 41
• Perceptual memory performs best on DrawPattern, symbolic memory performs better on PutFruits, both are moderate on RepickBlock, and perceptual memory is slightly better on TrackCube.Dynamic swapping often causes the VLM-based subgoal predictor to produce inaccurate grounding on TrackCube.Source: D. Real Robot Experiment · p. 41

E. Task Description

• RoboMME evaluates temporal, spatial, object, and procedural memory through four suites—Counting, Permanence, Reference, and Imitation—with four tasks per suite.Tasks use shared tabletop objects, including cubes, pegs, sticks, targets, buttons, containers, an aperture box, and an opaque bin.Source: E. Task Description · p. 43

E.1. BinFill

• BinFill requires placing exact color-specific cube counts into an opaque bin before pressing the termination button, under static or incrementally streaming cube availability.Instructions range from one specified color to three specified colors, and exceeding any required count causes immediate failure.Source: §E.1 · p. 44, Success Criteria · p. 44

E.2. PickXtimes

• PickXTimes tests temporal memory by requiring an exact number of pick-and-place repetitions on a colored cube before termination, with wrong-object, premature, and excessive actions failing immediately.A valid repetition lifts the cube above a height threshold with a valid grasp and places it below the specified placement height.Source: §E.2 · p. 45, Success Criteria · p. 45

E.3. SwingXtimes

• SwingXtimes requires transporting a specified cube between right- and left-side targets for an exact number of swing cycles, then placing it down and stopping.Success depends on reaching target centers within alignment tolerances, while wrong cubes, premature termination, or excessive swings fail the episode.Source: §E.3 · p. 46, Success Criteria · p. 46

E.4. StopCube

• StopCube tests temporal memory by requiring the robot to press the button precisely when an oscillating cube reaches its target on a specified ordinal visit.The robot must remain poised over the button and account for motion delay; pressing outside the target time window or on the wrong visit fails.Source: §E.4 · p. 47, Success Criteria · p. 47

E.5. VideoUnmask

• VideoUnmask requires inferring which containers hide specified cubes from a demonstration video and retrieving the correct containers in order without selecting an incorrect one.The execution phase follows a stationary video phase in which containers may include empty instances and each cube is hidden beneath one container.Source: §E.5 · p. 48, Success Criteria · p. 48

E.6. ButtonUnmask / E.8. ButtonUnmaskSwap

• ButtonUnmask and ButtonUnmaskSwap test active spatial memory: the robot first presses one or both buttons while cubes are masked, then tracks and retrieves the specified container(s) after hiding and, for Swap, positional changes.Picking before the button phase or selecting an incorrect container causes immediate failure, and multiple pickups must follow the instructed order.Source: §E.6 · p. 49, Task Characteristics · pp. 49, 51, Success Criteria · pp. 49, 51

E.9. PickHighlight

• PickHighlight requires remembering which cubes were highlighted during an earlier button press, picking all corresponding cubes, and pressing the button again to terminate.The highlighted regions are temporary white visual cues beneath objects, making the task history-dependent.Source: E. Task Description · p. 43, Figure 20 · p. 52, Table 27 · p. 52
• PickHighlight tests short-term object memory by requiring the robot to press a button, remember briefly highlighted cubes after the cues disappear, and pick every correct cube.Failure occurs if the robot picks before pressing the button or picks any non-highlighted cube.Source: §E.9 · p. 52, Task Characteristics · p. 52, Success Criteria · p. 52

Language Goals

• Across these tasks, success is defined by exact completion of the instructed object, action, temporal relation, order, or trajectory, while deviations generally terminate the episode immediately.The task suite combines video or transient visual cues with language instructions specifying sequential, temporal, ordinal, semantic, or spatial requirements.Source: §E.9 · p. 52, Language Goals · p. 53, §E.11 · p. 54

E.10. VideoRepick

• VideoRepick requires identifying the cube picked in a demonstration, tracking it after cube swaps, repeating the pick-and-place exactly N times, and stopping only afterward.Episodes contain video and execution phases; wrong-cube picks, premature stopping, or more than N repetitions cause immediate failure.Source: §E.10 · p. 53, Language Goals · p. 53, Success Criteria · p. 53

E.11. VideoPlaceButton / E.12. VideoPlaceOrder

• VideoPlaceButton and VideoPlaceOrder require grounding a cube to a target defined by video history: a before/after button-press relation or an ordinal placement position.Both tasks use video and execution phases, may swap target positions in Hard mode, and fail on wrong cubes or targets.Source: §E.11 · p. 54, Success Criteria · pp. 54–55, §E.12 · p. 55

E.13. MoveCube

• MoveCube tests whether the robot infers and reproduces one demonstrated manipulation primitive—hooking, pushing, or pick-and-place—after object and target positions are reset.Using a different primitive than the demonstrated method causes immediate failure.Source: §E.13 · p. 56, Success Criteria · p. 56

E.14. InsertPeg

• InsertPeg requires remembering the demonstrated peg, grasping its correct end, and inserting it from the demonstrated box side after the scene reset.Picking the wrong peg, grasping the wrong end, or inserting from the wrong side causes immediate failure.Source: §E.14 · p. 57, Success Criteria · p. 57

E.15. PatternLock

• PatternLock requires reproducing a complete demonstrated trajectory over an N×N target grid in the same order, with target and trajectory feedback during execution.Success requires visiting all targets in order; deviation terminates the episode.Source: §E.15 · p. 58, Success Criteria · p. 58

E.16. RouteStick

• RouteStick requires continuously reproducing a demonstrated obstacle-navigation path, including target order and clockwise or counterclockwise circling directions.Execution provides target highlighting and trajectory tracing; deviation from the demonstrated order causes immediate failure.Source: §E.16 · p. 59, Success Criteria · p. 59