RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation

• RoboCerebra combines an extended-horizon manipulation dataset, a System 2–System 1 coordination framework, and an evaluation protocol for System 2 performance.The benchmark targets long-horizon planning and high-level reasoning in robotic manipulation.Source: §1 · p. 2
• Simulation supports scalable, reproducible long-horizon evaluation, with the sim-to-real gap considered less critical because the focus is high-level reasoning.Source: §1 · p. 2
• GPT generates context-conditioned task instructions and subtask sequences, which human operators execute in simulation with dynamic object variations.The resulting dataset has 6× longer action sequences and denser subtask annotations than prior datasets.Source: §1 · p. 3
• The Hierarchical Planning and Execution framework pairs a low-frequency VLM planner and memory bank with a high-frequency VLA controller for fine-grained actions.At inference, the VLM updates the plan and memory while the VLA executes actions.Source: §1 · p. 3
• The evaluation fixes a subtask-level VLA as System 1 and assesses System 2 planning, reflection on task completion, and long-term memory through their interaction.The protocol evaluates GPT-4o, Qwen2.5-VL [21], and LLaVA-Next-Video as System 2 modules.Source: §1 · p. 3

Robotic Manipulation Benchmarks

• Recent compositional benchmarks extend earlier single-step tasks, but typically remain below 500 action steps and lack dynamic variations or memory requirements.Source: §2 · p. 3
• RoboCerebra extends action-sequence lengths and adds memory-dependent execution and dynamic scene changes to evaluate long-horizon reasoning.Source: §2 · p. 3

Vision-Language-Action (VLA) Models

• VLA models perform strongly on short-horizon tasks, but existing approaches struggle to maintain coherence across extended temporal contexts; this work uses OpenVLA as its low-level controller.Prior diffusion-based approaches address discrete action representations with continuous action spaces, improving expressiveness and control fidelity.Source: §2 · p. 3

3.1 Task setting

• The task setting covers six subtask types spanning static ideal tasks, memory, disturbances, observation mismatch, and combinations requiring continual re-planning under uncertainty.The design targets extended temporal dependencies, partial observability, and dynamically changing environments.Source: §3.1 · p. 4

3.2 Dataset construction pipeline

• A modular pipeline generates GPT task plans, instantiates and verifies executable scenes, then collects human demonstrations with fine-grained temporal labels.Scene verification combines symbolic checks with GPT-4o assessment of multi-view RGB-D renderings.Source: §3.2 · pp. 4–5

3.3 Data Analysis

• 1,000 human-annotated trajectories span 100 task variants, averaging 9.1 atomic steps and 2,972.4 simulation steps per trajectory—about 6× longer than existing long-horizon datasets.Tasks range from 2 to over 20 atomic steps; over 10% involve five or more action types.Source: §3.3 · pp. 5–6

3.4 Multi-dimensional Evaluation

• The evaluation protocol uses four complementary metrics: task success, planning accuracy, planning efficiency, and action completion accuracy.Planning efficiency divides success rate by average plan length.Source: §3.4 · p. 6

4 Hierarchical Planning and Execution Framework

• A low-frequency VLM generates and updates step-level subgoals while a high-frequency VLA executes fine-grained actions, coordinated through a shared memory bank.The VLM monitors recent observations and updates memory when it detects subgoal completion or deviation.Source: §4 · p. 7, §4.2 · p. 7

4.1 Training Procedure

• Two-stage supervised fine-tuning trains the VLA on image-instruction/action pairs and the VLM on video-instruction pairs labeled for task progress.The VLA uses discretized action tokens and next-token prediction; the VLM receives successful and incomplete executions with contrastive supervision.Source: §4.1 · p. 7

4.2 Hierarchical Task Planning and Execution

• At inference, the VLM periodically monitors execution and updates the stored subgoal or issues a refined instruction, while the VLA acts from high-frequency visual observations.The arrangement preserves temporal abstraction alongside reactive control.Source: §4.2 · p. 7

5.2 Main Results

• In Mix, where long-term memory and adaptive planning are essential, HPE achieves a 13.21% success rate versus Planner+OpenVLA’s 11.48%; in Ideal, HPE is 0.82% lower.Table 3 reports average success rates over 10 rollouts for each method across six sub-tasks.Source: §5.2 · p. 8, Table 3 · p. 8
• In Ideal, OpenVLA achieves 4.05% success and fine-tuned OpenVLA 7.84%, both below HPE’s 21.10%; fine-tuned OpenVLA scores 0.00% in Mix.The authors attribute the results to difficulty maintaining instruction fidelity across long temporal spans and handling temporal dependencies under partial observability.Source: Table 3 · p. 8, §5.2 · p. 8

5.1 Experimental Settings

• The evaluation compares pretrained, blind, and fine-tuned System 2 planners, each directing a fixed System 1 policy; OpenVLA is fine-tuned on 100 task instances for 200K steps.Methods are evaluated over 600 rollouts, with anchor points aligning System 1 transitions between subgoals.Source: §5.1 · pp. 7–8
• The baselines contrast one-shot subgoal planning with a hierarchical framework that monitors the environment, updates subgoals, and uses memory for long-horizon planning.The shared System 1 controller is intended to make task success primarily reflect System 2 reasoning.Source: §5.1 · p. 8

5.3 Ablation on Planner

• GPT-4o averages 16.04% success in the planner ablation, exceeding other VLMs by more than 4%, but remains over 9% below GT-plan.The authors suggest limited environmental interaction and a visual-understanding domain gap may contribute to the remaining performance gap.Source: Table 4 · p. 9, §5.3 · p. 9

5.4 Evaluation on System 2

• GPT-4o leads planning accuracy, task success, and plan efficiency, while fine-tuning improves Qwen2.5-VL’s reflection performance measured by AccC.GPT-4o still exceeds Qwen2.5-VL-7B-SFT in task success by more than 6.5%, despite relatively poor simulator observation performance.Source: Table 5 · p. 9, §5.4 · p. 9

6 Conclusion

• RoboCerebra targets System 2 evaluation in long-horizon manipulation, using top-down task generation, hierarchical planning, and evaluation of planning, reflection, and memory.Experiments with state-of-the-art VLMs highlight varying reasoning capabilities.Source: §6 · p. 9
• The framework is an initial plan-memory-based exploration, with limited System 1–System 2 interaction and evaluation that could add subtask-ordering and failure-recovery signals.Real-world deployment would add challenges and further validate long-horizon reasoning under realistic conditions.Source: §6 · p. 10
• The authors identify richer bidirectional communication as future work, including finer-grained, interpretable feedback.Source: §6 · p. 10

A.1 Detailed Task Suite

• Dynamic Scene tasks include Observation Mismatching and Random Disturbance, testing responses to inconsistencies, unexpected changes, and displaced objects.Examples include recovering from collisions or updating plans after items move mid-task.Source: §A.1 · p. 13
• Memorization Scene includes Memory Exploration, where VLMs probe the environment, and Memory Execution, where perceptual cues are removed and the task relies on memory.Exploration can include checking cabinet contents.Source: §A.1 · p. 13
• Ideal Scene is a static, fully observable control condition without memory or disturbance constraints, providing a baseline for comparison.Source: §A.1 · p. 13
• Scenarios appear as image sequences with numbered action steps, annotated time segments, and descriptions of System 2 behavior; compositions assess memory, temporal reasoning, causal inference, and goal-directed manipulation.Source: §A.1 · p. 14

A.2 Study on Different VLA models

• The VLM shows comparable performance across both controller configurations on all evaluated metrics, with slight variations in Execution and Exploration.The authors interpret the stable overall trend as evidence that VLM reasoning and decision-making transfer across these embodied control implementations.Source: Table 6 · p. 14, §A.2 · p. 14
• The results indicate that the VLM’s higher-level cognitive and planning abilities are independent of the specific VLA architecture tested.Source: Table 6 · p. 14, §A.2 · p. 14
• The comparison uses OpenVLA and π0-fast as alternative System 1 modules beneath the VLM’s System 2 reasoning.Source: §A.2 · p. 14

A.3 Study on Different Planners

• Embodied VLMs generally outperform general-purpose VLMs on Embodied-QA.The comparison covers RoboBrain-2.0, Cosmos-Reason1, and VeBrain.Source: Table 7 · p. 14, §A.3 · p. 14
• Specialized models show superior adaptability in Memory Exploration and Mix, scenarios requiring long-horizon reasoning and dynamic decision-making.Source: §A.3 · p. 14
• Specialization has not fully transferred to Embodied Planning, where coherent planning sequences and fine-grained action decomposition remain limited.The authors identify deeper integration of perception, reasoning, and long-term planning as a future research direction.Source: §A.3 · p. 15

A.4 Results on Memory Tasks

• Memory-task evaluation includes overall success and intermediate measures of exploration success, plan efficiency, and decision accuracy.Exploration efficiency combines plan correctness and conciseness, normalizing completeness by predicted-plan length and averaging across tasks.Source: §A.4 · p. 15, Table 8 · p. 15

A.5 Prompt Details of Task Generation

• Task-generation prompts successively decompose goals, verify steps, generate executable primitives and affordances, then check output format.Source: §A.5 · pp. 15–16

A.6 Prompt Details of Hierarchical Framework

• The hierarchical prompting process generates plans from visual and task inputs, infers memory-dependent subgoals, and updates plans using new perceptions and memory.Source: §A.6 · p. 17

A.7 Case Study on Planning

• GPT produces detailed, coherent plans across the illustrated tasks, while Qwen omits context and transitions and LLaVA adds incorrect or irrelevant steps.Source: §A.7 · p. 18

A.8 Case Study on Sub-Plan

• For retrieving butter from a multi-compartment cabinet, GPT explores compartments and includes fallback steps, whereas Qwen assumes the first compartment contains it.Source: §A.8 · pp. 18–19

A.9 Case Study on Memory Tasks

• In memory exploration and execution examples, GPT updates task completion correctly, while Qwen misses fulfillment and fails to treat an empty cabinet as sufficient evidence.Source: §A.9 · pp. 19–20