RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots

• Robot datasets remain limited in task diversity and coverage, while real-world evaluation is resource-intensive, time-consuming, and affected by experimental noise; existing simulation tools also cover relatively narrow tasks and environments.These limitations hinder robust policy training, reproducible method comparisons, and analysis of how task diversity, environment variation, and dataset scale affect generalization.Source: §1 · p. 2
• RoboCasa365 is a comprehensive household-robotics simulation benchmark built on the RoboCasa framework.It is designed for large-scale experiments and systematic analysis of factors affecting generalization.Source: §1 · p. 2
• The benchmark defines 365 tasks across 60 kitchen activities, spanning manipulation, semantic reasoning, long-horizon planning, and memory-dependent tasks.This task diversity supports evaluation across multiple dimensions of generalist robot capability.Source: §1 · p. 2
• It includes 2,500 unique kitchen scenes modeled from real U.S. kitchens, varying in layout, object configuration, and appearance.These scenes provide realistic contexts for a variety of everyday tasks.Source: §1 · p. 2
• RoboCasa365 provides over 2,000 hours of interaction data: 612 hours of human demonstrations and 1,615 hours of synthetic demonstrations generated using MimicGen (Mandlekar et al., 2023).Source: §1 · p. 2
• The benchmark supports massively multi-task training, foundation model training, and lifelong learning, with reproducible evaluation and analysis of generalization factors.Source: §1 · p. 2

Robot Simulation Frameworks

• The authors describe RoboCasa365 as the first simulation framework combining hundreds of tasks, thousands of room-scale scenes, large-scale demonstrations, and generalist-robot benchmarks.Its scope contrasts with frameworks focused on tabletop settings and prior room-scale simulations.Source: §2 · p. 3

Datasets and Benchmarks for Generalist Robots

• RoboCasa365 provides over 500k demonstrations across over 300 tasks and 2,500 scenes, versus Nasiriany et al. (2024)’s 100k demonstrations, 30 tasks, and 100 scenes.Li et al. (2023) offer substantial task and environment diversity but lack large-scale datasets for all tasks.Source: §2 · p. 3

Training Generalist Robots

• RoboCasa365 provides a model-agnostic benchmark suite for multi-task training, pretraining and target-data fine-tuning, and lifelong learning.This contrasts with prior simulation benchmarks’ narrow task and environment distributions and narrower focus on specific methods.Source: §2 · p. 3

3.1 EXPANDING THE SCOPE OF ASSETS

• RoboCasa365 expands object assets and kitchen fixtures, adding 57 object categories and scaling interactable appliances from 20 fixtures across 4 categories to 456 instances across 12 categories.New articulated appliances include toasters, toaster ovens, stand mixers, blenders, and electric kettles; fixture categories include 20–50 instances each.Source: §3.1 · pp. 3–4

3.2 DIVERSE KITCHEN SCENES

• The benchmark provides 2,500 pretraining kitchens from 50 layouts and 50 styles, plus 10 target kitchens, with target styles kept distinct from pretraining styles.Pretraining layouts are digital-cousin replicas of 50 listed U.S. homes; scenes are combinations of a layout and style.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 SUITE OF 365 EVERYDAY TASKS

• The suite contains 365 tasks—65 atomic and 300 composite—with 220 requiring mobile manipulation; LLM-generated blueprints organize composite tasks into 60 activities across 6 families.The 65 atomic tasks build on eight foundational skills. Composite tasks comprise 83 existing RoboCasa tasks and 217 new tasks, with blueprints specifying objects, fixtures, descriptions, and skill sequences.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4 DATASETS

• Datasets are divided into pretraining data collected in pretraining scenes and target data collected in target scenes for targeted collection and most evaluations.The pretraining kitchens support large-scale collection and synthetic-data generation.Source: §3.2 · p. 4, §3.4 · p. 5

3.4.1 PRETRAINING DATASETS

• Pretraining covers 300 tasks with 30,000 human demonstrations, while MimicGen produces 10,000 demonstrations for each of 60 atomic tasks, scaling their seed data 100×.Human demonstrations comprise 65 atomic and 235 composite tasks, with 100 teleoperated demonstrations per task; synthetic demonstrations use those human demonstrations as seeds.Source: §3.4.1 · pp. 5–6

3.4.2 TARGET DATASETS

• Target data covers 50 tasks—18 atomic, 16 composite-seen, and 16 composite-unseen—with 500 human demonstrations per task, totaling 25,000 demonstrations.The unseen composite tasks are absent from pretraining and cover a distinct set of 16 activities; seen composite tasks range from two subtasks to a longest task of 15.Source: §3.4.2 · p. 6

3.4.3 DATASET STATISTICS

• The datasets comprise 1,615 hours of synthetic pretraining data, 404 hours of human pretraining data, and 208 hours of human target data across 55,000 human episodes.Most tasks require one or two subtasks, though some require at least 15; most episodes last 10–60 seconds, with some exceeding three minutes.Source: §3.4.3 · p. 6, Figure 4 · p. 6

• The experiments are designed to identify key factors influencing generalist robot policy training through a comprehensive benchmark suite.Source: §4 · p. 6

4.1 MULTI-TASK TRAINING

• GR00T N1.5 performs best among the tested methods, followed by π0.5, π0, and Diffusion Policy, with non-zero success on Composite-Unseen tasks.Atomic tasks are easiest, followed by Composite-Seen and Composite-Unseen; the authors caution that the ranking does not establish GR00T N1.5 as conclusively superior.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 FOUNDATION MODEL TRAINING

• Roughly 3× greater data efficiency: pretraining achieves roughly the same downstream performance as target-only training with three times as many target demonstrations.Pretraining alone exceeds 40% success on atomic tasks but performs poorly on composite tasks; its gains are especially pronounced on Composite-Unseen tasks.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 LIFELONG LEARNING

• Success rates decline as lifelong-learning phases add progressively longer-horizon tasks, while performance on earlier tasks steadily falls, revealing catastrophic forgetting.The benchmark trains over four phases, from 65 atomic tasks through increasingly long composite tasks, and evaluates current and previous phases after each phase.Source: §4.3 · pp. 8–9, Table 3 · p. 9

4.4 PRETRAINING DATA COMPOSITION STUDY

• Human300 pretraining yields better downstream results than Human300 + MG60, while increasing human pretraining tasks from 50 to 300 especially benefits low-data Composite-Unseen tasks.The authors note that MimicGen demonstrations vary in quality; the largest gains from greater task diversity occur on Composite-Unseen tasks.Source: §4.4 · p. 9, Table 4 · p. 9

4.5 REAL-WORLD EXPERIMENTS

• 79.8% average success with Sim-and-Real training versus 61.8% with Real Only training across four real-world tasks.The Sim-and-Real model was mid-trained on simulation data and co-finetuned with real-world demonstrations.Source: Table 5 · p. 10, §4.5 · p. 10

5 CONCLUSION

• RoboCasa365 provides 2,500 realistic kitchen environments, 365 everyday tasks across over 50 activity categories, and over 2,000 hours of robot interaction data.The paper presents it as a large-scale simulation framework for training and benchmarking generalist robot models and one of the most diverse simulation resources to date.Source: §5 · p. 10
• Pretraining data improves downstream learning, with scale and task diversity both important, while lifelong learning trades off acquiring new tasks against retaining prior knowledge.The systematic study covers multi-task learning at scale, foundation model learning, and lifelong learning; generalist policies trained on large multi-task datasets acquire broad competence but still struggle with long-horizon tasks.Source: §5 · p. 10
• The benchmark is limited to kitchens and does not capture the full sensory and physical complexity of the real world, leaving transfer and sim-to-real deployment as future challenges.The conclusion identifies evaluation in other household settings or broader domains as an open question.Source: §5 · p. 10

A USE OF LARGE LANGUAGE MODELS

• Large language models support activity-label and task-blueprint creation using the process outlined by Nasiriany et al. (2024), as well as manuscript writing feedback.The paper specifies that language models were used for feedback on parts of the manuscript.Source: A USE OF LARGE LANGUAGE MODELS · p. 15

B.1 PHYSICS AND RENDERING ENGINE

• RoboCasa365 uses RoboSuite (Zhu et al., 2020) with MuJoCo physics, GPU-based rendering, and 20 Hz simulation that runs approximately in real time.Asynchronous environments run in parallel, with throughput scaling according to available CPU cores and GPUs and varying with scene complexity and hardware.Source: §B.1 · p. 16

B.2 ACTION SPACE

• The controller uses a 20 Hz Operational Space Controller with seven arm-action dimensions and five additional dimensions for mobile-base and torso control.The arm dimensions cover translation, rotation, and gripper opening and closing; the additional dimensions cover base translation and rotation, torso height, and a base-control gating mode.Source: §B.2 · p. 16

C SIMULATION ASSETS

• Simulation assets include objects across 57 categories, an inventory of interactive fixtures and appliances in Table 6, and 50 kitchen layouts modeled after homes across five U.S. locations.The homes span the Bay Area, Austin, Denver, Boston, and Atlanta.Source: §C.1 · p. 16, §C.2 · p. 16, Table 6 · p. 16

E.1 ATOMIC TASKS

• The appendix lists 65 atomic tasks, including appliance controls, navigation, and object pick-and-place tasks.Examples include OpenFridge, NavigateKitchen, and PickPlaceCounterToSink.Source: §E.1 · p. 17

E.2 TARGET TASKS

• Figure 7 lists 18 post-training atomic-seen tasks.Source: Figure 7 · p. 18
• The “MoMa req.” column indicates whether a task requires mobile manipulation or base navigation.Source: §E.2 · p. 18
• Figure 8 lists 16 post-training composite-seen tasks.Source: Figure 8 · p. 19
• Figure 9 lists 16 post-training composite-unseen tasks.Source: Figure 9 · p. 20
• The model receives 256 × 256 camera images from three views.Source: Figure 10 · p. 21

F DATASETS

• Table 7 presents dataset statistics across pretraining and target settings.The passage does not provide the table's values.Source: Table 7 · p. 21, F DATASETS · p. 21
• Each demonstration stores its language instruction.Source: F DATASETS · p. 21
• Stored proprioceptive information includes the robot base pose and end-effector pose.Source: F DATASETS · p. 21
• Stored proprioceptive information also includes gripper-state information.Source: F DATASETS · p. 21
• Each demonstration stores images from wrist, left third-person, and right third-person cameras.Source: F DATASETS · p. 21
• Each demonstration stores the actions.Source: F DATASETS · p. 21

G.1 MODEL ARCHITECTURES AND TRAINING PROTOCOL

• Vision-based models use proprioception, task instructions, and one wrist plus two third-person camera images as inputs.Each camera image has 256 × 256 resolution.Source: §G.1 · p. 21
• Diffusion Policy uses CLIP language embeddings with FiLM conditioning and a 12-layer, 512-dimensional transformer; π0/π0.5 use PaLI Gemma with a flow-matching action expert.The cited components are CLIP (Radford et al., 2021), FiLM (Perez et al., 2018), PaLI Gemma (Beyer et al., 2024), and flow matching (Lipman et al., 2023). Diffusion Policy trains for 250k steps with batch size 192; π0/π0.5 train for 75k steps with batch size 64 in the multi-task experiment.Source: §G.1 · p. 21
• GR00T N1.5 pairs the Eagle2 high-level encoder with a flow-matching action decoder and freezes its vision and language encoders.With batch size 128, it trains for 120k multi-task steps; foundation-model runs use 80k pretraining and 60k fine-tuning steps, while lifelong learning uses 100k stage-one steps and 60k for subsequent stages.Source: §G.1 · pp. 21–22

G.2 EVALUATION PROTOCOL

• Evaluation runs 30 trials per task to a task-dependent timestep limit and reports average success rate across tasks.An episode succeeds only if the agent meets the task’s binary success condition within the limit.Source: §G.2 · p. 22

H.1 PRETRAINING SCENE DIVERSITY

• Increasing pretraining scene diversity improves zero-shot performance and downstream performance after target-task fine-tuning.The comparison evaluates pretraining on 2,500 scenes (50 layouts × 50 styles), 25 scenes (5 layouts × 5 styles), or 5 scenes (5 layouts × 1 style).Source: §H.1 · p. 22, Table 8 · p. 22
• Pretraining demonstrations cover 17 atomic tasks in kitchens, with zero-shot evaluation on 10 fixed target scenes and an additional target-data learning condition.The target-data condition uses 50 demonstrations per task; MimicGen generates demonstrations for each pretraining-scene setting.Source: §H.1 · p. 22

• The evaluation tests GR00T N1.5 trained on the full pretraining and target mixture on Composite-Seen and Composite-Unseen tasks.Source: §H.2 · p. 22
• Perturbations include LLM-generated semantically similar instructions and Gaussian noise on starting joint angles, base position and yaw, and third-person and wrist camera poses.Source: §H.2 · p. 22
• The model is robust to language variations but can suffer under novel camera poses, joint angles, and base poses.Source: §H.2 · p. 22

H.3 JOINT CO-TRAINING OF PRETRAINING AND TARGET DATA

• After 120k steps of joint training on all pretraining data and 100% of target data, success rates were 44.1% Atomic-Seen, 9.0% Composite-Seen, 11.7% Composite-Unseen, and 22.5% Average.These are task success rates in the target kitchens.Source: §H.3 · p. 23
• Joint co-training performed substantially worse than pretraining followed by target-data fine-tuning, underscoring the importance of a dedicated fine-tuning phase for target tasks.Source: §H.3 · p. 23

H.4 LORA FINE-TUNING

• Full fine-tuning is critical to GR00T N1.5 multi-task performance compared with LoRA fine-tuning.The comparison used the same number of training steps and batch size for both variants.Source: §H.4 · p. 23, Table 10 · p. 23

I.1 FOUNDATION MODEL TRAINING ANALYSIS

• For the best variant (pretraining then fine-tuning on 100% of target data), Atomic-Seen failures center on precision and dexterity, while Composite-Seen/Unseen failures span diverse task characteristics.TurnOffStove and CloseBlenderLid are the worst Atomic-Seen tasks; the analysis introduces failure cases at 30% success or less.Source: §I.1 · p. 23
• Low-success manipulation tasks show unreliable grasping, placement, and object matching, including bowls, stove burners, lunch items, coffee mugs, hot dogs, and condiments.SteamInMicrowave, SearingMeat, PackIdenticalLunches, and PrepareCoffee are described alongside PortionHotDogs and CategorizeCondiments.Source: §I.1 · pp. 23–24
• Appliance and dynamic-task failures include incomplete toaster-lever presses, mishandling toaster racks, and failing to flip pan contents onto a plate.PanTransfer is identified as a dynamic task with little overlap with other benchmark tasks.Source: §I.1 · p. 24
• Other failures involve access and transport: the model struggles to open drawers and carry or place a straw, locate and return a mug, and place Tupperware in a tight freezer.These examples come from DeliverStraw, GatherTableware, and SeparateFreezerRack.Source: §I.1 · p. 24