Source-linked AI summary
RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots
Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, Yuke Zhu
TL;DR
RoboCasa365 addresses the lack of reproducible, large-scale benchmarks for generalist household robots by providing a broad simulation benchmark spanning tasks, environments, and learning settings. Experiments show broad competence alongside long-horizon and lifelong-learning challenges, while pretraining improves downstream learning and both dataset scale and task diversity matter.
Problem
Existing robot-learning datasets remain limited in diversity and task coverage, while real-world evaluation is resource-intensive, noisy, and difficult to reproduce systematically.
Method
RoboCasa365 is a simulation benchmark built on RoboCasa, combining 365 household tasks, 2,500 diverse kitchen environments, and evaluations of multi-task, foundation-model, and lifelong learning.
Results
Generalist policies acquire broad competence but struggle with long-horizon tasks; pretraining improves downstream learning, while lifelong learning involves trade-offs between acquiring new tasks and retaining prior knowledge.
Takeaways & Limitations
RoboCasa365 enables systematic study of generalist robot learning across task diversity, dataset scale, environment variation, and lifelong learning.
Takeaways & Limitations
The benchmark is limited to kitchen environments and does not capture the full sensory and physical complexity of the real world.
Abstract
from arXiv · showhide
Recent advances in robot learning have accelerated progress toward generalist robots that can perform everyday tasks in human environments. Yet it remains difficult to gauge how close we are to this vision. The field lacks a reproducible, large-scale benchmark for systematic evaluation. To fill this gap, we present RoboCasa365, a comprehensive simulation benchmark for household mobile manipulation. Built on the RoboCasa platform, RoboCasa365 introduces 365 everyday tasks across 2,500 diverse kitchen environments, with over 600 hours of human demonstration data and over 1600 hours of synthetically generated demonstration data -- making it one of the most diverse and large-scale resources for studying generalist policies. RoboCasa365 is designed to support systematic evaluations for different problem settings, including multi-task learning, robot foundation model training, and lifelong learning. We conduct extensive experiments on this benchmark with state-of-the-art methods and analyze the impacts of task diversity, dataset scale, and environment variation on generalization. Our results provide new insights into what factors most strongly affect the performance of generalist robots and inform strategies for future progress in the field.
1 INTRODUCTION
RoboCasa365 addresses limited task diversity and costly, noisy real-world evaluation by providing a large-scale simulation benchmark for systematic generalist-robot studies.
- Existing robot datasets remain limited in diversity and task coverage despite substantial growth, constraining robust generalist-policy training.
- Simulation enables large-scale interaction data, rapid experimentation, controlled evaluation, and reproducible benchmarking beyond the practical limits of real-world robotics.
- RoboCasa365 defines 365 tasks across 60 kitchen activities, covering manipulation, semantic reasoning, long-horizon planning, and memory-dependent tasks.
- The benchmark includes 2,500 unique kitchen scenes modeled from real kitchens across the United States, varying layouts, object configurations, and visual appearances.
- RoboCasa365 provides over 2,000 hours of interaction data, combining 612 hours of human demonstrations with 1615 hours of synthetic demonstrations.
- The benchmark supports massively multi-task training, foundation model training, and lifelong learning with reproducible evaluation and analysis of generalization factors.
2 RELATED WORK
Prior work includes robot simulation frameworks, real-world robot datasets, and generalist-policy training, while RoboCasa365 combines broad task coverage, diverse scenes, demonstrations, and systematic benchmarks.
- Robot Simulation Frameworks: Existing simulation frameworks span tabletop and room-scale settings, whereas RoboCasa365 targets hundreds of tasks across thousands of unique scenes.
- Datasets and Benchmarks for Generalist Robots: Real-world robot datasets have expanded, but systematic policy evaluation remains challenging because large-scale real-world benchmarking requires substantial resources.
- Training Generalist Robots: Generalist-policy research learns from large, diverse robot datasets, while RoboCasa365 remains model-agnostic and benchmarks capabilities across distinct learning settings.
3 ROBOCASA365: LARGE-SCALE SIMULATION OF 365 EVERYDAY TASKS
RoboCasa365 scales RoboCasa into a structured simulation resource with diverse assets, kitchen scenes, 365 tasks, and large human and synthetic datasets for generalist-robot training and evaluation.
- RoboCasa365 expands assets, environments, tasks, and datasets from RoboCasa while establishing benchmarks for state-of-the-art policy-learning methods.
- Assets: The framework adds high-quality 3D assets across 57 object categories to RoboCasa’s existing library of 2,509 objects spanning 153 categories.
- Assets: Interactable fixtures and appliances expand from 20 instances across 4 categories to 456 instances spanning 12 categories, including articulated appliances.
- Diverse Kitchen Scenes: Scenes are organized into pretraining and target splits, with each kitchen defined by a layout and style combination.
- Diverse Kitchen Scenes: Pretraining uses 50 new layouts sourced from 50 real-world homes, while target data contains 10 kitchen scenes formed from matched layouts and styles.
- Suite of 365 Everyday Tasks: The task suite combines 65 atomic tasks with 300 composite tasks generated from 60 high-level activities and task blueprints.
- Datasets: Pretraining covers 300 tasks with 30k human demonstrations, while MimicGen generates 10k synthetic demonstrations per task from 100 human seeds across 60 atomic tasks.
- Datasets: Target data samples 50 tasks across atomic, Composite-Seen, and Composite-Unseen splits, with 500 human demonstrations per task.
4 EXPERIMENTS
The experiments establish a systematic benchmark suite for studying factors that influence training generalist robot policies and their adaptation to novel tasks.
- The benchmark suite is designed to answer questions about the factors influencing training generalist robot policies.
1. How well do generalist robot models perform when trained on large multi-task datasets?
RoboCasa365 evaluates generalist robot learning across massively multitask training, foundation-model pretraining, lifelong learning, and pretraining-data composition. Results show stronger generalization with larger and more diverse data, while longer-horizon learning and continual adaptation remain challenging.
- Multi-task training: Across multi-task methods, Atomic tasks are easiest, Composite-Seen tasks are harder, and Composite-Unseen tasks are hardest.GR00T N1.5 performs best overall, followed by π0.5, π0, and Diffusion Policy, but the authors caution that compute and training choices also affect performance.
- Foundation model training: Pretraining improves downstream learning, especially for Composite-Unseen tasks, and provides roughly 3× greater data efficiency than target-task learning alone.Pretraining helps achieve approximately the same performance as target-only learning with three times fewer target-task demonstrations.
- Foundation model training: With pretraining alone, performance exceeds 40% on Atomic tasks but remains very poor on Composite tasks.This contrast indicates that pretraining transfers more readily to shorter-horizon Atomic tasks than to Composite tasks.
- Lifelong learning: In lifelong learning, success rates steadily decline as newly learned tasks become progressively longer-horizon.The benchmark trains across four phases and evaluates all tasks seen up to each phase.
- Lifelong learning: Previously learned-task performance also declines across lifelong-learning phases, exposing catastrophic forgetting.The authors describe this as performance degradation on prior tasks when the agent is not trained on them in subsequent phases.
- Pretraining data composition: Human-only pretraining outperforms mixed human-and-MimicGen pretraining, while expanding human pretraining from 50 to 300 tasks especially benefits low-data Composite-Unseen learning.The authors attribute the mixed-data result to variation in synthetic demonstration quality and identify better use of mixed-quality datasets as future work.
5 CONCLUSION
RoboCasa365 is presented as a large-scale simulation framework for training and benchmarking generalist robot models, combining diverse kitchen environments, everyday tasks, and extensive interaction data. Experiments identify broad competence, pretraining benefits, and lifelong-learning trade-offs, while the benchmark remains limited in domain and realism.
- RoboCasa365 provides 2,500 realistic kitchen environments, 365 everyday tasks spanning over 50 activity categories, and over 2,000 hours of robot interaction data.
- Experiments study multi-task learning at scale, foundation model learning, and lifelong learning using the benchmark.
- Large multi-task datasets support broad competence, but generalist policies still face challenges with long-horizon tasks.
- Pretraining data significantly improves downstream learning, with both dataset scale and task diversity playing key roles.
- Lifelong learning remains an open challenge because acquiring new tasks involves substantial trade-offs with retaining prior knowledge.
- The benchmark is limited to kitchen environments and does not capture the full sensory and physical complexity of the real world, leaving simulation-to-deployment transfer unresolved.
E.2 TARGET TASKS
The target-task suite organizes evaluation around atomic, composite-seen, and composite-unseen tasks, with figures documenting the post-training task groups and camera-image inputs. Task descriptions also indicate whether mobile manipulation or base navigation is required.
- The target-task overview covers post-training atomic-seen tasks, composite-seen tasks, and composite-unseen tasks.
- The “MoMa req.” column indicates whether a task requires mobile manipulation or base navigation.
- DeliverStraw requires taking a straw from a drawer and placing it inside a glass cup on the dining counter.
- Camera images are rendered at 256 × 256 resolution and supplied from three views.
F DATASETS
The dataset documentation presents statistics for pretraining and target settings and records multimodal demonstrations containing language, proprioception, camera images, and actions.
- The dataset statistics are summarized across pretraining and target settings.
- Each demonstration stores a language instruction, proprioceptive information, images from three cameras, and robot actions.
G.1 MODEL ARCHITECTURES AND TRAINING PROTOCOL
The experiments train vision-based models from proprioception, language instructions, and multi-view camera images, compare four model families, and evaluate checkpointed policies by repeated task trials. Pretraining scene diversity is varied to assess downstream performance.
- Vision-based models receive base pose, end-effector pose, gripper state, task language, and one wrist plus two third-person camera images.
- Each camera image has 256 × 256 resolution, and the study experiments with four models.
- Diffusion Policy generates action trajectories through conditional denoising with language conditioning fused into the visual encoder.
- π0 and π0.5 use a PaLI Gemma vision-language model with an action expert that outputs robot actions via flow matching.
- GR00T N1.5 uses a system1-system2 architecture, with a high-level vision-language encoder and a flow-matching action decoder.
- Evaluation runs 30 trials per task at a specified checkpoint and reports average success rates across tasks.
- Pretraining compares 2,500, 25, and 5 kitchen scenes, using MimicGen demonstrations and zero-shot evaluation on 10 fixed target scenes.
- Increasing the composition of scenes in pretraining data improves downstream task performance.
H.2 ROBUSTNESS EVALUATIONS
Robustness evaluations perturb task language and the robot’s physical starting conditions to assess generalization on seen and unseen composite tasks. The model handles language variation but is less robust to novel camera poses, joint angles, and base poses.
- Perturbations: The evaluation perturbs language instructions, starting joint angles, robot base pose, and camera pose on Composite-Seen and Composite-Unseen tasks.Novel language uses semantically similar instructions; the physical perturbations add Gaussian noise to the robot state or camera poses.
- Findings: The model is robust to language variations but can suffer from novel camera poses, joint angles, and base poses.
H.3 JOINT CO-TRAINING OF PRETRAINING AND TARGET DATA
Jointly training pretraining and target data in one phase performs substantially worse than the two-stage framework that fine-tunes after pretraining. The results support retaining a dedicated target-data fine-tuning phase.
- Joint co-training: 22.5% average success was achieved with joint co-training, including 44.1% on Atomic-Seen, 9.0% on Composite-Seen, and 11.7% on Composite-Unseen.The model was trained for 120k steps on all pretraining data and 100% of target data jointly.
- Comparison: Joint co-training performs substantially lower than pretraining followed by fine-tuning on target data.
- Fine-tuning: Full fine-tuning is critical to model performance.The multi-task comparison used GR00T N1.5 with LoRA fine-tuning under matched training conditions.
I.1 FOUNDATION MODEL TRAINING ANALYSIS
Foundation-model analysis identifies task-specific failures across navigation, manipulation, placement, precision, and dynamic interactions. Composite tasks exhibit failures spanning diverse characteristics, while several atomic failures involve high precision and dexterity.
- Per-task performance: Atomic-Seen failures are concentrated in TurnOffStove and CloseBlenderLid, which require high precision and dexterity.
- Failure modes: Composite-task failures include incorrect microwave placement, burner selection, fridge picking, tupperware placement, and coffee-mug positioning.
- Failure modes: Other failures involve drawer access, object transport, lever pressing, crowded-bowl picking, wrong-plate placement, condiment matching, and tight-space freezer placement.
- Failure modes: PanTransfer is difficult because flipping pan contents is dynamic and has little overlap with other benchmark tasks.
- Failure modes: HeatKebabSandwich often fails during toaster-rack handling, especially after the first item is placed.
- Failure modes: GatherTableware fails through unreliable navigation and incorrect or incomplete mug placement inside the cabinet.