PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots

Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, Yuke Zhu

arXiv:2406.02523v1cs.ROcs.AIcs.LG

TL;DR

Collecting ever-larger real-world robot datasets requires unrealistic amounts of capital and labor. RoboCasa offers a kitchen simulation framework with generated training data, and experiments report a scaling trend for synthetic data and increased task success with simulation co-training in real-robot deployment.

  • Problem

    Collecting ever-larger real-world robot datasets requires unrealistic amounts of capital and labor, motivating simulation as an alternative source of training data.

    Source: §I · pp. 1–2

  • Method

    RoboCasa provides realistic, diverse kitchen simulation scenes, 100 evaluation tasks, human demonstrations, and automatically generated trajectories for training generalist robots.

    Source: §I · p. 2

  • Results

    Experiments show a clear scaling trend with generated data, improved generalization, and increased task success when simulation data is used in real-robot co-training.

    Source: §I · p. 2

  • Takeaways & Limitations

    The experiments support simulation data as useful for scaling robot policy learning and for co-training in real-world deployment.

    Source: §I · p. 2

    Fine-tuning on composite tasks yields relatively low performance, and some generated trajectories exhibit jerky motions and collisions.

    Source: §VI · p. 8

Abstract

from arXiv · show

Recent advancements in Artificial Intelligence (AI) have largely been propelled by scaling. In Robotics, scaling is hindered by the lack of access to massive robot datasets. We advocate using realistic physical simulation as a means to scale environments, tasks, and datasets for robot learning methods. We present RoboCasa, a large-scale simulation framework for training generalist robots in everyday environments. RoboCasa features realistic and diverse scenes focusing on kitchen environments. We provide thousands of 3D assets across over 150 object categories and dozens of interactable furniture and appliances. We enrich the realism and diversity of our simulation with generative AI tools, such as object assets from text-to-3D models and environment textures from text-to-image models. We design a set of 100 tasks for systematic evaluation, including composite tasks generated by the guidance of large language models. To facilitate learning, we provide high-quality human demonstrations and integrate automated trajectory generation methods to substantially enlarge our datasets with minimal human burden. Our experiments show a clear scaling trend in using synthetically generated robot data for large-scale imitation learning and show great promise in harnessing simulation data in real-world tasks. Videos and open-source code are available at https://robocasa.ai/

I. INTRODUCTION

RoboCasa addresses the scarcity and cost of real-world robot data with a realistic, diverse kitchen simulator, evaluation tasks, and robot datasets. The paper reports that generated data improves generalization and simulation data benefits real-robot task success.

I. INTRODUCTION

  • Robot data scarcity and the cost of collecting larger real-world datasets motivate simulation, which can generate data cheaply and support rapid, reproducible research.The authors argue that useful simulation must combine realism in physics, rendering, and models with diversity in scenes, assets, and tasks, alongside large robot datasets.Source: §I · pp. 1–2
  • RoboCasa provides realistic, diverse kitchen scenes, over 2,500 objects across 150 categories, interactable furniture and appliances, and support for multiple mobile-robot embodiments.The framework builds on RoboSuite and uses generative AI tools for 3D objects and environment textures.Source: §I · p. 2
  • Its 100 evaluation tasks comprise 25 atomic foundational skills and 75 composite kitchen activities designed with large-language-model guidance.Examples of atomic skills include picking and placing, opening and closing doors, and twisting knobs; composite activities include washing dishes, frying, and restocking cabinets.Source: §I · p. 2
  • Generated data significantly improves generalization, and co-training with RoboCasa simulation data significantly increases task success in real-robot deployment.The authors extend MimicGen to generate 100K additional trajectories for atomic tasks and train policies with behavioral cloning on human demonstrations and generated data.Source: §I · p. 2

II. RELATED WORK

The related work presents RoboCasa as combining diverse, room-scale simulation with large-scale demonstration datasets, and situates its data-generation and policy-learning choices among existing approaches.

Simulation Frameworks for Robotics

  • RoboCasa combines room-scale mobile manipulation, realistic interaction physics, photorealistic rendering, multiple robot embodiments, and at-scale tasks and objects with AI-generated tasks and assets.It also provides large-scale task demonstrations using human teleoperation and MimicGen [34], alongside imitation-learning analysis across its task collection. The paper describes this combination of scenes, tasks, assets, and data as a need not addressed by other robotics simulation frameworks.Source: §II · pp. 2–3

Datasets and Benchmarks for Robotics

  • RoboCasa collects human demonstrations in simulation and uses MimicGen [34] to expand them, combining teleoperation quality and applicability with the scale of programmed demonstrators.The related work contrasts this approach with self-supervised trial-and-error collection, real-world teleoperation datasets, and simulation trajectory generators that may rely on privileged information, hand-designed heuristics, or engineered pipelines.Source: §II · p. 3

Learning from Large Offline Datasets

  • For learning from offline datasets, RoboCasa uses Behavioral Cloning with a Transformer-based visuomotor policy and also considers diffusion-model policies.Behavioral Cloning imitates dataset actions, while Offline Reinforcement Learning uses a reward function to prefer some dataset actions over others.Source: §II · p. 3

III. ROBOCASA SIMULATION … C. Assets

RoboCasa builds a scalable simulation framework for kitchen robotics, combining adaptable simulation and robot support with diverse kitchen environments, interactive appliances, and a large library of kitchen objects.

III. ROBOCASA SIMULATION

A. Core Simulation Platform

  • RoboCasa extends RoboSuite for room-scale simulation and supports mobile manipulators, including wheeled robots, humanoids, and quadrupeds with arms.It also supports photorealistic rendering with NVIDIA Omniverse.Source: A. Core Simulation Platform · p. 3

B. Kitchen Scenes

  • RoboCasa models 10 kitchen floor plans, from apartment layouts to high-end designs, and showcases their varied shapes and sizes.Source: Figure 3 · p. 4, B. Kitchen Scenes · p. 4
  • The 10 floor plans can each use any of 12 kitchen styles, yielding 120 scenes, with distinct design elements such as textures, appliances, cabinet panels, and handles.Source: B. Kitchen Scenes · p. 4
  • Each scene can use 100 AI-generated textures apiece for walls, floors, counters, and cabinet panels, increasing visual diversity through domain randomization.Source: B. Kitchen Scenes · p. 4

C. Assets

  • RoboCasa adapts online 3D assets into MuJoCo MJCF models and segments appliances into articulated parts for interactions such as opening microwave doors and turning stove knobs.Appliance state changes can simulate effects such as a stove burner turning on when its knob is turned.Source: C. Assets · p. 4, Figure 4 · p. 4
  • Its object library contains 2,509 high-quality assets across 153 categories, including 1,592 assets from Luma.ai, alongside objects sourced from Objaverse [8].Source: C. Assets · p. 4

IV. ROBOCASA ACTIVITY DATASET · A. Atomic Tasks: Building Blocks of Behavior · B. Creating Composite Tasks with Large Language Models

RoboCasa organizes kitchen activities into a 100-task suite, using atomic sensorimotor tasks and LLM-guided composite tasks to cover foundational skills and semantically meaningful household activities.

IV. ROBOCASA ACTIVITY DATASET

A. Atomic Tasks: Building Blocks of Behavior

  • The atomic-task set contains 25 tasks covering eight foundational skills, from pick-and-place and door or drawer operation to insertion and navigation.The authors note that this skill set is not exhaustive and leave deformable-manipulation behaviors for future work.Source: A. Atomic Tasks: Building Blocks of… · p. 5

B. Creating Composite Tasks with Large Language Models

  • Composite tasks sequence skills into semantically meaningful activities such as cooking and cleaning, aiming to reflect the ecological statistics of real-world household activities.Source: B. Creating Composite Tasks with Large… · p. 5
  • The authors first prompted ChatGPT (GPT-4) to list 20 common high-level kitchen activities, ranging from brewing coffee or tea to steaming vegetables.Source: B. Creating Composite Tasks with Large… · p. 5
  • They then prompted GPT-4 and Gemini 1.5 to propose representative tasks for each activity label, filtering or modifying outputs when the LLMs showed logical flaws.The authors cite broad human knowledge and reduced labor at scale as benefits of using LLMs for task definition.Source: B. Creating Composite Tasks with Large… · p. 5
  • The authors compiled 75 LLM-derived task blueprints and implemented them; all but select environment-specific composite tasks can be simulated in any RoboCasa kitchen scene.Source: B. Creating Composite Tasks with Large… · p. 5

C. RoboCasa Datasets · V. EXPERIMENTS

RoboCasa builds demonstrations for 100 tasks by starting with human teleoperation and expanding the data through automated trajectory generation. The experiments examine the learning value and scalability of these datasets, their transfer to downstream simulation tasks, and their use for real-world policy learning.

C. RoboCasa Datasets

  • Demonstrations for RoboCasa’s 100 tasks start with human teleoperation and are expanded using automated trajectory generation.The task set comprises 25 atomic and 75 LLM-created composite tasks.Source: C. RoboCasa Datasets · p. 5
  • Four operators collect 50 demonstrations per atomic task in randomized kitchen scenes, yielding 1,250 demonstrations that remain insufficient for most tasks.The authors attribute this insufficiency likely to the scope and diversity of the tasks and scenes.Source: C. RoboCasa Datasets · p. 5
  • MimicGen [34] expands human seeds by decomposing demonstrations into object-centric segments, adapting them to objects’ poses in new scenes, stitching them together, and executing the resulting trajectories.Successful executions become new task demonstrations.Source: C. RoboCasa Datasets · p. 5
  • RoboCasa makes MimicGen’s sequence and segment-annotation requirements reusable across tasks by specifying subtask sequences and automated end-of-subtask metrics for eight core skills.Tasks sharing a skill have the same or similar object-centric subtask sequence, often differing mainly in reference-object identity.Source: C. RoboCasa Datasets · p. 5
  • GPT-4 is prompted to generate diverse high-level kitchen activities; subsequently, for each activity, GPT-4 or Gemini 1.5 is prompted to suggest a diverse set of representative tasks.Source: Figure 6 · p. 6
  • MimicGen retains only generation attempts that succeed, and RoboCasa parallelizes generation across simulation processes.The success-only selection uses rejection sampling.Source: C. RoboCasa Datasets · p. 6

V. EXPERIMENTS

  • The experiments ask whether MimicGen trajectories compare favorably with human demonstrations, whether generalization scales with dataset size, and whether simulation data supports downstream simulation and real-world policy learning.Source: §V · p. 6

A. Imitation Learning for Atomic Tasks

Across 24 atomic tasks, imitation learning performance rises with the size of machine-generated datasets, and the full generated dataset outperforms human demonstrations. The evaluation tests generalization to unseen object instances and styles.

A. Imitation Learning for Atomic Tasks

  • The four training sets comprised 1,250 human demonstrations across 25 tasks, 72,000 generated demonstrations across 24 tasks, and generated subsets of 7,200 and 2,400 trajectories.The 3,000-per-task generated set was produced from 50 human demonstrations per task; the smaller sets contain 300 or 100 demonstrations per task.Source: A. Imitation Learning for Atomic Tasks · p. 6
  • The reported experiments used Objaverse objects and excluded kitchen navigation because MimicGen could not generate mobile-manipulation trajectories; an additional 28K AI-object trajectories brought the released dataset to 100K.Kitchen navigation was left for future work.Source: A. Imitation Learning for Atomic Tasks · p. 6
  • Policies used behavioral cloning with RoboMimic’s BC-Transformer for a Franka Panda with an Omron mobile base, training on AI-textured images and evaluation in human-textured scenes.Evaluation used 50 trials per task across five fixed scenes; all test object instances were unseen, and two scenes had unseen styles.Source: A. Imitation Learning for Atomic Tasks · p. 6
  • 47.6% success rate with the fully generated dataset exceeded 28.8% with human data, and success rates rose steadily as generated-data volume increased.Results cover 24 atomic tasks grouped by sensorimotor skill.Source: A. Imitation Learning for Atomic Tasks · p. 6, Figure 7 · p. 7
  • Opening and closing doors and drawers were easier to learn than pick-and-place and insertion tasks, which the authors associate with greater object diversity and dexterity demands.Pick-and-place spans dozens of object categories and affordances, while door-opening involves six door instances.Source: A. Imitation Learning for Atomic Tasks · p. 7

B. Imitation Learning for Composite Tasks

Fine-tuning policies pretrained on machine-generated atomic-task data outperformed learning from scratch on five challenging composite tasks, though robust performance remained elusive. The study used 50 human demonstrations per task and found persistent manipulation and stage-transition failures.

B. Imitation Learning for Composite Tasks

  • Each composite task used a separate policy trained with 50 human demonstrations, either from scratch or by fine-tuning an atomic-task policy pretrained on the full MimicGen dataset.The authors chose single-task policies because composite tasks require multiple skills, lengthen the task horizon, and complicate multi-task learning.Source: B. Imitation Learning for Composite… · p. 7
  • Fine-tuning achieved non-zero success on 4/5 tasks, while scratch learning failed to achieve any non-zero success on 4/5.The figure summarizes the comparison across five representative composite tasks; fine-tuned policies still struggled to learn robust behaviors.Source: Figure 8 · p. 7, B. Imitation Learning for Composite… · p. 7
  • Fine-tuned models showed more robust picking and placing qualitatively, but still struggled with fine-grained manipulation and transitioning effectively between task stages.The authors attribute the improved behaviors to pretraining on a large dataset of atomic behaviors.Source: B. Imitation Learning for Composite… · p. 7
  • The benchmark leaves substantial room for improvement, and policy architecture, learning algorithm, and fine-tuning strategy may critically affect performance.The authors identify these factors as directions for future investigation.Source: B. Imitation Learning for Composite… · p. 7

C. Transfer to Real World Environments

Co-training real-world demonstrations with RoboCasa simulation data improves real-robot policy success on seen objects, and also improves performance on unseen objects despite lower absolute performance. The experiments evaluate three pick-and-place tasks in a real kitchen using a Franka Panda robot.

C. Transfer to Real World Environments

  • The real-world setup uses a Franka Emika Panda on DROID hardware, with differences from simulation in control, camera calibration, lighting, and robot-base placement.Both robots use workspace end-effector control, but only the simulated robot uses Operational Space Control; the controllers run at 20 Hz in simulation and 15 Hz on the real robot.Source: C. Transfer to Real World Environments · p. 8
  • The real-kitchen evaluation covers counter-to-sink, sink-to-counter, and counter-to-cabinet pick-and-place tasks, with 50 demonstrations per task across five object categories.A separate policy is trained for each task.Source: C. Transfer to Real World Environments · p. 8
  • Real only trains on target-task demonstrations, while Real + Sim co-trains on those demonstrations and RoboCasa MimicGen demonstrations across all single-stage tasks.The two settings compare real-world-only training with co-training using real and simulated data.Source: C. Transfer to Real World Environments · p. 8
  • Performance suffers on unseen objects, but incorporating simulation data still yields a significant improvement.The unseen categories are those not represented in the real-world demonstrations; no success-rate values are provided for them in the supplied passage.Source: C. Transfer to Real World Environments · p. 8
  • 24.4% average success on seen objects with Real + Sim, versus 13.6% with Real only, a 79% relative improvement.Figure 10 reports mean and standard deviation over three seeds; each seed evaluates five seen and three unseen object categories.Source: Figure 10 · p. 8, C. Transfer to Real World Environments · p. 8

VI. CONCLUSION · VII. SIMULATOR · VIII. TASKS AND DATASETS

RoboCasa combines realistic kitchen simulation, generative AI, and large-scale task and trajectory datasets for generalist robot learning. Its simulator approaches real-time speed, while the paper identifies limitations in task performance, generated trajectories, and dataset scope.

VI. CONCLUSION

  • RoboCasa provides 120 realistic scenes, dozens of appliances, 2,500+ 3D objects across 150+ categories, 100 tasks, and 100K+ trajectories.Generative AI contributes text-to-3D object assets, text-to-image environment textures, and LLM-generated kitchen activities and tasks; experiments show synthetic simulation data can scale robot policy learning.Source: §VI · p. 8
  • Fine-tuning on composite tasks performs relatively poorly, and machine-generated trajectories can contain jerky motions and collisions.The paper proposes improving policies and generated datasets, and automatically detecting and discarding trajectories with undesirable behaviors using simulation states.Source: §VI · p. 8
  • LLM-generated tasks still require human-written implementations; future work could use LLMs to propose and implement scenes and tasks with less guidance.The authors also plan to extend RoboCasa beyond kitchen environments and tasks.Source: §VI · p. 8
  • The dataset covers coarse manipulation but not highly dexterous, deformable, or bimanual tasks.Future work could combine RoboCasa simulation data with data from other simulators, internet videos, and real-robot datasets to build a robotics foundation model.Source: §VI · p. 8

VII. SIMULATOR

  • With rendering, simulator stepping averages 25.2 fps, near the 25 fps corresponding to each 0.04-second timestep.Across 10 randomly spawned PickPlaceCounterToCab episodes, average reset time is 9.50 seconds; without rendering, reset time is 9.46 seconds and stepping speed is 31.9 fps.Source: §VII · p. 12

A. Atomic Tasks · B. Composite Tasks

The section defines 25 atomic tasks across eight foundational robot skills and develops composite kitchen tasks from LLM suggestions through manual selection and filtering.

A. Atomic Tasks

  • The atomic suite contains 25 tasks representing eight foundational robot skills, from pick-and-place and opening doors or drawers to insertion and navigation.The other skills are twisting knobs, turning levers, and pressing buttons.Source: A. Atomic Tasks · p. 12
  • Task variants use language to disambiguate goals, such as specifying the object category, stove burner, or appliance target.The paper gives pick-and-place, stove-control, and navigation tasks as examples.Source: A. Atomic Tasks · p. 12

B. Composite Tasks

  • The authors prompted ChatGPT for 30 everyday kitchen activities, then manually selected 20 unique activities as composite-task candidates.Source: B. Composite Tasks · p. 12
  • Composite-task suggestions were elicited with prompts specifying robot skills, object categories, fixtures, simulation constraints, and example blueprints as few-shot guidance.The constraints included small objects that cannot be grasped and limited support for deformable manipulation.Source: B. Composite Tasks · p. 12
  • The authors filtered suggestions with logical flaws, including a blending-station task that assumes a nonexistent blender fixture.The example proposes placing dairy ingredients by a blender and pushing them toward it.Source: B. Composite Tasks · p. 12
  • They also rejected a wine-selection task that incorrectly uses a coffee-machine button press to simulate uncorking a bottle.Source: B. Composite Tasks · p. 12
  • Another rejected suggestion asks the robot to grasp utensils, which the example identifies as objects that should not be grasped.Source: B. Composite Tasks · p. 12

C. Datasets · IX. POLICY LEARNING IMPLEMENTATION

The datasets span human and machine-generated data for more than 25 atomic tasks, while the policy section describes a language-conditioned BC-Transformer and compares it with diffusion policy. The BC-Transformer performs better on PickPlaceCounterToSink in the reported comparison.

C. Datasets

  • Human and machine-generated datasets cover more than 25 atomic tasks.Source: C. Datasets · p. 12
  • Dataset images use randomly sampled AI-generated textures and the lightweight MuJoCo renderer; the public release will offer Omniverse rendering.Source: C. Datasets · p. 12

IX. POLICY LEARNING IMPLEMENTATION

  • BC-Transformer takes the past 10 observations and a language goal as input, predicts 10 actions, and replans after executing the first action.Language goals are encoded with a CLIP sentence encoder.Source: §IX · p. 12
  • Each observation encodes proprioception and images from three cameras using dedicated ResNet-18 stacks, fuses visual features with FiLM, and passes them to a 6-layer Transformer.Source: §IX · p. 12
  • The BC-Transformer has approximately 20M trainable parameters and is trained for 500k gradient steps at a learning rate of 1e−4 with warmup.Source: §IX · p. 13
  • 56% success for BC-Transformer versus 12% for Diffusion Policy on PickPlaceCounterToSink; the authors suggest its 10-observation history may help explain the difference.Diffusion Policy uses a 2-observation history, the default choice used by Chi et al.Source: §IX · p. 13

Related papers

Loading 2406.02523v1…