PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation

Abhiram Maddukuri, Zhenyu Jiang, Lawrence Yunliang Chen, Soroush Nasiriany, Yuqi Xie, Yu Fang, Wenqi Huang, Zu Wang, Zhenjia Xu, Nikita Chernyadev, Scott Reed, Ken Goldberg, Ajay Mandlekar, Linxi Fan, Yuke Zhu

arXiv:2503.24361v2cs.ROcs.AIcs.LG

TL;DR

Real-world robot data are expensive to scale, and simulation alone often requires substantial effort to bridge the reality gap. The paper systematically studies co-training policies on simulation and real-world datasets and derives a practical recipe. Across two robot domains and diverse tasks, simulation data improve real-world task performance by an average of 38%, including under notable dataset differences.

  • Problem

    The field lacks a systematic understanding of how to use simulation data effectively for real-robot learning while reducing reliance on costly real-world data collection.

  • Method

    The paper conducts comprehensive experiments co-training vision-based manipulation policies on real-world data with task-aware or prior task-agnostic simulation data.

  • Results

    38% average policy-performance improvement across two domains is reported, with simulation benefits despite notable differences between simulation and real-world data.

  • Takeaways & Limitations

    Simulation data can complement real-world data and improve generalization to real-world scenarios not covered by demonstrations.

  • Takeaways & Limitations

    Most evaluated tasks are pick-and-place, while broader, longer-horizon, deformable-object, and liquid-manipulation tasks remain limited by simulation difficulty.

Abstract

from arXiv · show

Large real-world robot datasets hold great potential to train generalist robot models, but scaling real-world human data collection is time-consuming and resource-intensive. Simulation has great potential in supplementing large-scale data, especially with recent advances in generative AI and automated data generation tools that enable scalable creation of robot behavior datasets. However, training a policy solely in simulation and transferring it to the real world often demands substantial human effort to bridge the reality gap. A compelling alternative is to co-train the policy on a mixture of simulation and real-world datasets. Preliminary studies have recently shown this strategy to substantially improve the performance of a policy over one trained on a limited amount of real-world data. Nonetheless, the community lacks a systematic understanding of sim-and-real co-training and what it takes to reap the benefits of simulation data for real-robot learning. This work presents a simple yet effective recipe for utilizing simulation data to solve vision-based robotic manipulation tasks. We derive this recipe from comprehensive experiments that validate the co-training strategy on various simulation and real-world datasets. Using two domains--a robot arm and a humanoid--across diverse tasks, we demonstrate that simulation data can enhance real-world task performance by an average of 38%, even with notable differences between the simulation and real-world data. Videos and additional results can be found at https://co-training.github.io/

I. INTRODUCTION

Real-world data are valuable for generalist robotic learning but costly to scale, while simulation offers scalable data at the cost of a reality gap. This work studies co-training on both sources and reports broad real-world gains across embodiments and tasks.

  • Scaling real-world robot data remains costly, time-consuming, and potentially insufficient for training generalist robot models.
  • Simulation can generate diverse robot trajectories with minimal human effort, but visual and physical mismatches create a reality gap.
  • Co-training policies on simulation and real-world data can outperform training on real-world data alone without requiring the high alignment typically needed for sim-to-real transfer.
  • The paper develops a recipe through experiments spanning task-aware and task-agnostic simulation, a robot arm and humanoid, and diverse manipulation tasks.
  • 38% average policy-performance improvement is reported across two domains, with benefits even when simulation and real-world data differ substantially.

III. PROBLEM STATEMENT AND PRELIMINARIES

The paper formulates co-training as behavioral-cloning optimization over real and simulation demonstrations, weighted by a co-training ratio. It analyzes how dataset composition factors and alignment gaps shape transfer to real-world tasks.

  • The policy is trained on real demonstrations Dreal and simulation demonstrations Dsim, typically with many more simulation trajectories than real trajectories.
  • The total objective combines simulation and real-world behavioral-cloning losses, with α controlling their relative contribution during training.
  • Data composition factors: Simulation and real-world datasets may differ in task, scene, object, initialization, camera, and dynamics composition.
  • The study does not assume perfect simulation–reality alignment and instead examines which composition factors should align for successful co-training.
  • Data composition factors: Camera parameters include intrinsics and extrinsics, while dynamics parameters include friction, object mass, inertia, and controller variables.

C. Automated Synthetic Data Generation

The study uses automated synthetic data generation to expand simulation demonstrations, then evaluates sim-and-real co-training across two robot domains and diverse manipulation tasks.

  • C. Automated Synthetic Data Generation: Automated tools generate large, high-quality simulation datasets from dozens of source human demonstrations for policy co-training.MimicGen is used for single-arm tasks, while DexMimicGen supports bimanual and humanoid robots.
  • C. Automated Synthetic Data Generation: The workflow starts from a real-world target task and prior simulation data, builds semantically similar digital cousins, consolidates datasets, and co-trains a policy.The policy is deployed directly to the real robot after training, with simulation sampled using ratio α.
  • C. Automated Synthetic Data Generation: The Panda Kitchen domain includes object-transfer and cabinet-door tasks, while Humanoid Tabletop includes cup placement, milk placement, and pouring.The study collects 50 demonstrations per Panda task and 20 per humanoid task.
  • C. Automated Synthetic Data Generation: The experiments compare prior simulation, task-aware digital cousin, and real-world data across Kitchen Panda and Humanoid Tabletop domains.The domains include a Franka Panda kitchen setup and a Fourier GR-1 tabletop setup with multiple manipulation tasks.

B. Prior Task-Agnostic Simulation Data

Prior task-agnostic simulation data offers a convenient co-training source despite substantial discrepancies from real-world tasks, while task-aware digital cousins preserve selected task elements without requiring perfect digital twins.

  • B. Prior Task-Agnostic Simulation Data: Prior task-agnostic datasets predate the downstream real-world task and are used out of the box without designing new simulation tasks or collecting new data.They generally use the same robot embodiment and action space while covering broad tasks and environments.
  • B. Prior Task-Agnostic Simulation Data: RoboCasa provides 72k demonstrations across 24 tasks and 100 scenes for Panda Kitchen, with 3,000 generated demonstrations per task.Its semantically corresponding tasks still differ in robot states, controllers, physical parameters, object categories, and base position.
  • B. Prior Task-Agnostic Simulation Data: Camera-pose re-rendering approximately aligns prior simulation demonstrations with real-world views, without further changes to the prior data.The alignment is imperfect but is reported to help significantly.
  • B. Prior Task-Agnostic Simulation Data: The humanoid prior dataset contains 60k demonstrations across 20 tasks, including tasks with differing receptacles, objects, visual textures, distractors, and physical parameters.Ten RoboCasa tasks are synthesized with DexMimicGen at 1,000 demonstrations per task, and none is semantically equivalent to the real-world tasks.
  • B. Prior Task-Agnostic Simulation Data: A task-aware digital cousin preserves the same robot and action space, task goal, object categories, and environmental fixture categories as the real-world task.It is intentionally close to, but not perfectly aligned with, the real-world counterpart.

D. Training and Evaluation Protocol

The study co-trains visuomotor policies on mixtures of real, prior simulation, and digital-cousin data, then evaluates them directly on six real-world tasks across two robot platforms.

  • D. Training and Evaluation Protocol: The evaluation compares Real, Prior, and task-aware digital cousin (DC) data mixtures using Diffusion Policy visuomotor policies.Policies take RGB images and proprioceptive information as input and produce action sequences.
  • D. Training and Evaluation Protocol: The experiments assess in-domain real-world performance and generalization to novel scenarios while identifying factors important for effective co-training.The study spans six tasks across two robot platforms.
  • D. Training and Evaluation Protocol: 35.8% higher average success rate is achieved by Real and DC policies than by policies trained only on Real.The digital cousins share task definitions, similar scene setups, and comparable camera views without perfect alignment.
  • D. Training and Evaluation Protocol: 31.5% average success-rate improvement is achieved by combining Real and Prior data, which outperforms Real-only training across all tasks.The prior data were generated without knowledge of the real-world tasks.
  • D. Training and Evaluation Protocol: 37.9% average improvement over real-only policies is achieved by combining Real, DC, and Prior data.This combination generally performs best across the evaluated tasks.

B. Generalization Beyond Real Demonstrations

The paper tests whether simulation exposure improves real-world generalization beyond demonstrated objects and positions, then examines how data scale and co-training design affect performance. Co-training remains beneficial with more real data, but depends on sufficient simulation data, a tuned ratio, and camera alignment.

  • Generalization to novel objects: Simulation diversity improves robustness to novel objects, with co-trained policies reaching 50% and 80% success versus 33% and 10% for Real-only policies.The evaluations use new object categories, altered instances, differently colored cups, and novel objects.
  • Generalization to novel positions: Co-training with digital-cousin simulation achieves twice the success rate of Real-only policies for unseen object positions on both humanoid and Panda experiments.Real demonstrations exclude center placements, while simulation includes uniformly distributed positions.
  • Real-data scaling: Co-training consistently outperforms Real-only training even as real demonstrations increase to 400, showing benefits in data-rich settings.Figure 4 fixes 4,000 digital-cousin demonstrations while varying real demonstrations from 40 to 400.
  • Simulation-data quantity: Reducing simulation demonstrations lowers success from 67% to 53% for Panda and from 95% to 75% for GR-1, indicating that sufficient simulation data is crucial.The tested reductions are 10k to 500 demonstrations for Panda and 1k to 100 for GR-1.
  • Co-training ratio: A 99% simulation sampling ratio performs best, whereas increasing it to 99.5% or 99.9% reduces success from 95% to 60%.The study evaluates the CupPnP task with 20 real and 1,000 simulation demonstrations.
  • Camera alignment: Camera misalignment reduces co-training success from 67% to 56% on Panda and from 95% to 70% on GR-1, although perfect alignment is unnecessary.The comparison uses default unaligned versus approximately aligned digital-cousin camera views.

E. A Simple Recipe for Sim-and-Real Co-Training

The paper recommends co-training real robot data with simulation chosen for task relevance and diversity, while matching task semantics and approximately aligning viewpoints. It also identifies sufficient simulation volume and careful ratio tuning as practical requirements, but notes that the evidence is concentrated on pick-and-place tasks.

  • Task and scene composition: Task-aware digital cousins produce the greatest gains, while large task-agnostic multi-task simulations still provide meaningful benefits despite composition differences.The comparison concerns task and scene composition relative to the real-world setting.
  • Object and initialization diversity: Diverse simulated objects and placements help real-world policies generalize to unseen scenarios.The recommendation links object composition and initialization diversity to broader real-world coverage.
  • Task and camera alignment: Simulation and real tasks should share definitions and success criteria, while similar camera viewpoints improve performance without requiring perfect alignment.The recommendation distinguishes semantic task alignment from approximate visual alignment.
  • Co-training hyperparameters: Practitioners should use a large simulation dataset, ideally orders of magnitude larger than the real dataset, and tune the co-training ratio.These are presented as co-training hyperparameter recommendations.
  • Scope and limitations: The approach is mainly evaluated on pick-and-place tasks, while broader manipulation and tasks involving deformable objects or liquids remain challenging.The authors identify high-precision, longer-horizon, deformable-object, and liquid tasks as boundaries for future work.
  • Overall conclusion: Across the studied settings, large-scale simulation complements real-world data despite discrepancies and produces policies that outperform real-only training.The conclusion summarizes the paper’s systematic investigation of physics-simulation data for real-world vision-based manipulation.

VIII. APPENDIX

The appendix organizes supplementary material covering datasets, experiments, training, evaluation, task setups, visual-realism methods, and author contributions. It also includes additional task-specific results and frequently asked questions.

  • Datasets: Dataset appendices describe task-agnostic simulations, task-aware digital cousins, and data-composition comparisons.These sections cover simulation environments, digital-cousin tuning, large-scale data generation, and composition analysis.
  • Methods and evaluation: Training and evaluation appendices provide the algorithm, model architecture, protocols, and policy-evaluation procedures.Generalization experiment details are also included.
  • Task documentation: The appendix documents task setups, including MultiTaskPnP and real-world domains and tasks.These materials are listed under Appendices VIII-J and VIII-C.
  • Additional experiments: Additional experiments address visual realism with Vid2Vid and performance on the real-world CloseDoor task.These are listed as Appendices VIII-K and VIII-L.
  • Supplementary information: The appendix also contains frequently asked questions and author contributions, including project leads, experiments, infrastructure, digital release, and technical advice.The listed contribution categories identify roles across the project.

C. Real-World Tasks

The study uses Panda and Fourier GR-1 platforms with teleoperated real-world data and simulated datasets spanning kitchen and tabletop manipulation tasks. Object placements and robot configurations are varied according to each platform’s collection setup.

  • Panda Kitchen: Panda Kitchen uses a modified DROID setup with two side-view cameras and an eye-in-hand camera, collecting teleoperated demonstrations with a SpaceMouse.The Panda starts from a fixed initial position and samples task-specific object placements uniformly.
  • Humanoid Tabletop: The GR-1 uses dexterous hands, locked lower body and waist joints, and glove-and-tracker teleoperation with an inverse-kinematic controller.The setup captures finger and wrist poses during humanoid data collection.
  • Humanoid real-world data: For three humanoid single tasks, the study collects 20 real demonstrations per task with randomized initial robot poses and object positions.The tasks are CupPnP, MilkPnP, and Pouring, each using one object set.
  • Humanoid multi-task data: The humanoid multi-task setting includes four tasks and multiple object instances during data collection.The tasks are Cuttingboard2Basket, Cuttingboard2Pan, Mat2Basket, and Plate2Bowl.
  • Panda simulation data: The Panda simulation dataset uses 60k RoboCasa trajectories across 20 tasks after excluding four tasks because camera alignment reduced visibility.The excluded tasks are OpenDrawer, CloseDrawer, TurnOnStove, and TurnOffStove.
  • Simulation task coverage: The simulated task inventory includes object pick-and-place variants such as counter-to-cabinet, cabinet-to-counter, counter-to-sink, and sink-to-counter.The appendix also lists microwave, stove, and coffee-serving tasks.
  • Humanoid simulation data: The humanoid tabletop simulation datasets contain ten RoboCasa tasks, including object placement into plates, bowls, and other tabletop targets.The listed examples include apple-to-plate, can-to-bowl, mug-to-plate, fruit placement, and kettle-to-plate.

E. Task-Aware Digital Cousin Datasets

The paper constructs task-aware digital cousins for Panda Kitchen and Humanoid Tabletop tasks by approximately aligning task, object, and initialization factors with real-world data.

  • Dataset design: Digital cousins align task composition, object composition, and initialization distribution with corresponding real-world settings.The comparisons cover Panda Kitchen and Humanoid Tabletop domains.
  • Task composition: Task composition matches task semantics, language instructions, motions, and approximately matched initial robot configurations and positions.
  • Object composition: Panda digital cousins share 9/10 or 8/10 object categories with real-world CounterToSinkPnP and CounterToCabPnP tasks.
  • Initialization distribution: Initialization alignment adjusts robot starting states and object sampling regions to better match real-world conditions.
  • Humanoid Tabletop: Humanoid digital cousins use matching object categories or smaller fixed object sets, aligned initialization regions, and rendered tabletop textures.For CupPnP, MilkPnP, and Pouring, 10 human demonstrations yield 1000 synthetic demonstrations per task.

G. Training Details

Training uses vision-based diffusion policies with domain-specific conditioning and explicit sampling weights for simulation and real-world data, followed by success-rate evaluation across task-specific settings.

  • Policy training: The Panda policy uses three image views, proprioception, language conditioning, and 7-DoF delta end-effector and gripper actions.
  • Policy training: The sampling rule assigns probability α to simulation observations and 1−α to real-world observations within each training batch.
  • Policy training: Humanoid training uses task-specific diffusion policies with vision encoders and joint-position actions; α values of 0.9 and 0.99 are optimal, with 0.99 used thereafter.
  • Evaluation: Evaluation selects the highest success rate across three equally spaced checkpoints and uses randomized task-specific placements or joint angles.
  • Evaluation: Standard evaluation uses seen objects and training-range initial conditions, while generalization tests vary objects or object initialization positions.

J. MultiTaskPnP Task Setup

MultiTaskPnP evaluates a single humanoid policy across four pick-and-place tasks using task-aware simulation data, and co-training consistently improves performance over real-only training.

  • Task setup: The multi-task setup contains four tasks transferring objects from cutting boards, mats, or plates into baskets, pans, or bowls.
  • Task-aware simulation: Each task uses same-category simulated objects and approximately aligned initial object-position distributions.
  • Task-aware simulation: The digital-cousin datasets contain 1,000 generated demonstrations per task, totaling 4,000 demonstrations.
  • Evaluation: A single policy implicitly identifies the task from image observations and is evaluated on three unseen objects across three initial positions per task.This produces 36 evaluations across the four tasks.
  • Results: Co-trained policies consistently outperform real-only policies across tested quantities of real-world demonstrations.
  • Results: 75.7% success is achieved with Real + DC + Prior, compared with 70.8% for Real + DC, 68.8% for Real + Prior, and 30.6% for Real.
  • Visual realism: The visual-realism pipeline fine-tunes a video diffusion model and initializes diffusion from noised simulation videos to preserve object positions.
  • Visual realism: Vid2Vid-enhanced simulation improves policy performance by 5–10% on average, especially when simulation or real-world trajectories are scarce.Its benefit diminishes when sufficient real-world or large diverse simulation datasets are available.

L. Training CloseDoor with More Demos

The paper tests whether more real demonstrations close the CloseDoor performance gap and reports additional evidence on dynamics alignment, camera mismatch, and bimanual manipulation.

  • CloseDoor: Doubling CloseDoor real demonstrations to 100 raises success only to 80%, rather than achieving 100% success.
  • Alignment analysis: Dynamic alignment produces no success-rate difference on GR-1 CupPnP, which remains 95% with or without alignment.
  • Alignment analysis: Reported camera deltas include 37 cm and 20° for Panda third-person cameras, 9 cm and 180° for its wrist camera, and 36 cm in GR-1 position.
  • Co-training comparison: Domain randomization and domain adaptation are complementary but not strictly necessary for co-training gains.
  • Bimanual manipulation: BimanualPnP success increases from 15% with 50 real demonstrations to 50% when co-trained with 1,000 digital-cousin demonstrations.
Loading 2503.24361v2…