PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation

Abhiram Maddukuri, Zhenyu Jiang, Lawrence Yunliang Chen, Soroush Nasiriany, Yuqi Xie, Yu Fang, Wenqi Huang, Zu Wang, Zhenjia Xu, Nikita Chernyadev, Scott Reed, Ken Goldberg, Ajay Mandlekar, Linxi Fan, Yuke Zhu

arXiv:2503.24361v2cs.ROcs.AIcs.LG

TL;DR

Real-world robot data are expensive to scale, and simulation alone often requires substantial effort to bridge the reality gap. The paper systematically studies co-training policies on simulation and real-world datasets and derives a practical recipe. Across two robot domains and diverse tasks, simulation data improve real-world task performance by an average of 38%, including under notable dataset differences.

  • Problem

    The field lacks a systematic understanding of how to use simulation data effectively for real-robot learning while reducing reliance on costly real-world data collection.

    Source: §I · pp. 1–2

  • Method

    The paper conducts comprehensive experiments co-training vision-based manipulation policies on real-world data with task-aware or prior task-agnostic simulation data.

    Source: §I · p. 2, B. Prior Task-Agnostic Simulation Data · p. 5, C. Building Task-Aware Simulation… · p. 6

  • Results

    38% average policy-performance improvement across two domains is reported, with simulation benefits despite notable differences between simulation and real-world data.

    Source: §I · p. 2

  • Takeaways & Limitations

    Simulation data can complement real-world data and improve generalization to real-world scenarios not covered by demonstrations.

    Source: §VII · p. 9

    Most evaluated tasks are pick-and-place, while broader, longer-horizon, deformable-object, and liquid-manipulation tasks remain limited by simulation difficulty.

    Source: §VI · p. 9

Abstract

from arXiv · show

Large real-world robot datasets hold great potential to train generalist robot models, but scaling real-world human data collection is time-consuming and resource-intensive. Simulation has great potential in supplementing large-scale data, especially with recent advances in generative AI and automated data generation tools that enable scalable creation of robot behavior datasets. However, training a policy solely in simulation and transferring it to the real world often demands substantial human effort to bridge the reality gap. A compelling alternative is to co-train the policy on a mixture of simulation and real-world datasets. Preliminary studies have recently shown this strategy to substantially improve the performance of a policy over one trained on a limited amount of real-world data. Nonetheless, the community lacks a systematic understanding of sim-and-real co-training and what it takes to reap the benefits of simulation data for real-robot learning. This work presents a simple yet effective recipe for utilizing simulation data to solve vision-based robotic manipulation tasks. We derive this recipe from comprehensive experiments that validate the co-training strategy on various simulation and real-world datasets. Using two domains--a robot arm and a humanoid--across diverse tasks, we demonstrate that simulation data can enhance real-world task performance by an average of 38%, even with notable differences between the simulation and real-world data. Videos and additional results can be found at https://co-training.github.io/

I. INTRODUCTION

Real-world data are valuable for generalist robotic learning but costly to scale, while simulation offers scalable data at the cost of a reality gap. This work studies co-training on both sources and reports broad real-world gains across embodiments and tasks.

  • Scaling real-world robot data remains costly, time-consuming, and potentially insufficient for training generalist robot models.Source: §I · p. 1
  • Simulation can generate diverse robot trajectories with minimal human effort, but visual and physical mismatches create a reality gap.Source: §I · pp. 1–2
  • Co-training policies on simulation and real-world data can outperform training on real-world data alone without requiring the high alignment typically needed for sim-to-real transfer.Source: §I · p. 2
  • The paper develops a recipe through experiments spanning task-aware and task-agnostic simulation, a robot arm and humanoid, and diverse manipulation tasks.Source: §I · p. 2
  • 38% average policy-performance improvement is reported across two domains, with benefits even when simulation and real-world data differ substantially.Source: §I · p. 2

III. PROBLEM STATEMENT AND PRELIMINARIES

The paper formulates co-training as behavioral-cloning optimization over real and simulation demonstrations, weighted by a co-training ratio. It analyzes how dataset composition factors and alignment gaps shape transfer to real-world tasks.

Data composition factors

  • The study does not assume perfect simulation–reality alignment and instead examines which composition factors should align for successful co-training.Source: B. Data Composition Factors · p. 3

Data composition factors

  • Camera parameters include intrinsics and extrinsics, while dynamics parameters include friction, object mass, inertia, and controller variables.Source: B. Data Composition Factors · p. 3

C. Automated Synthetic Data Generation

The study uses automated synthetic data generation to expand simulation demonstrations, then evaluates sim-and-real co-training across two robot domains and diverse manipulation tasks.

C. Automated Synthetic Data Generation

  • Automated tools generate large, high-quality simulation datasets from dozens of source human demonstrations for policy co-training.MimicGen is used for single-arm tasks, while DexMimicGen supports bimanual and humanoid robots.Source: C. Automated Synthetic Data Generation · p. 3
  • The workflow starts from a real-world target task and prior simulation data, builds semantically similar digital cousins, consolidates datasets, and co-trains a policy.The policy is deployed directly to the real robot after training, with simulation sampled using ratio α.Source: Figure 2 · p. 4
  • The Panda Kitchen domain includes object-transfer and cabinet-door tasks, while Humanoid Tabletop includes cup placement, milk placement, and pouring.The study collects 50 demonstrations per Panda task and 20 per humanoid task.Source: A. Real-World Domains · p. 4
  • The experiments compare prior simulation, task-aware digital cousin, and real-world data across Kitchen Panda and Humanoid Tabletop domains.The domains include a Franka Panda kitchen setup and a Fourier GR-1 tabletop setup with multiple manipulation tasks.Source: A. Real-World Domains · p. 4, Figure 3 · p. 5

B. Prior Task-Agnostic Simulation Data

Prior task-agnostic simulation data offers a convenient co-training source despite substantial discrepancies from real-world tasks, while task-aware digital cousins preserve selected task elements without requiring perfect digital twins.

B. Prior Task-Agnostic Simulation Data

  • Prior task-agnostic datasets predate the downstream real-world task and are used out of the box without designing new simulation tasks or collecting new data.They generally use the same robot embodiment and action space while covering broad tasks and environments.Source: B. Prior Task-Agnostic Simulation Data · p. 5
  • RoboCasa provides 72k demonstrations across 24 tasks and 100 scenes for Panda Kitchen, with 3,000 generated demonstrations per task.Its semantically corresponding tasks still differ in robot states, controllers, physical parameters, object categories, and base position.Source: B. Prior Task-Agnostic Simulation Data · p. 5
  • Camera-pose re-rendering approximately aligns prior simulation demonstrations with real-world views, without further changes to the prior data.The alignment is imperfect but is reported to help significantly.Source: B. Prior Task-Agnostic Simulation Data · p. 5
  • The humanoid prior dataset contains 60k demonstrations across 20 tasks, including tasks with differing receptacles, objects, visual textures, distractors, and physical parameters.Ten RoboCasa tasks are synthesized with DexMimicGen at 1,000 demonstrations per task, and none is semantically equivalent to the real-world tasks.Source: B. Prior Task-Agnostic Simulation Data · pp. 5–6
  • A task-aware digital cousin preserves the same robot and action space, task goal, object categories, and environmental fixture categories as the real-world task.It is intentionally close to, but not perfectly aligned with, the real-world counterpart.Source: C. Building Task-Aware Simulation… · p. 6

D. Training and Evaluation Protocol

The study co-trains visuomotor policies on mixtures of real, prior simulation, and digital-cousin data, then evaluates them directly on six real-world tasks across two robot platforms.

D. Training and Evaluation Protocol

  • The evaluation compares Real, Prior, and task-aware digital cousin (DC) data mixtures using Diffusion Policy visuomotor policies.Policies take RGB images and proprioceptive information as input and produce action sequences.Source: D. Training and Evaluation Protocol · p. 6
  • The experiments assess in-domain real-world performance and generalization to novel scenarios while identifying factors important for effective co-training.The study spans six tasks across two robot platforms.Source: §V · p. 6, Table I · p. 7
  • 35.8% higher average success rate is achieved by Real and DC policies than by policies trained only on Real.The digital cousins share task definitions, similar scene setups, and comparable camera views without perfect alignment.Source: A. Effectiveness of Sim-and-Real… · p. 7
  • 31.5% average success-rate improvement is achieved by combining Real and Prior data, which outperforms Real-only training across all tasks.The prior data were generated without knowledge of the real-world tasks.Source: A. Effectiveness of Sim-and-Real… · p. 7
  • 37.9% average improvement over real-only policies is achieved by combining Real, DC, and Prior data.This combination generally performs best across the evaluated tasks.Source: A. Effectiveness of Sim-and-Real… · p. 7

B. Generalization Beyond Real Demonstrations

The paper tests whether simulation exposure improves real-world generalization beyond demonstrated objects and positions, then examines how data scale and co-training design affect performance. Co-training remains beneficial with more real data, but depends on sufficient simulation data, a tuned ratio, and camera alignment.

Generalization to novel objects

  • Simulation diversity improves robustness to novel objects, with co-trained policies reaching 50% and 80% success versus 33% and 10% for Real-only policies.The evaluations use new object categories, altered instances, differently colored cups, and novel objects.Source: B. Generalization Beyond Real… · p. 7

Generalization to novel positions

  • Co-training with digital-cousin simulation achieves twice the success rate of Real-only policies for unseen object positions on both humanoid and Panda experiments.Real demonstrations exclude center placements, while simulation includes uniformly distributed positions.Source: B. Generalization Beyond Real… · pp. 7–8

Real-data scaling

  • Co-training consistently outperforms Real-only training even as real demonstrations increase to 400, showing benefits in data-rich settings.Figure 4 fixes 4,000 digital-cousin demonstrations while varying real demonstrations from 40 to 400.Source: C. Effectiveness of Co-Training in… · p. 8, Figure 4 · p. 8

Simulation-data quantity

  • Reducing simulation demonstrations lowers success from 67% to 53% for Panda and from 95% to 75% for GR-1, indicating that sufficient simulation data is crucial.The tested reductions are 10k to 500 demonstrations for Panda and 1k to 100 for GR-1.Source: D. Key Elements of Effective Co-Training · p. 8

Co-training ratio

Camera alignment

E. A Simple Recipe for Sim-and-Real Co-Training

The paper recommends co-training real robot data with simulation chosen for task relevance and diversity, while matching task semantics and approximately aligning viewpoints. It also identifies sufficient simulation volume and careful ratio tuning as practical requirements, but notes that the evidence is concentrated on pick-and-place tasks.

Task and scene composition

  • Task-aware digital cousins produce the greatest gains, while large task-agnostic multi-task simulations still provide meaningful benefits despite composition differences.The comparison concerns task and scene composition relative to the real-world setting.Source: E. A Simple Recipe for Sim-and-Real… · p. 9

Object and initialization diversity

  • Diverse simulated objects and placements help real-world policies generalize to unseen scenarios.The recommendation links object composition and initialization diversity to broader real-world coverage.Source: E. A Simple Recipe for Sim-and-Real… · p. 9

Task and camera alignment

  • Simulation and real tasks should share definitions and success criteria, while similar camera viewpoints improve performance without requiring perfect alignment.The recommendation distinguishes semantic task alignment from approximate visual alignment.Source: E. A Simple Recipe for Sim-and-Real… · p. 9

Co-training hyperparameters

  • Practitioners should use a large simulation dataset, ideally orders of magnitude larger than the real dataset, and tune the co-training ratio.These are presented as co-training hyperparameter recommendations.Source: E. A Simple Recipe for Sim-and-Real… · p. 9

Scope and limitations

  • The approach is mainly evaluated on pick-and-place tasks, while broader manipulation and tasks involving deformable objects or liquids remain challenging.The authors identify high-precision, longer-horizon, deformable-object, and liquid tasks as boundaries for future work.Source: §VI · p. 9

Overall conclusion

  • Across the studied settings, large-scale simulation complements real-world data despite discrepancies and produces policies that outperform real-only training.The conclusion summarizes the paper’s systematic investigation of physics-simulation data for real-world vision-based manipulation.Source: §VII · p. 9

VIII. APPENDIX

The appendix organizes supplementary material covering datasets, experiments, training, evaluation, task setups, visual-realism methods, and author contributions. It also includes additional task-specific results and frequently asked questions.

Datasets

  • Dataset appendices describe task-agnostic simulations, task-aware digital cousins, and data-composition comparisons.These sections cover simulation environments, digital-cousin tuning, large-scale data generation, and composition analysis.Source: A. Overview · p. 12

Methods and evaluation

  • Training and evaluation appendices provide the algorithm, model architecture, protocols, and policy-evaluation procedures.Generalization experiment details are also included.Source: A. Overview · p. 12

Task documentation

  • The appendix documents task setups, including MultiTaskPnP and real-world domains and tasks.These materials are listed under Appendices VIII-J and VIII-C.Source: A. Overview · p. 12

Additional experiments

  • Additional experiments address visual realism with Vid2Vid and performance on the real-world CloseDoor task.These are listed as Appendices VIII-K and VIII-L.Source: A. Overview · p. 12

Supplementary information

  • The appendix also contains frequently asked questions and author contributions, including project leads, experiments, infrastructure, digital release, and technical advice.The listed contribution categories identify roles across the project.Source: A. Overview · p. 12, B. Author Contributions • · p. 12

C. Real-World Tasks

The study uses Panda and Fourier GR-1 platforms with teleoperated real-world data and simulated datasets spanning kitchen and tabletop manipulation tasks. Object placements and robot configurations are varied according to each platform’s collection setup.

Panda Kitchen

  • Panda Kitchen uses a modified DROID setup with two side-view cameras and an eye-in-hand camera, collecting teleoperated demonstrations with a SpaceMouse.The Panda starts from a fixed initial position and samples task-specific object placements uniformly.Source: C. Real-World Tasks · p. 12

Humanoid Tabletop

  • The GR-1 uses dexterous hands, locked lower body and waist joints, and glove-and-tracker teleoperation with an inverse-kinematic controller.The setup captures finger and wrist poses during humanoid data collection.Source: C. Real-World Tasks · p. 12

Humanoid real-world data

  • For three humanoid single tasks, the study collects 20 real demonstrations per task with randomized initial robot poses and object positions.The tasks are CupPnP, MilkPnP, and Pouring, each using one object set.Source: C. Real-World Tasks · p. 12

Humanoid multi-task data

  • The humanoid multi-task setting includes four tasks and multiple object instances during data collection.The tasks are Cuttingboard2Basket, Cuttingboard2Pan, Mat2Basket, and Plate2Bowl.Source: C. Real-World Tasks · p. 12

Panda simulation data

  • The Panda simulation dataset uses 60k RoboCasa trajectories across 20 tasks after excluding four tasks because camera alignment reduced visibility.The excluded tasks are OpenDrawer, CloseDrawer, TurnOnStove, and TurnOffStove.Source: D. Task-Agnostic Simulation Datasets · p. 12

Simulation task coverage

  • The simulated task inventory includes object pick-and-place variants such as counter-to-cabinet, cabinet-to-counter, counter-to-sink, and sink-to-counter.The appendix also lists microwave, stove, and coffee-serving tasks.Source: D. Task-Agnostic Simulation Datasets · pp. 12–13

Humanoid simulation data

  • The humanoid tabletop simulation datasets contain ten RoboCasa tasks, including object placement into plates, bowls, and other tabletop targets.The listed examples include apple-to-plate, can-to-bowl, mug-to-plate, fruit placement, and kettle-to-plate.Source: D. Task-Agnostic Simulation Datasets · p. 13

E. Task-Aware Digital Cousin Datasets

The paper constructs task-aware digital cousins for Panda Kitchen and Humanoid Tabletop tasks by approximately aligning task, object, and initialization factors with real-world data.

Dataset design

Task composition

Object composition

Initialization distribution

Humanoid Tabletop

  • Humanoid digital cousins use matching object categories or smaller fixed object sets, aligned initialization regions, and rendered tabletop textures.For CupPnP, MilkPnP, and Pouring, 10 human demonstrations yield 1000 synthetic demonstrations per task.Source: E. Task-Aware Digital Cousin Datasets · p. 14

G. Training Details

Training uses vision-based diffusion policies with domain-specific conditioning and explicit sampling weights for simulation and real-world data, followed by success-rate evaluation across task-specific settings.

Policy training

  • The Panda policy uses three image views, proprioception, language conditioning, and 7-DoF delta end-effector and gripper actions.Source: G. Training Details · p. 14
  • The sampling rule assigns probability α to simulation observations and 1−α to real-world observations within each training batch.Source: G. Training Details · p. 14
  • Humanoid training uses task-specific diffusion policies with vision encoders and joint-position actions; α values of 0.9 and 0.99 are optimal, with 0.99 used thereafter.Source: G. Training Details · p. 14

Evaluation

  • Evaluation selects the highest success rate across three equally spaced checkpoints and uses randomized task-specific placements or joint angles.Source: H. Policy Evaluation · pp. 14–15
  • Standard evaluation uses seen objects and training-range initial conditions, while generalization tests vary objects or object initialization positions.Source: H. Policy Evaluation · p. 15, §I · p. 15

J. MultiTaskPnP Task Setup

MultiTaskPnP evaluates a single humanoid policy across four pick-and-place tasks using task-aware simulation data, and co-training consistently improves performance over real-only training.

Task setup

Task-aware simulation

Evaluation

  • A single policy implicitly identifies the task from image observations and is evaluated on three unseen objects across three initial positions per task.This produces 36 evaluations across the four tasks.Source: J. MultiTaskPnP Task Setup · p. 16

Results

Visual realism

L. Training CloseDoor with More Demos

The paper tests whether more real demonstrations close the CloseDoor performance gap and reports additional evidence on dynamics alignment, camera mismatch, and bimanual manipulation.

CloseDoor

Alignment analysis

  • Dynamic alignment produces no success-rate difference on GR-1 CupPnP, which remains 95% with or without alignment.Source: M. FAQs · p. 17
  • Reported camera deltas include 37 cm and 20° for Panda third-person cameras, 9 cm and 180° for its wrist camera, and 36 cm in GR-1 position.Source: M. FAQs · p. 17

Co-training comparison

  • Domain randomization and domain adaptation are complementary but not strictly necessary for co-training gains.Source: M. FAQs · p. 17

Bimanual manipulation

  • BimanualPnP success increases from 15% with 50 real demonstrations to 50% when co-trained with 1,000 digital-cousin demonstrations.Source: M. FAQs · p. 17

Related papers

Loading 2503.24361v2…