PaperlayerView exact revision on arXiv

Source-linked AI summary

Getting Out and Getting Back: World and Behavior Grounding in Real2Sim2Real Co-Training

Samuel Liu, Youngsun Kim, Martin Matak, Gilwoo Lee

arXiv:2610.00821v1cs.RO

TL;DR

Simulation can expand scarce real demonstrations, but the separate effects of world fidelity and human-like simulated behavior on co-training remain unclear. The paper varies both grounding axes independently and finds complementary roles, with fully grounded co-training improving success and grounded simulation benefiting foundation-model post-training.

  • Problem

    The separate effects of world fidelity and human-like simulated trajectories on co-trained policy performance remain unclear, while real demonstrations are costly and can be inconsistent.

    Source: §1 · p. 1

  • Method

    A real2sim2real pipeline independently varies world grounding and behavior grounding to co-train policies using real demonstrations and simulated data.

    Source: §1 · p. 2

  • Results

    Fully grounded co-training raised success, and world grounding tended to contribute more than behavior grounding; with a foundation model, 100 real demonstrations plus grounded simulation matched fine-tuning on 1,000 real demonstrations.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Co-trained policies follow real-derived behavior in states covered by demonstrations and rely on simulated behavior elsewhere, where its real-world success depends on world fidelity.

    Source: §4.5 · p. 5

    The study covers one task with limited trials per condition, and its grounded world varies visual and dynamic fidelity together, leaving their individual contributions unseparated.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

Simulation can expand scarce real demonstrations for co-training, yet how world fidelity and similarity to human behavior affect policy performance remains unclear. We distinguish world grounding, which aligns simulation with the real system, and behavior grounding, which aligns simulated trajectories with human motion. We build a real2sim2real pipeline that varies these axes independently to generate data for co-training. On a dynamic dexterous pick-and-sort task, fully grounded co-training raises success from 52% to 86%; averaged across configurations, world grounding improves success by 18 percentage points and behavior grounding by 10. Deployed policies behave like a mixture of real-derived and simulation-derived policies, imitating real demonstrations in covered states and relying on simulated behavior elsewhere, which we examine through latent-space analysis. Together, these results suggest complementary roles: world grounding lets policies use simulated experience beyond real-data coverage, while behavior grounding matters mainly when world grounding is imperfect. Grounded simulation remains beneficial when co-training foundation models.

1 Introduction

The paper separates world grounding from behavior grounding to study their unresolved individual and combined effects on co-training. It proposes independently varying both axes and examines how co-trained policies use real and simulated behavior.

1 Introduction

  • Real demonstrations are costly to collect and can be inconsistent, while simulation can generate diverse, success-filtered data for co-training.Domain randomization adds diversity, and success filtering preserves quality.Source: §1 · p. 1
  • The individual and combined contributions of world fidelity and human-like simulated behavior to policy performance remain unclear.World grounding concerns differences in rendering, physics, and robot control; behavior grounding concerns differences between simulated trajectories and human demonstrations.Source: §1 · p. 1
  • The authors hypothesize that world grounding determines whether simulated actions transfer, while behavior grounding offers a fallback when world grounding is imperfect.Source: §1 · p. 1
  • The pipeline independently varies both grounding axes in a dynamic dexterous pick-and-sort task, co-training policies with real demonstrations and simulated episodes.Policies are evaluated in real-world rollouts under nominal and shifted conditions.Source: §1 · p. 2
  • The contributions include within-rollout switching between real-derived and simulated behavior and benefits from grounded simulation when post-training foundation models.Policies may rely on simulation-derived behavior in states not covered by real demonstrations.Source: §1 · p. 2

2 Related Work

Prior co-training work finds benefits from combining scarce real demonstrations with larger simulated datasets, while simulation mismatch and the value of human-like trajectories vary across settings.

2 Related Work

  • Co-training combines scarce real demonstrations with larger simulated datasets and can improve generalization to conditions absent from real data.Prior findings report greater sensitivity to physics gaps than visual gaps in some settings, with small visual differences sometimes improving performance [10].Source: §2 · p. 2
  • The importance of world-grounding components varies across settings, with prior results differing on the relative roles of physical and rendering fidelity [10] [18].Source: §2 · pp. 2–3
  • Synthetic trajectories can come from adapted demonstrations, reinforcement learning, or motion planning, but whether human-like motion independently improves co-training remains unclear.The paper compares demonstration-derived and machine-generated trajectories in grounded and ungrounded worlds.Source: §2 · p. 3

3 Method

The method generates four simulated datasets by varying world and behavior grounding independently, then combines them with real demonstrations for policy training.

3.1 Data Generation Pipeline

  • Each of four grounding configurations produces about 1,500 success-filtered trajectories spanning the same objects and belt-speed range for co-training.Trajectories are generated in MuJoCo Warp for physics and Isaac Sim for rendering, then combined with real demonstrations to train policies from scratch or fine-tune foundation models.Source: §3.1 · p. 3

3.2 World and Behavior Grounding

  • Grounded worlds align simulation through visual calibration, scene reconstruction, and robot system identification; ungrounded worlds use ideal cameras and nominal system settings.Both use Isaac Sim’s native RTX renderer, so the configurations differ in calibration and identification rather than rendering method.Source: §3.2 · p. 3
  • Grounded behavior adapts teleoperated demonstrations to new object poses, sizes, and belt speeds, whereas ungrounded behavior uses synthesized grasps and planned motions.The ungrounded pipeline refines proposed grasps and plans collision-checked routes through approach, interception, carry, and release.Source: §3.2 · p. 3

4 Experiments and Results

On a dynamic dexterous pick-and-sort task, co-training with grounded simulation improved success, and world grounding contributed more on average than behavior grounding. Qualitative rollouts and latent-space analysis examined how policies use simulated behavior and when better world grounding may reduce the need for behavior grounding.

4.1 Task Setup

  • Each policy was evaluated in 50 real-world trials, including nominal conditions and shifts absent from the demonstrations but present in simulation.The setup used two Flexiv Rizon 4S arms with Wuji Hand 2 hands and a moving conveyor belt.Source: §4.1 · p. 3

4.2 Real-Only vs. Co-Training

  • 52% to 86%: co-training with fully grounded simulation raised task success over training on 100 real demonstrations alone.Figure 2 covers flow-matching policies and foundation models post-trained with sim-real mixture data.Source: Figure 2 · p. 4, §4.2 · p. 4

4.3 Effect of World and Behavior Grounding

  • 18 percentage points for world grounding and 10 percentage points for behavior grounding: average improvements in overall success across the other axis.World grounding’s gains were greater in conditions the real data does not cover.Source: §4.3 · p. 4

4.4 Grounding for Foundation Models

  • 90%: our foundation model reached this success with 100 real demonstrations and 1,500 fully grounded simulated trajectories, at least matching 1,000 real demonstrations.Grounding improved success by 10 points over fully ungrounded data for the foundation model, versus 28 points for the policy trained from scratch.Source: §4.4 · p. 4

4.5 Qualitative Observations and Discussion

  • Co-trained policies followed real-derived behavior in covered states and relied on simulation-derived behavior elsewhere, including a simulated-style recovery after a missed pick.Post-grasp failures to release or place the object occurred mainly with policies trained on world- and behavior-ungrounded data; performance beyond real-data coverage depended on world match.Source: §4.5 · pp. 4–5

4.6 Latent-Space Analysis

  • 17/24: after visual transfer, both behavior-grounded and behavior-ungrounded policies succeeded in this many trials, on par with the real-only policy at 16/24.A diagnostic policy trained with 10 real demonstrations entered the simulated-data region and adopted a two-finger pinch; the latent-space analysis is qualitative and based on selected rollouts.Source: §4.6 · pp. 5–6, Figure 3 · p. 5

5 Conclusion

The paper varies world and behavior grounding independently in real2sim2real co-training for dynamic dexterous pick-and-sort. It reports complementary roles for the two axes, describes policies switching between real- and simulation-derived behavior, and identifies limits to the study.

5 Conclusion

  • Fully grounded co-training raised success, and world grounding tended to contribute more than behavior grounding.With a pretrained foundation model, 100 real demonstrations plus grounded simulation matched fine-tuning on 1,000 real demonstrations.Source: §5 · p. 6
  • Co-trained policies follow real-derived behavior where demonstrations cover the state and switch to simulation-derived behavior elsewhere.The latent-space analysis illustrates rather than establishes this proposed mechanism.Source: §5 · p. 6, §5.1 · p. 6

5.1 Limitations

  • The study covers one task and limited trials per condition, with many differences falling within overlapping confidence intervals.Visual and dynamic fidelity vary together, so their individual contributions remain unseparated.Source: §5.1 · p. 6
  • The proposed mechanism remains illustrative; directly attributing actions to real or simulated data and testing longer-horizon tasks are future directions.The authors suggest returning to demonstrated behavior may matter more in longer-horizon tasks.Source: §5.1 · p. 6

Appendix A World Grounding Details

Appendix A describes world configurations and validation checks for the simulated scene and robot. Shared controls and residual fidelity differences qualify what the grounding comparisons represent.

Appendix A World Grounding Details

  • Table 1 presents world configurations, with all other listed controls shared.The shared set includes object geometry and mass, contact parameters, appearance randomization, initial poses, belt speed, head intrinsics, hand/TCP registration, and left-arm control.Source: Table 1 · p. 9, Appendix A · p. 9
  • Contact settings are effective simulation parameters, not measured material constants, while belt height varies within the same 8 mm range around each world’s layout.Source: Appendix A · p. 9
  • Joint calibration from 14 checkerboard images yields 0.85 px leave-one-out reprojection error, versus 6.44 px for the CAD mount with fitted intrinsics.The grounded and ungrounded wrist focal lengths are 252 and 229 px; head placement differs by 30.1 mm and 1.7°.Source: Appendix A · p. 9
  • External RGB-D reconstruction uses SAM 2 and FoundationPose, accepting held-out object poses at mask IoU ≥0.6 after stable-contact correction.The ungrounded belt is displaced horizontally by 24/18 mm and is 14 mm lower.Source: Appendix A · p. 9
  • The surrogate arm controller has 1.65 mm/0.38° TCP position/orientation RMS error on a held-out 15 s recording, while the 79 s fitting recording shows residual drift.The hand model uses 12 training and 6 validation episodes, with median fingertip RMS error 0.64 mm.Source: Appendix A · p. 9

Appendix B Behavior Grounding Details

Appendix B describes two ways to generate behavior-grounded and ungrounded simulated trajectories. Both retain successful physics-based pickups and placements to produce about 1,500 trajectories per world.

Appendix B Behavior Grounding Details

  • MimicGen transforms demonstration phases to new object poses, sizes, and belt speeds, then executes transformed commands through the simulated controller.Source: Appendix B · p. 9
  • Ungrounded trajectories use GraspGen-X proposals adapted to hand closure geometry, surface refinement, and collision-checked inverse-kinematics routes.Routes cover approach, interception, carry, and release.Source: Appendix B · p. 9
  • Both conditions retain only successful physics-based pickups and placements into the assigned bin, yielding about 1,500 trajectories per world.The trajectories cover the same objects and belt-speed range.Source: Appendix B · p. 9

Appendix C Policy Models

Appendix C specifies the policy inputs and model components, distinguishes a task-trained 500M model from a pretrained 2B foundation model, and reports approximate inference latency.

Appendix C Policy Models

  • Both models use three 256 × 256 RGB views, 20 proprioceptive samples at 50 Hz, a text instruction, and an embodiment identifier.They share a frozen DINOv3 ViT-B/16 image backbone with rank-8 adapters in its final four blocks, a frozen CLIP text encoder, and a learned resampler providing eight task tokens.Source: Appendix C · p. 9
  • The 500M model’s policy transformer and action head are trained from scratch on task-specific data, while the 2B model is pretrained across tasks and embodiments before target-task fine-tuning.Both use conditional flow matching with a two-step Euler solver to generate 40-step action chunks on a 50 Hz action grid.Source: Appendix C · p. 9
  • Approximate inference latency on an NVIDIA RTX 5090 GPU is 30–70 ms per prediction, depending on model size.Source: Appendix C · p. 10

Appendix D Hardware and Teleoperation

The hardware setup combines wrist and head cameras with glove-and-tracker teleoperation for controlling the robot’s fingers and arm.

Appendix D Hardware and Teleoperation

  • The robot uses two wrist-mounted fisheye cameras and a head-mounted ZED Mini stereo camera.Source: Appendix D · p. 10
  • A Manus Metaglove Pro controls the fingers by retargeting the operator’s hand skeleton to Wuji joint angles.Source: Appendix D · p. 10
  • A Vive Tracker 3.0 maps wrist motion to the arm’s TCP pose.Source: Appendix D · p. 10

Appendix E Experimental Results

The appendix describes 50 real-world evaluation trials per policy, including nominal and shifted conditions, and notes that Table 2 aggregates successes by condition.

Appendix E Experimental Results

  • Each policy is evaluated over 50 real-world trials: 24 nominal and 26 under conditions not covered by the real demonstrations.The shifted trials include red lighting, a faster belt, novel objects, and one trial combining all new conditions.Source: Appendix E · p. 10
  • Co-trained policies use 100 real demonstrations and 1,500 simulated trajectories.Source: Appendix E · p. 10
  • Table 2 aggregates successes by condition, with Shifted defined as all non-nominal trials.Source: Table 2 · p. 10, Appendix E · p. 10

Related papers

Loading 2610.00821v1…