PaperlayerView exact revision on arXiv

Source-linked AI summary

Real2Gym: Building Gyms from Videos, Bringing Skills to Robots

Kerui Ren, Yingxiang Xu, Kaiwen Song, Linning Xu, Bo Dai, Mulin Yu, Tao Lu

arXiv:2609.37089v1cs.CV

TL;DR

Real-world trial-and-error on physical robots is costly, and building faithful interactive simulation environments remains cumbersome. Real2Gym reconstructs demonstration videos as executable simulation environments and accumulates reusable skills from execution feedback; it reports 87.5% task success with approximately 75% fewer policy-execution tokens than GPT-6 Astra, alongside higher overall success in physical robot execution.

  • Problem

    Physical trial-and-error is costly because motion and resets take time, failures risk damage, and repeated trials require supervision; building interactive simulations also involves cumbersome, manually constrained workflows.

    Source: §1 · p. 2

  • Method

    Real2Gym reconstructs human or robot demonstration videos into visually aligned, physics-verified simulation environments, where an agent uses execution feedback to extract reusable skills for simulation and physical robots.

    Source: §1 · pp. 2–3

  • Results

    87.5% task success with approximately 75% fewer policy-execution tokens than GPT-6 Astra; physical Franka experiments also show higher overall success than direct GPT-6 Astra control.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Skills refined in simulation enable a previously failed plate-placement task, completing the Real2Sim2Real loop.

    Source: §6 · p. 10

    Stage-level code generation lacks the local fine-grained reactivity needed for highly constrained manipulation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.

1 INTRODUCTION

Real-world self-improvement is costly and risky, while preparing interactive simulations for new tasks remains labor-intensive and existing reconstructions can lack physical and geometric fidelity. Real2Gym couples physics-verified environment construction with feedback-driven skill accumulation, reporting gains in simulation and real-robot deployment.

1 INTRODUCTION

  • Real-world trial-and-error is bottlenecked by costly resets, safety risks, supervision demands, hardware wear, and limited opportunities for parallel exploration.Reliable restoration of initial physical states after failure can also be difficult.Source: §1 · p. 2
  • GPT-6 Astra reconstructions can misestimate object dimensions, spatial layouts, and camera extrinsics, leading to unreliable native-physics execution.The reported consequences include interpenetration, missed contacts, and failed grasps that undermine downstream exploration and skill accumulation.Source: §1 · p. 2
  • Real2Gym achieves 87.5% success versus 70.8% for GPT-6 Astra Direct Mode, using approximately 74.9% fewer policy-execution tokens across benchmark scenes.Across 24 reconstructed environments, the paper also reports consistent gains in reconstruction fidelity, task success, and token efficiency, with real-robot experiments validating closed-loop self-improvement.Source: §1 · pp. 2–3
  • Real2Gym constructs executable digital twins from demonstrations through event-centered correction, native-physics validation, and task-conditioned augmentation with action-feasibility checks.The stated goal is to produce diverse, execution-ready gyms for agent exploration.Source: §1 · pp. 2–3
  • Its experience-driven agent combines perception tools, executable operation stages, and feedback-driven skill extraction for simulation and physical-robot use.The introduction reports real-robot deployments in which simulation-refined skills succeed on tasks that initially failed.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers both constructing physics-connected digital twins from real observations and using agents to translate language and visual feedback into robot behavior. Recent work increasingly emphasizes interactive execution, self-correction, and reusable skills.

2.1 REAL-TO-SIMULATION RECONSTRUCTION

2.2 AGENTS FOR ROBOT CONTROL

3 METHOD

Real2Gym has two phases: construct executable, visually aligned simulation environments from demonstrations, then accumulate reusable manipulation skills through agent interaction. Construction combines scene alignment, physics validation, and feasibility-checked variation; the agent generates subtask code and extracts skills from execution outcomes.

3 METHOD

  • The framework transforms demonstration video and a robot description into executable environments, then uses an agent policy to build a reusable skill library.Each environment combines a visually aligned Blender scene with a physically interactive MuJoCo model.Source: §3 · p. 4

3.1 INTERACTIVE GYM CONSTRUCTION

  • Scene reconstruction aligns object geometry, viewpoints, robot pose, and camera parameters to the input demonstration using multi-view cues and iterative reprojection checks.The scene is initialized from single- or multi-view geometry estimates, with semantic segmentation and sparse frames supporting object identification and completion of occluded surfaces.Source: §3.1 · p. 4
  • Demonstrated or retargeted actions are organized around contact and support events, corrected through diagnostic inspection, and executed end-to-end in MuJoCo to verify physical consistency and task completion.Validation checks include gripper closure, grasp retention, support stability, and object release; the resulting physics trajectory is compared with Blender renderings.Source: §3.1 · pp. 4–5
  • Validated seed environments are varied across object, support, material, distractor, background, and lighting factors, with adapted manipulation stages and native-physics checks filtering candidates.Failed candidates return for refinement; validated candidates join the environment pool.Source: §3.1 · p. 5

3.2 AGENT EXECUTION AND SKILL ACCUMULATION

  • At each manipulation subtask, the agent receives the task, observations, episode history, and selected skills, then generates executable Python code with checks and an observable completion condition.The code combines perception, coordinate computations, and robot-control commands; execution returns updated observations and feedback for the next decision.Source: §3.2 · p. 5
  • After each episode, skill extraction distinguishes supported positive and negative effects, producing procedures, applicability conditions, object-relative motions, effect checks, and recovery guidance.Skills are grounded in current observations for reuse in simulation and on real robots, and experience accumulation leaves the underlying model parameters fixed.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments evaluate Real2Gym on reconstructed DROID and EgoDex environments, agent policies, and four physical Franka tasks. The reported results include stronger reconstruction and policy performance, plus simulation-acquired skills that improve a previously failed real-world task.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers 24 reconstructed scenes from DROID and EgoDex, policy comparisons with GPT-6 Astra and GPT-5.6 Sol, and four physical Franka tasks.The four physical tasks include short-horizon pick-and-place, handle hanging, articulated microwave manipulation, and narrow-clearance shelf insertion.Source: §4.1 · pp. 6–7, §4.3 · p. 8

4.2 COMPARISON

  • Our agent completes bowl stacking after both baselines fail; later skill-conditioned runs improve cabinet manipulation and make adapter placement more direct.The authors report that reusing lessons reduces repeated exploration, consistent with lower average token usage for the agent with skills.Source: Figure 4 · p. 7, Figure 5 · p. 8, §4.2 · p. 8
  • On DROID, viewpoint alignment improves by 40.83 points over GPT-6 Astra; on EgoDex, simulation success score reaches 85.76, a 51.5% relative improvement over that baseline.Across both datasets, Real2Gym outperforms both baselines on all four reported reconstruction metrics.Source: §4.2 · p. 7
  • Before skill accumulation, success averages 79.2%, with 67.3% fewer tokens than GPT-6 Astra; stage-level code coordinates multiple actions and verification checks per response.Averaged across both datasets, the agent also has higher success, fewer responses, lower token use, and shorter execution time than both baselines.Source: §4.2 · p. 8

4.3 REAL-WORLD DEPLOYMENT AND SELF-EVOLUTION

  • Zero-shot, our agent matches or exceeds Direct Mode on all four tasks: both comparisons are 100% SR (3/3) vs 33% (1/3), task 2 ties, and task 4 is 0% for both.The two 100% vs 33% comparisons are tasks 1 and 3.Source: §4.3 · pp. 8–9, Figure 6 · p. 9
  • After two failed simulation attempts, the skill succeeds by the third evolution iteration at 100% simulation SR and transfers at 100% physical SR.The task is Place Plate on Shelf; the skill is acquired in a twin reconstructed from a human demonstration.Source: Figure 6 · p. 9, §4.3 · p. 9

5 LIMITATIONS

The authors identify limits in reconstruction fidelity, control granularity, and tested robot embodiments.

5 LIMITATIONS

  • Misalignment between wrist-mounted and static cameras can leave ghosting, layered surfaces, and residual geometric or textural discrepancies in noisy Pi3X point clouds.Improving cross-view registration and appearance refinement is identified as a future priority.Source: §5 · p. 9
  • Stage-level code generation improves interaction efficiency but lacks local fine-grained reactivity for highly constrained manipulation.The authors plan to explore adaptive switching between stage-level and fine action-level control.Source: §5 · p. 9
  • Real-world experiments are limited to Franka arms with parallel-jaw grippers, leaving dexterous hands and diverse humanoid platforms untested.The authors identify broader embodiment coverage as necessary to assess practical generality.Source: §5 · p. 9

6 CONCLUSION

Real2Gym turns human and robot demonstrations into interactive simulation environments where execution feedback accumulates reusable skills. The reported evaluations show reconstruction and policy gains, as well as simulation-refined skills succeeding on a previously failed physical task.

6 CONCLUSION

  • Real2Gym converts demonstrations into interactive simulation environments and uses execution feedback to accumulate reusable robot skills.The framework supports building environment collections from human demonstrations, with skills guiding later physical execution.Source: §6 · p. 10
  • In simulation, the agent with accumulated skills achieves 87.5% task success with approximately 75% fewer policy-execution tokens than GPT-6 Astra.Physical Franka experiments report higher overall success than direct GPT-6 Astra control, and simulation-refined skills complete a previously failed plate-placement task.Source: §6 · p. 10
  • Generative AI assisted with synthetic datasets, implementation, translation, manuscript readability, and literature search; authors report reviewing the AI-assisted work.The authors state that AI tools did not design methods or experiments, interpret results, or formulate mathematical claims or proofs.Source: AI USE STATEMENT · p. 10

A PER-TASK REAL2SIM RESULTS

Table 3 compares Real2Gym with two model baselines on DROID and EgoDex, reporting task-level results and dataset means. Real2Gym generally improves visual alignment and interaction fidelity, though gains vary by task.

A PER-TASK REAL2SIM RESULTS

  • Real2Gym generally improves visual alignment and interaction fidelity across DROID and EgoDex, although gains vary by task.Dataset means summarize performance across listed tasks.Source: A PER-TASK REAL2SIM RESULTS · p. 14
  • The evaluation measures content alignment, viewpoint alignment, action fidelity, and simulation success score.The simulation success score uses a 0–100 scale.Source: A PER-TASK REAL2SIM RESULTS · p. 14, Table 3 · p. 14
  • Table 3 reports per-task results, with bold marking the top performance per task and metric, including ties.Source: Table 3 · p. 14

B PER-TASK SIMULATION CONTROL RESULTS

The simulation-control results report added task successes and lower token use on most tasks with skills, while some failures remain. The accompanying figures show manipulation sequences and a qualitative model-and-effort comparison.

B PER-TASK SIMULATION CONTROL RESULTS

  • With skills, the agent additionally solves D10 and D11 and reduces token usage on most tasks, though several failures remain.Table 4 compares task success and execution efficiency across baselines and the method with and without skills on DROID and EgoDex.Source: B PER-TASK SIMULATION CONTROL RESULTS · p. 14, Table 4 · p. 15

DROID EgoDex

C REAL-WORLD EXECUTION SEQUENCES

  • Figure 8 compares 12 model-and-reasoning-effort configurations on the EgoDex assembly task using columns indexed by source-video timestamps.Failed-attempt frames marked FAILED retain a frozen state; they do not indicate continued execution.Source: Figure 8 · p. 17

D ABLATION ON VLM MODEL AND REASONING EFFORT

The assembly-task ablation shows that performance varies across models and reasoning effort: only GPT-6 Astra passes the full-task simulation criteria at all three tested effort levels. Other models exhibit earlier failures or reconstruction discrepancies.

D ABLATION ON VLM MODEL AND REASONING EFFORT

  • Only GPT-6 Astra passes the full-task simulation criteria at all three tested reasoning-effort levels, completing an adapted execution through nut tightening and final insertion.Source: D ABLATION ON VLM MODEL AND REASONING… · p. 17
  • GPT-6 Sol reaches the second-wheel stage at xhigh but fails to release the wheel, while its high- and medium-effort runs fail earlier.The described failures occur during grasping or before assembling the first wheel.Source: D ABLATION ON VLM MODEL AND REASONING… · p. 17
  • GPT-6 Luna and GPT-5.6 Sol show larger geometry, object-relationship, or viewpoint discrepancies and fail before completing the assembly sequence.The case study highlights a gap between visually plausible reconstruction and physically executable long-horizon manipulation.Source: D ABLATION ON VLM MODEL AND REASONING… · p. 18

E REAL2SIM EVALUATION CRITERIA AND PROMPT TEMPLATE

The evaluation prompt uses an explicit rubric to assess reconstruction and demonstration reproduction. Its simulation success score is distinct from task-success outcomes and does not replace native task-success checks.

E REAL2SIM EVALUATION CRITERIA AND PROMPT TEMPLATE

SHARED EVALUATION RULES

The evaluation rules score reconstruction, viewpoint alignment, action fidelity, and simulation success against the source demonstration, using shared criteria across methods. Simulation success combines action fidelity and task completion, with a zero-score override for confirmed physical simulation failures.

E1 CONTENT ALIGNMENT

  • Reconstruction content alignment totals 100 points, split equally between source-to-MuJoCo and source-to-Blender alignment.Each engine is scored from its own evidence; quality in one engine cannot stand in for the other.Source: SHARED EVALUATION RULES · p. 18

E2 VIEWPOINT ALIGNMENT

  • Viewpoint alignment scores camera position, orientation, distance, field of view, framing, visibility, and occlusion against the source using the submitted scene’s existing camera.Do not refit or crop the camera to conceal differences, or interpret image displacement across views as calibrated 3D position error.Source: SHARED EVALUATION RULES · p. 19

E3 3. ACTION FIDELITY

  • Action fidelity scores goals and objects, action order, contact relations, motion paths, timing, and initial and final states, with a 40-point cap for non-equivalent manipulation logic.The cap does not apply solely for embodiment differences, inverse kinematics, motion smoothing, or global speed scaling.Source: SHARED EVALUATION RULES · pp. 19–20

E4 4. SIMULATION SUCCESS SCORE

  • Simulation success rewards executions that both complete the manipulation and reproduce the source demonstration; task completion through a non-aligned process receives low credit.Task completion evaluates required steps, intended final state, and physically plausible execution.Source: SHARED EVALUATION RULES · p. 20
  • Simulation success is the per-scene product of action fidelity and task completion divided by 100, averaged across scenes; confirmed physical simulation failure overrides the score to zero.Task completion scores required steps, final state, and physical execution; ordinary omissions or timeouts alone do not trigger the override.Source: SHARED EVALUATION RULES · pp. 20–21

SHARED EVALUATION RULES

  • Use evidence-checkable criteria with identical weights across methods for each scene, and permit replacing human arms with the target robot without penalty by itself.Assess whether task objects, layout, contact relations, and manipulation causality remain preserved.Source: SHARED EVALUATION RULES · pp. 18, 20

F REAL2SIM BASELINE IMPLEMENTATION

The implementation prompts specify how baselines reconstruct and execute demonstrations in Blender and MuJoCo, while agent prompts route the selected backend to shared and backend-specific instructions. The physical Franka instructions distinguish real-robot actions, observations, and recovery from simulation procedures.

F REAL2SIM BASELINE IMPLEMENTATION

  • Real2Sim baseline prompts require reconstruction of Blender and MuJoCo scenes from supplied demonstrations and reproduction of the demonstrated manipulation in simulation.Inputs include videos, robot trajectory data, robot model, and task description; the prompt specifies the manipulation sequence and robot constraints.Source: F REAL2SIM BASELINE IMPLEMENTATION · p. 21, CONSTRUCTION PROMPT · p. 21

CONSTRUCTION PROMPT

  • The construction prompt preserves the supplied Franka Panda and Robotiq 2F-85 model structures and physical parameters across both engines and augmented scenes.Constraints cover link and joint structure, gripper coupling, geometry, mass, centers of mass, and inertias.Source: CONSTRUCTION PROMPT · p. 21

C2 Human Demonstrations: EgoDex (continued)

G AGENT POLICY BASELINE PROMPTS

G.2 SHARED SKILL INSTRUCTIONS

  • Shared skill instructions define observation, execution, verification, and memory procedures, while the common API document specifies shared observation fields, action formats, and return values.The shared instructions direct the agent to the selected backend documentation.Source: §G.2 · p. 22, §G.3 · p. 24

G.4 SIMULATION INSTRUCTIONS

  • Simulation instructions cover reset behavior, warmup, control modes, and action execution; warmup uses validated directional probes when no matching setup memory exists.The probes are observed individually, with the robot returned to its neutral pose before the next probe.Source: §G.4 · p. 24

G.5 PHYSICAL FRANKA INSTRUCTIONS

  • Physical Franka instructions add TCP control, camera calibration, reset semantics, and fault handling, while physical reset only refreshes or checks observations.The physical API provides no depth or point clouds, and calibration alone does not establish metric depth.Source: §G.5 · p. 25, P4 Physical Franka Instructions… · p. 26

P4 Physical Franka Instructions (continued)

  • On the Franka, use short action chunks, inspect fresh observations after failures, and replan from measured state when required; do not assume recovery restored the prior pose.Respect workspace and path checks, but visually verify scene clearance because unverified tool collision geometry is absent from the URDF.Source: P4 Physical Franka Instructions… · p. 26
Loading 2609.37089v1…