PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldMirror: Agentic Real-to-Sim Generation of Interactive Worlds from In-the-Wild Robot Demonstrations

Tianling Xu, Weijie Wan, Honghao Su, Kailun Su, Junyuan Tang, Huichi Zhou, Haozheng Zhang, Qi Xiong, Ping Luo, Wenbo Ding, Tianxing Chen, Yue Chen

arXiv:2610.31834v1cs.ROcs.CV

TL;DR

Simulation can provide scalable data generation and repeatable evaluation, but real-to-sim reconstruction must match both scene appearance and physical interactions. WorldMirror uses an agent to refine an executable world from robot demonstrations, and DROID experiments report improved spatial alignment, motion replay, and interaction fidelity; the evaluation remains limited in scope.

  • Problem

    Simulation offers scalable data generation and controlled, repeatable evaluation, but interactive environments must reproduce real-world appearance and physical interactions, which require matching geometry, object placement, viewpoints, and robot configuration.

    Source: §1 · p. 1

  • Method

    WorldMirror converts robot videos and trajectories into interactive digital twins by using a coding agent to coordinate specialized tools and verification-guided revisions within a camera-pose-anchored world representation.

    Source: §1 · p. 2

  • Results

    Experiments on DROID demonstrate improved spatial alignment, motion replay, and interaction fidelity, with competitive reconstruction quality and improved replay fidelity on evaluated demonstrations.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Executable world states provide a useful interface between real demonstrations and simulation-based robot learning, while physical verification and trajectory-level evaluation matter for replaying recorded trajectories.

    Source: §5 · p. 10

    Evaluation is limited to a small set of DROID demonstrations and does not yet include downstream policy learning, real-world deployment, or broader cross-task generalization.

    Source: §5 · p. 10

Abstract

from arXiv · show

Existing approaches for robot learning in simulation often require substantial manual effort to construct scenes and specify tasks, limiting training scalability and data diversity. Meanwhile, in-the-wild robot demonstrations naturally provide diverse and realistic observations of real-world interactions. Leveraging these demonstrations for simulation requires both high-fidelity visual reconstruction and physically consistent interactions. General-purpose agents offer flexible adaptation to diverse scenarios and strong reasoning about 3D spatial relationships, while specialized generative models provide high-fidelity reconstruction of scene appearance and geometry. We introduce WorldMirror, an agentic real-to-sim framework that constructs interactive digital twins from in-the-wild robot demonstrations. Our central idea is to formulate real-to-sim construction as the iterative refinement of an executable world program. A coding agent decomposes the observed scene, plans reconstruction steps, and orchestrates specialized image and 3D generation tools, refining the program through verification feedback. A camera-pose-anchored scene representation organizes the world into static visual context, support geometry, and editable interactive assets while preserving spatial relationships among the scene, objects, and robot. The agent uses this representation to diagnose and correct reconstruction errors, enabling physically consistent interaction and trajectory replay. Experiments on DROID demonstrate that WorldMirror can reconstruct visually faithful and geometrically consistent interactive worlds while preserving the physical validity of recorded robot trajectories.

1 INTRODUCTION

WorldMirror addresses the need for scalable simulation environments by turning robot demonstrations into interactive digital twins through adaptive, verification-guided reconstruction. Its camera-pose-anchored representation preserves spatial relationships for trajectory replay, and evaluation on DROID reports competitive reconstruction quality and improved replay fidelity on the evaluated demonstrations.

1 INTRODUCTION

  • Simulation can support scalable data generation and controlled, repeatable evaluation, but needs interactive environments that reproduce real-world appearance and physical interactions.Collecting demonstrations and testing policies across environments remain costly and labor-intensive; even small scale or contact-geometry errors can alter task outcomes.Source: §1 · p. 1
  • WorldMirror coordinates specialized reconstruction tools through an iterative executable-world program, selecting revisions based on world state and verification feedback.Different errors call for different corrections, such as adjusting layout, revisiting decomposition and asset reconstruction, or revising collision geometry.Source: §1 · p. 2
  • Its camera-pose-anchored representation separates static visual context, support geometry, and editable interactive assets in a shared coordinate system for trajectory replay.The representation preserves spatial relationships among the scene, objects, and robot, allowing recorded trajectories to be replayed without modification.Source: §1 · p. 2
  • WorldMirror demonstrates competitive reconstruction quality and improved replay fidelity on evaluated DROID demonstrations.Figure 1 reports better visual fidelity, collision geometry, and physical consistency than GPT-6 Astra, which is competitive in spatial structure and layout.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Prior work develops interactive real-to-sim reconstruction, agentic 3D modeling, and simulation environments for robot learning. WorldMirror situates its approach among these efforts by organizing component-level reconstruction and alignment in an editable executable world guided by verification feedback.

Real-to-sim world construction

Agentic 3D Modeling

Simulation for Robot Learning

3 METHOD

WorldMirror formulates digital-twin construction as iterative refinement of an executable world program, coordinating reconstruction through shared state and verification feedback while preserving camera-anchored spatial relationships.

3.1 PROBLEM FORMULATION

  • WorldMirror maps RGB video and robot trajectories to a simulation-ready twin whose visual scene, editable task objects, and robot share a coordinate frame that keeps the recorded trajectory executable.Source: §3.1 · pp. 4–5

3.2 STRUCTURED WORLD STATE MEMORY

  • Persistent shared memory records static scene state, interactive objects, spatial relationships, simulation attributes, and verification feedback to guide later routing decisions.Source: §3.2 · p. 5

3.3 AGENTIC WORLD CONSTRUCTION

  • A router traverses a fixed dependency graph, calling specialized agents and reusable skills while selecting legal next actions from the current world state and verification feedback.Specialized capabilities cover reconstruction, calibration, layout, and replay.Source: §3.3 · pp. 5–6

3.4 VERIFICATION-DRIVEN REFINEMENT

  • Candidate updates are committed only after stage-specific checks pass; typed failures trigger legal rewinds and rebuilding of dependent components until checks pass or the refinement budget is reached.Source: §3.2 · p. 5, §3.4 · p. 6

3.5 CAMERA POSE-ANCHORED COORDINATE SYSTEM

  • A camera-anchored world frame places the scene, support, objects, and aligned robot together, allowing recorded arm trajectories to replay without replanning or retargeting.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

WorldMirror is evaluated on DROID demonstrations for visual, spatial, and interaction consistency, including direct replay of recorded trajectories. Its strongest reported result is replay success, while spatial results vary by metric and targeted, feedback-driven refinement outperforms non-adaptive strategies.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses DROID manipulation demonstrations and assesses visual fidelity, spatial consistency, and interaction consistency against the original demonstrations.The setup compares RoLA, RoboSnap, SimFoundry, and GPT-6 Astra; fidelity metrics span visual, spatial, and replay measures.Source: §4 · p. 7, §4.1 · p. 7

4.2 MIRRORING REAL-WORLD APPEARANCE

  • On ten DROID scenes, WorldMirror achieves the lowest LPIPS and DISTS, with competitive PSNR and aesthetic scores among evaluated methods.The comparison uses first-frame simulation renders and real exterior-camera images; Figure 3 compares reconstructed views and Figure 4 shows wrist-view alignment over time.Source: §4.2 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8

4.3 PRESERVING REAL-WORLD SPATIAL STRUCTURE

  • WorldMirror outperforms evaluated open-source baselines across four spatial metrics, but has lower mask IoU and higher position and relative-vector errors than GPT-6 Astra.It achieves the lowest relative distance error, but not a consistent advantage across spatial metrics.Source: §4.3 · p. 7, Table 3 · p. 9

4.4 SUPPORTING DIRECT TRAJECTORY REPLAY AND INTERACTION

  • WorldMirror achieves the lowest errors across four motion metrics, higher contact F1 than GPT-6 Astra, and 100% replay success versus 40% for GPT-6 Astra.Recorded trajectories are replayed in Isaac Sim without trajectory modification or additional motion retargeting.Source: §4.4 · pp. 8–9, Table 4 · p. 9

4.5 ABLATION

  • Full WorldMirror achieves substantially lower replay errors than One-pass and Fixed Iterative, and alone successfully completes the interaction in the described comparison.Fixed Iterative repeatedly refines without failure-aware diagnosis; Full WorldMirror diagnoses failures and selects targeted refinements.Source: §4.5 · p. 9, Figure 5 · p. 9, Table 5 · p. 10
  • Removing adaptive routing leads to replay failure and larger trajectory errors, while removing feedback-driven refinement causes the largest degradation and prevents successful object movement.Removing stateful memory causes only a minor change; its stated benefit is preserving intermediate construction results and verification outcomes across stages.Source: §4.5 · p. 10, Table 6 · p. 10

5 CONCLUSION

WorldMirror constructs simulation-ready digital twins from in-the-wild demonstrations, combining structured world state with verification-driven refinement. The evaluation is limited in scope and does not yet cover downstream policy learning, real-world deployment, or broader cross-task generalization.

5 CONCLUSION

  • WorldMirror constructs simulation-ready digital twins from in-the-wild robot demonstrations.The approach combines structured world state, specialized construction agents, and verification-driven refinement to preserve consistency for unmodified trajectory replay.Source: §5 · p. 10
  • Experiments on DROID demonstrate improved spatial alignment, motion replay, and interaction fidelity.The conclusion highlights coupling visual reconstruction with physical verification and trajectory-level evaluation.Source: §5 · p. 10
  • Evaluation is limited to a small set of DROID demonstrations and excludes downstream policy learning, real-world deployment, and broader cross-task generalization.Reconstruction quality also depends on the available reconstruction and simulation tools.Source: §5 · p. 10
Loading 2610.31834v1…