PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic RSR: Real-to-Sim-to-Real through Scene Reconstruction and Execution-Grounded Robot Policies

Yihan Li, Yating Feng, Shengjiu Sun, Jianing Chen, Hao Ren, Bowen Yang, Weisheng Xu, Qiwei Wu, Hui Cheng, Renjing Xu

arXiv:2610.10479v1cs.ROcs.CV

TL;DR

Robot policies developed in simulation must preserve task-relevant interactions while relying on observations available on hardware. Agentic RSR connects video-based workspace reconstruction, staged executable-policy development, and feedback-guided robot execution. Across tasks, 80.0% of simulation-successful tasks also succeed on the real robot.

  • Problem

    Simulation for a captured workspace must preserve task-relevant interactions, and its policies must operate on observations available to the real robot.

    Source: §1 · p. 1

  • Method

    Agentic RSR anchors a reconstructed scene to known robot geometry, refines it with visual and physical checks, and develops a staged executable policy for transfer with experience memory.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Agentic RSR achieves 80.0% paired transfer: 8 of 10 simulation-successful tasks also succeed on the real robot.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The framework connects reconstruction of a captured workspace, policy development, and real-robot execution through a shared task-level loop.

    Source: §6 · p. 9

    Variability in the closed-source model service makes agent behavior difficult to reproduce; the authors call for clearer versioning and stable invocation interfaces.

    Source: §5 · p. 9

Abstract

from arXiv · show

A simulation of a real robot workspace must preserve task-relevant interactions, while policies developed in it must operate on observations available to the real robot. Yet scene reconstruction and policy development are often treated separately. We present Agentic Real-to-Sim-to-Real (Agentic RSR), a framework that links scene reconstruction, policy development, and real-robot execution through the same manipulation task. Given a workspace video, a task description, and a known robot model, an agent recovers metric scale, iteratively refines the scene using visual feedback, and checks task-relevant interactions in MuJoCo. A coding agent then develops an executable policy, progressing from privileged object poses to visual observations and randomized simulation. The policy can interleave multiple observations and actions within one invocation, while the agent uses execution feedback to continue, retry, or revise its approach. A shared task-level interface carries the policy and accumulated experience to the real robot, where fresh observations and safety checks guide execution. Across 18 reconstructed scenes involving two robots, the mean four-view Depth MAE against reference depth estimates is 0.1057 m, the mean Lab $\Delta E_{76}$ is 11.04, and the mean grayscale SSIM is 0.6990. In real-robot experiments, the aggregate task success rate reaches 80% of the simulation task success rate, indicating substantial retention of simulated performance on hardware. Code and reconstructed scene data will be made publicly available.

1 INTRODUCTION

Real-to-sim policy development must preserve task-relevant interactions while using observations available to the real robot. Agentic RSR connects workspace reconstruction, policy development, and hardware execution through a shared task-level interface.

1 INTRODUCTION

  • Real-world policy trials are costly to scale, while simulated policies need task-relevant interactions and observations available to the real robot.Prior systems bridge portions of reconstruction, simulation, programmatic control, and transfer; the paper asks how to couple these elements for one captured workspace and task.Source: §1 · p. 1
  • Agentic RSR links metrically aligned scene reconstruction, executable policy development, and feedback-guided real-robot execution.A shared task-level interface carries the policy and experience memory to hardware, where fresh observations support verification and recovery.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Scene reconstruction anchors metric scale to known robot geometry, uses visual feedback, and checks task-relevant geometric and physical interactions.The scene is refined so it can support robot execution.Source: §1 · p. 2
  • The policy agent develops an executable program with reusable steps and uses execution results and fresh observations to continue, switch steps, retry, or recover.The policy bundle remains locked during an episode while the agent responds to execution feedback.Source: §1 · p. 2

2 RELATED WORK

Related work spans real-scene reconstruction and simulation, programmatic robot policies, and execution-grounded agents. Agentic RSR focuses on connecting reconstruction of a particular captured workspace with a programmatic policy for the same task.

2 RELATED WORK

  • Prior scene-reconstruction work ranges from simulation-ready assets and interaction geometry to video-based interactive scenes and digital cousins for policy evaluation or training.Examples include Re3Sim and RoboSimGS, and Agentic Real2Sim and SimFoundry [Han et al., 2025].Source: §2 · pp. 2–3
  • Programmatic-policy research composes perception and actions into executable programs, while other work studies feedback-driven planning, skill development, and code-policy transfer.Code as Policies and Instruct2Act generate programs; CaP-X studies multi-turn coding agents and transfer across simulation and hardware [Huang et al., 2023a].Source: §2 · p. 3
  • Agentic RSR focuses on a particular captured workspace and the programmatic policy for its task, connecting reconstruction, policy development, and real-robot trials.Table 1 compares reported capabilities in real-scene reconstruction, agentic scene revision, code-policy generation, execution-time recovery, and sim-to-real trials.Source: §2 · p. 3, Table 1 · p. 3

3 METHOD

The method connects robot-anchored scene reconstruction and task-interaction checks with staged policy development, then transfers policy and experience to the real robot. Visual and execution feedback guide refinement and adaptation across the loop.

3.1 REAL-TO-SIM SCENE RECONSTRUCTION FOR EXECUTABLE TASKS

  • Known meter-scale robot geometry anchors registration, converting video-estimated depths and camera poses into the metric robot-base frame.Masked robot depths form a point cloud aligned to the URDF geometry; dividing estimated depth by the recovered scale converts it to meters.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • Scene refinement edits geometry, layout, materials, and lighting against four fixed reference views, with acceptance requiring both depth and appearance thresholds.The visual check also imposes per-view depth, color, and valid-depth-coverage thresholds; mean-depth success alone is insufficient.Source: §3.1 · p. 4
  • After visual acceptance, task objects receive reviewable physical classifications and MuJoCo interaction probes test settling, simplified grasping, or articulated motion.Revised candidates are probed again, and visible geometry or pose changes trigger reference-view regression checks.Source: §3.1 · pp. 4–5

3.2 SIMULATION-BASED POLICY GENERATION AND EXECUTION

  • Policy development progresses from privileged object poses to visual-interface task completion, then to the same interface under bounded scene and action variation.These are development passes; the submitted policy is fixed during final evaluation but can still respond to observations.Source: §3.2 · p. 5, Figure 4 · p. 5
  • The policy interface provides robot state, images, object-position estimates, and pose, gripper, and trajectory actions without a built-in task-success oracle.A program invocation can interleave observations and actions; after failures or yielding, the agent can continue, reset and retry, or revise the policy.Source: §3.2 · pp. 5–6

3.3 SIM-TO-REAL TRANSFER OF POLICY AND EXPERIENCE MEMORY

  • Transfer carries validated policy steps and experience memory, while a fresh agent conversation must reassess task state from current real observations.Robot-side checks enforce coordinate, calibration, action, and workspace constraints; real observations guide continued action or stopping.Source: §3.3 · p. 6

4 EXPERIMENTS

Across 18 scenes, Agentic RSR reconstructs visually and geometrically evaluated task scenes, then tests policies in simulation and on two robots. The experiments report scene-quality metrics, real-robot transfer, and scene-refinement ablations.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers 18 manipulation scenes across two robots, using workspace videos and scene-quality metrics; policy success is measured in simulation and on real robots.The policy evaluation uses task-equal aggregates, with different success-rate reporting for Code as Policies than for Instruct2Act and Agentic RSR.Source: §4 · p. 6, §4.1 · p. 6

4.2 REAL-TO-SIM SCENE RECONSTRUCTION

  • Across 18 scenes, Agentic RSR has the lowest comparable depth and color errors, highest SSIM, and best available object-geometry scores; mean Depth MAE is 0.1057 m, Lab ∆E76 is 11.04, and grayscale SSIM is 0.6990.The depth and appearance metrics improve from 0.9036 m to 0.1057 m and from 24.84 to 11.04, respectively; the mean full-frame grayscale SSIM is 0.6990.Source: Table 2 · p. 7, §4.2 · p. 7
  • Across 42 measured objects, mean symmetric Chamfer distance is 0.0303 m, F1@1cm is 0.3655, and visible bounding-box center error is 0.0869 m.The reported results indicate that visual feedback improves appearance and depth consistency while recovering task-object arrangement.Source: §4.2 · p. 7

4.3 SIMULATION AND REAL-ROBOT POLICY EVALUATION

  • Code as Policies and Instruct2Act report descriptive conversion rates of 51.1% and 71.4%, but their conversion is not paired task transfer.Code as Policies' real-robot scenarios do not fully match the indexed tasks, so those comparisons are descriptive.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • Agentic RSR succeeds on 10/18 simulation tasks, with 8 also succeeding on the real robot, yielding Sim SR 55.6%, Real SR 44.4%, and paired transfer 80.0%.Table 3 covers policy results across 18 tasks; its conversion measure is paired transfer for Agentic RSR.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 ABLATIONS

  • Without scalar metric feedback, only 6/18 scenes pass the full visual check; without initial calibration, only 3/18 pass, while replacing the refinement agent also worsens visual results.Without scalar feedback, mean Depth MAE rises to 0.3746 m; without calibration it rises to 0.2583 m, and the model swap raises it to 0.2119 m.Source: §4.4 · p. 8

5 LIMITATIONS

The authors report limits in reproducibility when relying on a closed-source model service and in responsiveness to rapidly changing contacts.

5 LIMITATIONS

  • Agent behavior varied across devices, invocation interfaces, and runs, but the observations were exploratory rather than measured under controlled matched inputs.The causes remain unclear, and the authors say this variability makes behavior difficult to reproduce with a closed-source model service.Source: §5 · pp. 8–9
  • The authors call for greater transparency, clear model versioning, and stable invocation interfaces to support consistent access and reasonable reproducibility.Source: §5 · p. 9
  • Model-call latency and discrete checkpoints limit response to rapidly changing contacts compared with high-frequency visuomotor control.The authors suggest combining high-level programmatic reasoning with faster low-level closed-loop control.Source: §5 · p. 9

6 CONCLUSION

Agentic RSR connects scene reconstruction, executable policy development, and real-robot execution in a task-level loop. Real observations and execution feedback guide hardware behavior, with robot-side safety mechanisms constraining actions.

6 CONCLUSION

  • The central contribution is a task-level loop in which scene reconstruction, executable policies, and real-world execution feedback inform one another.The policy and experience memory carry task knowledge into hardware execution, where fresh observations guide subsequent behavior.Source: §6 · p. 9
  • Policy development progresses from privileged information to visual observations and randomized simulation, while MuJoCo checks task-relevant physical interactions in reconstructed scenes.Programs can observe and act repeatedly within one invocation, and execution outcomes guide continuation or recovery.Source: §6 · p. 9
  • Generative AI tools assisted with workflow implementation, evaluation planning, records, references, and manuscript preparation, while authors retained responsibility for study design and verification.The authors also retained responsibility for the reported results, citations, and final manuscript.Source: AI USE STATEMENT · p. 9

A TASK CATALOG AND ID MAPPING

The task catalog maps identifiers to descriptions for the scene and policy evaluations, covering tasks for two robot platforms.

A TASK CATALOG AND ID MAPPING

  • Task identifiers are used throughout the scene and policy evaluations.Source: A TASK CATALOG AND ID MAPPING · p. 14
  • The catalog covers nine Franka FR3 tasks and nine Agilex Piper tasks.Table 5 is identified as the source of task IDs and descriptions for both robot groups.Source: Table 5 · p. 14
  • Table 5 maps the task IDs to descriptions for the nine tasks associated with each robot.Source: Table 5 · p. 14

B.1 REGISTRATION OBJECTIVE AND SETTINGS

The registration objective aligns a known meter-scale robot surface with its reconstructed point cloud and ranks candidate transforms using geometric and depth-based criteria.

B.1 REGISTRATION OBJECTIVE AND SETTINGS

  • Candidate registrations are scored using projected robot-mask agreement, bidirectional nearest-neighbor fit, point-cloud coverage, and depth consistency.The passage defines I and R as mean mask IoU and recall, and ρ as a normalized nearest-neighbor distance.Source: §B.1 · p. 14
  • The similarity transform T(x) = sRx + t aligns the meter-scale robot surface to the reconstructed robot point cloud, with s converting meters to VGGT units.Source: §B.1 · p. 14
  • The expression L = 0.50(1 − I) + 0.12(1 − R) + 0.14 min(5ρm→o, 1) + 0.08 min(5ρo→m, 1) uses four terms weighted 0.50, 0.12, 0.14, and 0.08.Source: §B.1 · p. 14
  • When valid depth overlap is unavailable, the final loss term receives its maximum penalty.Table 6 summarizes the registration settings.Source: §B.1 · p. 14, Table 6 · p. 14

B.2 REGISTRATION SUMMARY

The registration summary reports fit statistics across captures for each robot and distinguishes them from reference-pose error and downstream scene-quality metrics.

B.2 REGISTRATION SUMMARY

  • Table 7 summarizes registration fit across nine captures per robot.The table is identified as reporting robot-registration results.Source: §B.2 · p. 14, Table 7 · p. 15
  • The summary defines Lf as final ranking loss and averages IoU across frames.Source: §B.2 · p. 14
  • The task-median projected-depth residual is reported in millimeters and is distinct from reference-pose error and downstream scene-quality metrics.Its units are obtained by converting the VGGT-scale residual using s.Source: §B.2 · p. 14

C COMPLETE QUANTITATIVE RESULTS

This section presents per-scene reconstruction metrics for Franka and Piper and explains how to interpret the reported measures.

C.1 PER-SCENE RECONSTRUCTION COMPARISON

  • CD is symmetric Chamfer distance in meters, F1 uses a 1 cm threshold, and BBox is visible bounding-box center error in meters.Source: §C.1 · p. 15
  • Lab ΔE76 is lower-is-better, grayscale SSIM is higher-is-better, and a dash denotes an unreported metric.Source: §C.1 · p. 15
  • Table 8 reports per-scene reconstruction metrics for Franka and Piper, with bold marking the best comparable result among reported methods.Source: Table 8 · p. 15, §C.1 · p. 15

C.2 SCENE-REFINEMENT ABLATIONS

This section compares scene-refinement variants across 18 scenes and documents the related workflow for reconstruction, evidence-driven simulation checks, and policy development toward visual, randomized execution. It also specifies how task context and accumulated experience are summarized for a fresh agent.

C.2 SCENE-REFINEMENT ABLATIONS

  • GPT-5.6 Sol passed the visual check on 10 of 18 scenes; the other variants retain GPT-6 Astra while removing scalar metric feedback or initial calibration.Each variant was evaluated on the same 18 scenes with at most 10 refinement rounds; unsuccessful scenes use their last valid evaluation, or the round-0 baseline if none was produced.Source: §C.2 · p. 15

D AGENT PROMPT TEMPLATES

  • The appendix reproduces task-independent prompt templates with runtime substitutions, omitting local paths, private run locations, identifying metadata, images, and public execution reports.The templates describe those accompanying materials without reproducing them.Source: D AGENT PROMPT TEMPLATES · p. 16

D.1 REAL-TO-SIM

  • The real-to-sim prompts have a planning stage based on four RGB views and a refinement stage that uses fixed cameras, visual comparisons, and scalar depth and Lab feedback.Refinement proceeds from geometry and contact through appearance, then checks all four rendered views and the latest metrics; the robot origin and camera configuration remain fixed.Source: §D.1 · pp. 17–19
  • The physical-scene prompts classify each non-robot mesh by contact role and collider, and request image-based broad priors for movable objects when free-joint objects are present.Classification distinguishes visual-only, static, dynamic, operable, and compound parts; the image-only prior is explicitly uncertain and is not a real-world measurement or calibration.Source: §D.1 · pp. 19–20
  • Scene checks probe object stability and grasp or articulation behavior, but simulation results alone do not verify real-world physics or manipulation.For free objects, the grasp probe examines contact, hold, rise, and slip; joint-only scenes use an articulation probe instead. The runner sets real_world_verified=false.Source: §D.1 · pp. 21–22
  • Repairs are conditional: change one justified cause at a time, compare candidates with the unchanged scene under matched tests, and retain changes only if physical checks and visual alignment hold.If the original is stable and its grasp-force bracket is plausible, retain it unchanged; a successful simulation grasp or visual prior does not establish true mass or friction.Source: §D.1 · pp. 20, 22

D.2 SIM POLICY ITERATION

  • Policy authoring progresses through privileged_exact, visual_exact, and visual_randomized stages, with randomization of object poses, camera calibration, and pose execution in the final stage.The stages share one authoring template and may pass the preceding policy and accumulated experience forward.Source: §D.2 · p. 23
  • Policies re-observe after meaningful actions and report completion only with fresh post-action evidence; execution feedback can prompt continuation, reset, policy revision, or an outer-agent checkpoint.During authoring, practice returns a command result and images to the same conversation before policy submission.Source: §D.2 · pp. 24–25
  • Each completed SIM stage prompts a summary for the next fresh agent, covering task requirements, estimated useful observations and coordinates, successful motion and verification patterns, failures, and next steps.The next stage starts a new conversation and reads the summary document first.Source: §D.2 · p. 25
  • The handoff must rely on attached images as visual evidence and exclude simulator internals, hidden object truth, private identifiers, and unsupported certainty.The prompt requests JSON with one field, experience_markdown.Source: §D.2 · p. 25
Loading 2610.10479v1…