PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation

Sanya Verma, Luca Cilio, Velissarios Christodoulou, Tyler Fermelis, Qi Ting Ng, Rachel Chung, Sofia Liang

arXiv:2610.00731v2cs.RO

TL;DR

Hardware evaluation is costly and difficult to repeat, and the value of simulation depends on how closely its outcomes track the real robot. The paper compares authored and default reconstructions of one robot cell, finding lower score error and progress disagreement with the authored reconstruction across five tasks and two policies.

  • Problem

    Hardware policy evaluation is costly and labour-intensive, and stochastic real rollouts can be difficult to repeat, while prior studies did not test how reconstructing a fixed set of rigid objects affects real-to-sim agreement.

    Source: §1 · pp. 1–2

  • Method

    The study compares an authored reconstruction using metrically scaled object geometry, authored physics, and a custom scene against a default open-source recipe, with other run settings held fixed.

    Source: §1 · p. 2

  • Results

    10.56 percentage points lower score error and 8.32 percentage points lower progress disagreement were reported for the authored reconstruction across five tasks and two policies, with 95% intervals excluding zero.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this evaluator-operated setup, the authored reconstruction bundle narrows score and progress disagreement relative to the default recipe.

    Source: §1 · p. 2

    The study covers only five tasks and two policies, and ten cells leave correlation and failure-stage disagreement intervals too wide to separate the reconstructions.

    Source: §6 · p. 14

Abstract

from arXiv · show

Simulated evaluation is increasingly used alongside real-world evaluation of robot policies because it is cheaper and easier to repeat; however, its value depends on how closely its outcomes track the real robot's. We test whether our reconstruction pipeline, combining metrically scaled object geometry, authored physical parameters and scene reconstruction, reduces disagreement between simulated and real robot scores relative to a default open-source recipe. We constructed two simulated versions of one bimanual robot cell: an authored reconstruction, using object geometry at estimated metric scale, projected textures, authored physics and our own scene splat; and a baseline, referred to as the default reconstruction, using the open-source recipe of a generative single-image mesh, engine-default physics and a Gaussian-splat scene. Both reconstructions use the same object photographs and scene video. Two policies ran five tasks each, giving ten task-policy pairs, which we call cells; each cell was run twenty times in each reconstruction with all other settings held fixed. Both reconstructions were scored against the same real trials, graded by a third-party evaluator. Pearson correlation between the ten simulated and real cell means is r = 0.90 for the authored reconstruction and 0.51 for the default. Mean score error is 6.97 percentage points for the authored reconstruction and 17.54 for the default, a reduction of 10.56 percentage points. These results show that improving the quality of the environment reconstruction through higher visual fidelity, authored physics and metric scale makes the simulation more faithful to the real world and narrows the sim-to-real gap. We release the harness, the per-trial scores, every reported run's configuration, and the assets and scenes of both reconstructions.

1 Introduction

Hardware evaluation is costly and difficult to repeat, while simulated evaluation is useful insofar as its scores track the real robot. This study tests whether reconstructing objects and the scene differently reduces that disagreement.

1 Introduction

  • Digital twins offer an alternative to costly, labour-intensive hardware evaluation, whose stochastic rollouts can be difficult to repeat exactly.Prior video-scan systems reported Pearson correlations of 0.90 and 0.91 between simulated and real scores for their studied policies and tasks.Source: §1 · p. 1
  • Earlier studies varied rendering, task factors, or object identity, but did not measure how reconstructing a fixed set of rigid objects changes real–simulated agreement.Prior asset pipelines left physics at engine defaults or estimated it without measurement validation; one complete pipeline authored mass and friction but did not isolate their effect.Source: §1 · p. 2
  • The study compares an authored reconstruction with metrically scaled geometry, per-object physics, and a custom scene reconstruction against a default open-source recipe.Both use the same policies, tasks, real trials, object photographs, and scene video, with other run settings held fixed.Source: §1 · p. 2
  • 10.56 percentage points lower score error and 8.32 percentage points lower progress disagreement were reported for the authored reconstruction across five tasks and two policies.Both 95% intervals excluded zero; correlation and failure-stage disagreement did not separate the reconstructions with ten cells.Source: §1 · p. 2

2 Related work

Prior work spans real-to-sim evaluation, asset-building pipelines, and physical-property estimation. The reviewed studies do not measure how changing the fidelity of a fixed set of rigid objects changes simulated-to-real agreement.

Real-to-sim evaluation systems

  • Existing real-to-sim systems report agreement and examine rendering, task factors, or object identity, but none builds the same setup twice to measure how agreement changes.PolaRiS [2] found both correlation and MMRV worsened as texture was removed; SimFoundry [3] varied no assets, while Wang et al. [4] varied object identity rather than reconstruction quality.Source: §2 · p. 3

Asset-creation pipelines

  • Asset pipelines differ in geometry, scale, collision, and physics: PolaRiS [2] leaves mass unset by default, while SimFoundry [3] automates object meshes, colliders, mass, and friction.SimFoundry’s convex decomposition covers its objects, whereas the authors’ pipeline uses signed distance fields on seven of sixteen object kinds and combines model estimates with a per-class friction table.Source: §2 · p. 3

Physical-property estimation

  • Physical-property methods improve estimates, but the reviewed rigid-object studies do not test whether estimate quality changes real-to-sim agreement.For deformable objects, removing per-spring stiffness optimisation raised MMRV on a rope-routing task from 0.174 to 0.248.Source: §2 · p. 3

3 Experimental setup

The experiment compares two reconstructions of one bimanual robot station across five tasks and two policies. It holds other run settings fixed while varying object assets and the scene reconstruction.

3 Experimental setup

  • The study covers ten task-policy cells, with twenty simulated runs per cell in each reconstruction, for 400 simulated trials paired against 200 real trials.The five tasks include two evaluator-standard tasks and three specified for this study.Source: §3 · pp. 3–4
  • Real trials were run and graded by Robocurve, which retained control of real-side methodology, scoring, and interpretation.Per-trial scores, rubrics, step budgets, and rig assignments are published by Robocurve and the scores are also released with the paper.Source: §3 · p. 4
  • The authored reconstruction uses photograph-based geometry scaled with a ChArUco board, authored object physics, and a custom scene pipeline; the default uses TRELLIS meshes, engine-default physics, and a Gaussian-splat scene.The default recipe hand-scales objects and uses convex decomposition colliders, which fill concave shapes such as bowls and bins.Source: §3 · p. 5, Figure 1 · p. 6
  • Both scene splats come from the same handheld video and are aligned, scaled, positioned, and cut to share a common frame and remove above-table objects.The table sets splat scale at 0.807 m per unit; its measured height sets the table surface at 0.750 m.Source: §3 · p. 7
  • Trials start from fixed per-cell placements based on a real frame, with each simulated run shifting objects by up to 1 cm; placement varies less than in real trials.Placement is identical between reconstructions, so it does not contribute to their difference; Figure 2 shows the first top-camera frame for each cell.Source: §3 · p. 7, Figure 2 · p. 7

4 Method

The method holds the robot model, tasks and real scores, and policy fixed while changing the reconstruction. It measures score-level and task-progress agreement with real trials, then compares reconstructions using paired cell-level uncertainty estimates.

4 Method

  • The harness runs each policy for twenty trials per cell in Isaac Sim and records a per-trial score under the evaluator’s rubric.The robot model, tasks with real per-trial scores, and policy are fixed; only the reconstruction changes.Source: §4 · p. 7
  • Nine logged settings must match across reconstructions, and the gate rejects runs that fail any check.The gate rejected six default-reconstruction runs rendered at the wrong camera height; it cannot detect properties the runner does not print, such as per-trial placement shift.Source: §4 · p. 8, Table 2 · p. 8
  • Four agreement measures compare score values and task progress: score error, Pearson correlation, progress disagreement, and failure-stage disagreement.Scores are expressed as percentages of each task’s maximum so tasks with different point scales can be averaged.Source: §4 · pp. 9–10
  • A cell is called consistent with real when its interval on the simulated-minus-real mean difference includes zero.The 95% bootstrap percentile intervals use 20,000 resamples; each resample draws twenty simulated and twenty real scores with replacement from the cell.Source: §4 · p. 10
  • Reconstruction comparisons use paired cell differences, with ten cells resampled with replacement; correlation differences are recomputed for each resample.A positive difference indicates the authored reconstruction is closer to real for score error, progress disagreement, and failure-stage disagreement.Source: §4 · p. 10

5 Results

The authored reconstruction has lower score error and progress disagreement than the default, while correlation and failure-stage disagreement do not separate the reconstructions with ten cells. Across individual cells, the authored reconstruction is nearer the real mean more often, and its progress disagreement is closer on most cells.

5 Results

  • Correlation is r = 0.90 for the authored reconstruction and 0.51 for the default, while failure-stage disagreement is 42.00 and 48.50; neither measure separates them with ten cells.The intervals on the paired differences include zero.Source: §5 · p. 11, Table 4 · p. 11
  • Score error is 6.97 percentage points for the authored reconstruction versus 17.54 for the default, a significant paired difference of 10.56 percentage points (95% CI 5.03 to 17.11).The 95% bootstrap interval excludes zero.Source: Table 4 · p. 11, §5 · p. 11
  • Progress disagreement is 15.90 percentage points for the authored reconstruction versus 24.23 for the default, a significant paired difference of 8.32 percentage points (95% CI 1.78 to 16.74).The 95% bootstrap interval excludes zero.Source: Table 4 · p. 11, §5 · p. 11
  • The authored reconstruction’s mean is nearer the real mean in 9 of 10 cells; treating three small margins as ties leaves 7 clear authored wins and none clearly favoring the default.The three small margins are within the run-to-run spread of a single configuration.Source: §5 · p. 11, Table 5 · p. 12
  • The authored reconstruction is closer on progress disagreement in 7 of 10 cells; excluding blocks with MolmoAct2, the paired difference remains 4.80 percentage points (95% CI 0.73 to 9.44).That cell has only one rubric stage above zero in the real data; progress disagreement compares fractions reaching the real robot’s rubric stages.Source: §5 · p. 13, Table 6 · p. 13, Figure 4 · p. 13

6 Discussion

The authored reconstruction reduces score error and progress disagreement relative to the PolaRiS-inspired recipe, but the study cannot isolate which reconstruction changes account for the reductions. Shared failures, grading differences, limited task-policy coverage, and setup choices qualify the comparison.

6 Discussion

  • Score error falls 10.56 percentage points and progress disagreement 8.32 percentage points relative to the PolaRiS-inspired recipe.Source: §6 · pp. 13–14
  • Because geometry, texture, scale, collision geometry, mass, friction, restitution, and scene all change together, the study cannot attribute the reduction to individual factors.Source: §6 · p. 14
  • Blocks with MolmoAct2 over-score in both reconstructions: real trials scored zero 13/20 times, while neither simulator produced a zero.The real mean was 3.50 rubric points, versus 7.75 for authored and 12.75 for default.Source: §6 · p. 14
  • Different human and rule-based graders can shift stage boundaries; a 30-trial spot-check agreed on 28, with two opposite one-stage differences.Source: §6 · p. 14
  • With five tasks and two policies, real policy ordering is confident on only two tasks, and narrower simulated placement leaves part of the remaining gap attributable to placement.The study also reports that intervals on correlation and failure-stage disagreement are too wide to distinguish the reconstructions.Source: §6 · p. 14
  • Two setup choices could favor the authored reconstruction: concave default colliders fill shapes, and its assets use one photograph per object with scale set by hand.Source: §6 · p. 14

7 Conclusion

The conclusion reports closer real-score tracking for the authored reconstruction than for the default recipe, while noting that the combined pipeline comparison cannot identify individual factors. The study releases its evaluation materials and reconstruction assets.

7 Conclusion

  • Mean score error is 6.97 percentage points for authored and 17.54 for default, a 10.56-point reduction; progress disagreement falls 8.32 points.The authored reconstruction is consistent with real scores in 9 of 10 cells versus 5 of 10 for default, with correlations r = 0.90 and 0.51.Source: §7 · p. 15
  • Because geometry, scale, physics, and scene differ together, the reported effect is for the pipeline as a whole, not its individual factors.Source: §7 · p. 15
  • The release includes the harness, paired per-trial scores, every reported run’s configuration, and both reconstructions’ assets and scenes.Source: §7 · p. 15

Appendix A Provenance and batch selection

The appendix documents how reported batches were selected and shows all gate-passing batches alongside the real-score reference and uncertainty. Batch spread varied substantially across cell-reconstruction pairs with multiple passing batches.

Appendix A Provenance and batch selection

  • Across 13 cell-reconstruction pairs with multiple passing batches, within-configuration spread ranged from 0.00 to 8.35 rubric points, with median 1.45.Source: Appendix A · p. 17
  • When multiple batches passed the gate, the reported batch was the most recent by its logged start time; Table 7 lists every passing batch and marks the reported one.Source: Appendix A · p. 17, Table 7 · p. 18
  • Figure 5 plots the reported authored and default batches against each cell’s real mean and 95% bootstrap interval, with faint bars showing passing-batch ranges.A cell without a bar had a single passing batch; the figure notes that a marker inside the interval is not the text’s consistency test.Source: Figure 5 · p. 19

Appendix B Policy ranking

Policy-ranking measures require a real-robot reference to establish task order, but only two task orderings were confident. The reconstructions agreed with real on one and disagreed on the other, so MMRV is not reported.

Appendix B Policy ranking

  • The real-robot ordering of MolmoAct2 versus π0.5 was confident on only two of five tasks; the reconstructions matched real on one and disagreed on the other, so MMRV is omitted.Bootstrap probabilities that MolmoAct2 outscores π0.5 were 0.36 for bottles, 0.53 for bowls, 0.04 for blocks, 0.98 for latte, and 0.36 for clear table.Source: Appendix B · p. 19

Appendix C Reproduction

Appendix C documents how the reported tables and figures can be reproduced and identifies the released code, run artefacts, scores, and reconstruction assets.

Appendix C Reproduction

  • build_table.py builds Table 5 from the committed run list, while agreement.py produces Tables 4 and 6 and median_table.py provides a median check.Figures 3 to 5 are generated by scripts in figures/ from the same JSON.Source: Appendix C · p. 19
  • Both scoring scripts default to bootstrap seed 0, reproducing every interval in Tables 4 to 6.Source: Appendix C · p. 19
  • The paper releases the scoring scripts, trial runner, run artefacts, real per-trial scores, and assets and scenes for both reconstructions.The reported-run runner is identified by md5 3274dcbf; datasets are linked from the repository README.Source: Appendix C · p. 19

Funding and competing interests

Kaedim funded the real-robot trials and contributed to task and reconstruction design; Robocurve retained control of real-world methodology, scoring, and interpretation.

Funding and competing interests

  • Kaedim commissioned and paid for the real-robot trials under a benchmarking agreement with Robocurve.Source: Funding and competing interests · p. 20
  • Kaedim specified the three custom tasks and agreed the policies with Robocurve, which designed the rubrics and ran and graded the real trials.Source: Funding and competing interests · p. 20
  • Robocurve retained control of real-world methodology, scoring, and interpretation; the authors are Kaedim employees, and Kaedim pipelines produced the authored reconstruction.The passage cautions that naming Robocurve does not imply its endorsement of the results.Source: Funding and competing interests · p. 20

Related papers

Loading 2610.00731v2…