Source-linked AI summary
YUBI-Real2Sim: 3DGS Scene Verification for Reliable Policy Evaluation
Hiroki Nishizawa, Takehiko Ohkawa, Nicola Scianca, Tatsuya Matsushima, Shigeo Morishima, Kei Ota
TL;DR
Real-robot policy evaluation requires repeated executions and physical resets, while YUBI’s independently moving fisheye wrist cameras make reliable scene-based policy selection an important evaluation setting. YUBI-Real2Sim combines scene reconstruction, object modeling, and simulated policy evaluation, reproducing clean-scene rankings in 8/9 comparisons while quality-based reliability prediction remains limited.
Problem
Repeated real-robot executions and physical resets limit policy and environment comparisons, and YUBI’s moving fisheye wrist cameras require evidence that reconstructed scenes support reliable policy selection.
Source: §I · p. 2Method
YUBI-Real2Sim integrates scene reconstruction and dual-fisheye policy evaluation on two bimanual platforms, using simulated wrist-camera rendering and MuJoCo.
Source: §I · p. 2, §III · p. 4Results
Clean simulation matches real policy rankings in 8/9 scene–task comparisons; agreement is 7/9 under holes and floaters and 6/9 under Random 30% removal, while OBS-based NVS-SQA helps Cup but not aggregate accuracy.
Source: §VI · p. 10, §V-E · p. 10Takeaways & Limitations
Real-ranking agreement, statistical separation between policies, and quality-based prediction of ranking reliability are distinct evaluation outcomes.
Source: §VI · p. 10Quality scores require separate validation before they are used as a scene-rejection rule.
Source: §I · p. 2
Abstract
from arXiv · showhide
Evaluating VLA policies on real robots is costly, time-consuming, and difficult to reproduce, making real-to-sim simulation essential for scalable evaluation. While 3D Gaussian Splatting (3DGS) enables rapid real-to-sim scene generation, the reconstruction quality required for reliable policy evaluation remains unquantified. In particular, policies relying on wrist-mounted fisheye cameras (e.g., YUBI handheld gripper data) are heavily exposed to visual artifacts (holes, floaters) in wide-angle views, directly degrading task performance. To address this, we present YUBI-Real2Sim, a unified framework for 3DGS-based scene reconstruction and real-to-sim policy evaluation under dual fisheye wrist views. We develop an integrated pipeline combining 200-degree fisheye camera capture, 3DGS training, 3D foundation models for object modeling, and MuJoCo-based physical simulation, reproducing the observed real-world policy ordering in eight of nine scene-task comparisons. Using this framework, we construct a benchmark across two bimanual robot platforms, three scenes, three tasks, and dual pi0.5-based policies. This systematically measures real-to-sim rank agreement and evaluates policy sensitivity against counterfactual 3DGS artifacts (ceiling holes, floaters, warping) and capture disturbances. We further analyze whether novel-view quality scores can indicate policy-ranking reliability under a scene-held-out evaluation protocol. The results show task-dependent performance and limitations of quality-based reliability prediction, highlighting the need to assess reconstruction quality when using Real2Sim for policy evaluation. These results do not yet establish a generally reliable scene-acceptance rule. Project Page: https://yubi-real2sim.airoa.io.
I. Introduction
YUBI-Real2Sim addresses the need to evaluate whether reconstructed scenes preserve useful policy comparisons, especially for policies using moving fisheye wrist cameras. Its benchmark evaluates clean-scene agreement, reconstruction artifacts, and image-quality metrics as possible indicators of ranking reliability.
I. Introduction
- Real-robot evaluation requires repeated executions and physical resets, while the practical decision is which of two candidate policies to deploy.The relevant outcome for that decision is whether simulation preserves the real-world policy ranking.Source: §I · p. 2
- Dual, independently moving fisheye wrist cameras expose YUBI policies to wide-angle views that include regions beyond the manipulation target.Their viewpoints change during execution, motivating a simulator and evidence that scene reconstruction supports policy selection.Source: §I · p. 2
- 88.9% of real policy rankings are reproduced by clean simulation across three scenes and three tasks.The benchmark compares two π0.5-based candidate policies per task and includes Cup, Tape, and Pen.Source: §I · p. 2
- The benchmark spans two bimanual platforms and evaluates two π0.5-based candidate policies across three scenes and three tasks.The platforms are dual Franka FR3 and TX-G2; the tasks are Cup, Tape, and Pen.Source: §I · p. 2
- The benchmark tests whether ceiling holes, floaters, and warping in the 3DGS background alter policy rankings under fixed physics, objects, and rollout conditions.Reduced-coverage reconstructions separately test whether limited image support produces analogous visible defects.Source: §I · p. 2
- The paper analyzes whether PSNR, SSIM, LPIPS, and NVS-SQA relate to agreement between real and simulated policy rankings.It treats these image-quality measures as candidate indicators of ranking reliability.Source: §I · p. 2
II. Related Work
Prior real-to-sim work evaluates policy comparisons in reconstructed environments, but a validated reconstruction-quality metric for identifying scenes that support reliable comparisons remains an open gap. This paper focuses on pairwise policy ranking and tests visual reconstruction and novel-view metrics against that outcome.
Real-to-sim policy evaluation
- SIMPLER evaluates whether visually matched simulation preserves policy performance comparisons, while PolaRiS and SimFoundry extend evaluation to reconstructed scenes.The latter systems include mesh objects and Gaussian-splat backgrounds.Source: §II · p. 2
- This work measures whether changing visual reconstruction alters which of two policies simulation selects for the same task and scene.Its pairwise ranking measure is distinct from success-rate correlation and mean maximum rank violation.Source: §II · pp. 2–3
Evaluating novel-view synthesis
- A validated reconstruction-quality metric for identifying scenes that support reliable policy comparisons remains missing, and NVS metrics are candidate diagnostics.The relevance of novel-view measures depends on the viewpoints evaluated; rendering quality alone does not establish ranking reliability.Source: §II · p. 3
- YUBI’s independently moving fisheye wrist cameras may observe viewpoints poorly supported by capture, making interpolation-only holdouts an incomplete match to the evaluation setting.Spatial-sector holdouts reveal a 3–12 dB interpolation–extrapolation gap that can reverse reconstruction-method rankings.Source: §II · p. 3
III. YUBI-Real2Sim System
YUBI-Real2Sim combines static-scene reconstruction, separate object modeling, and closed-loop policy evaluation using dual fisheye wrist observations. The system registers the reconstructed background with MuJoCo and composites background and object renders into calibrated wrist-camera images.
III. YUBI-Real2Sim System
- The system combines environment capture, static-scene reconstruction, object modeling, and closed-loop policy evaluation under dual fisheye wrist observations.Policies receive rendered wrist images and proprioception, then return action chunks that drive the simulated robot.Source: §III · p. 3
Evaluation setting
- The evaluation covers three scenes on two bimanual platforms and three tasks: Cup, Tape, and Pen.Duo and Flat use dual Franka FR3; TX-G2 uses a TX-G2 platform, with a fisheye camera on each wrist.Source: §III · p. 4
Capture
- A single Insta360 left fisheye lens provides broad angular capture coverage, with temporally selected frames chosen by maximum minimum left/right Laplacian variance.The same source timestamps support comparison of fisheye-derived projections with a central pinhole view.Source: §III · p. 4
Reconstruction
- Each source frame is reprojected into nine pinhole views, then COLMAP poses and an OpenMVS seed initialize full-resolution Gaussian Wrapping reconstruction.The background is registered to MuJoCo coordinates; task objects are modeled separately as meshes with CoACD collision geometry.Source: §III · p. 4
Simulation and rendering
- MuJoCo renders the robot and objects as meshes while 3DGS renders the static background, with depth compositing forming calibrated fisheye wrist images.Camera poses are held fixed across policies, tasks, and scene perturbations; grasps use a contact rule that attaches objects when both inner finger surfaces touch them.Source: §III · p. 4
Policy interface
- The π0.5 policies receive two fisheye wrist images and proprioception, then predict left-hand-referenced end-effector delta actions and gripper opening.Actions are predicted in chunks of 32, with 16 executed per Cup chunk and 8 per Tape and Pen chunk on real and simulated robots.Source: §III · p. 4
IV. Benchmark Protocol
The benchmark compares two task-specific policies across three scenes and three tasks, then specifies controlled reconstruction conditions and metrics for evaluating policy rankings and scene quality.
IV. Benchmark Protocol
- The benchmark covers Duo and Flat on dual Franka FR3 and TX-G2, with Cup, Tape, and Pen tasks and two task-specific π0.5-based candidates per task.Each of the nine scene–task pairs contributes one binary policy comparison.Source: §IV · pp. 4–5
- Each policy cell uses 20 real and 80 simulated rollouts, with matched layouts and shared initial-condition schedules across policies and reconstruction conditions.Task time limits are one or three minutes, depending on the scene–task pair.Source: §IV · p. 5
IV-A. Evaluation conditions
- Artifact conditions edit only background Gaussians, while reduced-coverage conditions retrain 3DGS with fewer source images for defect and ranking analyses.Clean, edited, and reduced-coverage scenes are compared at the same camera poses; artifact evaluations hold physics, objects, and rollout conditions fixed.Source: §IV-A · pp. 5–6
IV-B. Metrics and reporting
- Pairwise ranking accuracy counts correct simulated policy choices over eligible real-ranked comparisons; reversals and simulation ties score zero, unlike changes in success rate alone.The comparison set is fixed before perturbation testing, and absolute-rate error is reported separately from ranking accuracy.Source: §IV-B · p. 7
- 8/9 scene–task rankings (88.9%) agree between real and simulated evaluation; Flat Cup is reversed.Figure 4 reports the comparison across three tasks and three scenes.Source: Figure 4 · p. 8
- Scene-held-out verification predicts ranking disagreement, including ties, from scene-level quality scores using task-specific thresholds selected on the other two scenes.NVS-SQA is evaluated on Test and OBS views; PSNR, SSIM, and LPIPS are reported for aligned-reference Test views.Source: §IV-B · pp. 7–8
V. Policy Ranking and Reconstruction Quality
The experiments test whether clean simulation preserves real-robot policy choices, whether reconstruction defects alter those choices, and whether image-quality scores identify unreliable rankings.
V-A. Clean policy ranking
- Clean simulation reproduces real policy rankings in 8 of 9 scene–task comparisons (88.9%), with one inversion in Flat/Cup.The reported clean mean absolute error is 31.3889 percentage points (0.3139 on the unit-interval scale), a distinct outcome from ranking agreement.Source: §V-A · pp. 8–9
V-B. Counterfactual artifacts
- Holes and floaters reduce ranking accuracy to 7/9 (77.8%), while warping preserves 8/9; significance status can change even when rankings do not.Hole 90° removes significance in four comparisons; warping retains the TX-G2 Tape ranking while making its difference significant (14/80 versus 5/80, p = 0.048).Source: §V-B · p. 9, Table I · p. 9
V-C. Reconstruction from reduced image coverage
- Reduced coverage yields 8/9 agreement for Central pinhole, 7/9 for Contiguous 30% removal, and 6/9 for Random 30% removal.Central pinhole preserves all Clean rankings but removes policy success-rate significance in five comparisons; all reduced-coverage conditions use Clean’s per-task time limits.Source: §V-C · p. 9
V-D. Quality response to view coverage
- Across view-coverage disturbances, NVS-SQA changes are scene- and mode-dependent and small relative to the 0.224 spread between scenes’ clean scores.For example, Contiguous 30% removal changes held-out Test-view means by −0.060 for Duo, +0.031 for Flat, and −0.018 for TX-G2.Source: §V-D · p. 10
V-E. Scene verification across tasks and scenes
- In scene-held-out classification, SSIM has the highest pooled task-wise accuracy (32/45), while Test NVS-SQA leads all-three reliability decisions (13/15).OBS NVS-SQA improves Cup decisions but not aggregate accuracy (29/45 versus 30/45 for Test); the always-reliable predictor scores 35/45, exceeding all five quality-based methods. Pen’s perfect score reflects constant predictions, and every quality method misses the three Tape disagreements, all in TX-G2.Source: §V-E · p. 10, Table II · p. 10
VI. Conclusion
YUBI-Real2Sim tests whether reconstructed scenes support policy evaluation without target-scene real-robot verification. Clean rankings often agree with real evaluation, but artifacts and quality-based reliability prediction expose distinct limitations.
VI. Conclusion
- Clean simulation agrees with real policy rankings in 8/9 scene–task comparisons.Agreement falls to 7/9 under holes and floaters and 6/9 under Random 30% removal.Source: §VI · p. 10
- Unchanged rankings can still lose statistical separation between policies, so ranking agreement and significance are distinct evaluation outcomes.Source: §VI · p. 10
- OBS-based NVS-SQA improves Cup decisions, but the benefit does not extend to the aggregate three-task result.The conclusion distinguishes ranking agreement, policy statistical separation, and quality-based reliability prediction as separate outcomes.Source: §VI · p. 10