PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions

Kaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Xiaochen Hu, Changxi Zheng, Yunzhu Li

arXiv:2511.04665v2cs.ROcs.CVcs.LG

TL;DR

Real-world evaluation of manipulation policies is costly and difficult to reproduce, while existing simulators often miss the coupled visual and physical complexity of soft-body interactions. The paper combines Gaussian Splatting with video-optimized soft-body digital twins and finds strong agreement between simulated and real-world policy outcomes across representative tasks. This supports simulation as a reproducible proxy for policy evaluation, checkpoint selection, and performance estimation within the studied scope.

  • Problem

    Real-world policy evaluation is costly, time-consuming, and difficult to reproduce, while existing simulators provide limited fidelity for soft-body interactions.

    Source: §I · p. 1

  • Method

    The framework combines photorealistic Gaussian Splatting with physics-informed soft-body digital twins reconstructed from interaction videos.

    Source: A. Problem Definition · p. 2, §V · p. 7

  • Results

    Across representative deformable and rigid manipulation tasks, simulated policy success rates strongly correlate with real-world outcomes, reaching r > 0.9.

    Source: §V · p. 7

  • Takeaways & Limitations

    The simulator can serve as a practical proxy for checkpoint selection and performance estimation, while physics optimization and color alignment are critical to closing performance gaps.

    Source: §V · p. 7, E. Ablation Study · p. 7

Abstract

from arXiv · show

Robotic manipulation policies are advancing rapidly, but their direct evaluation in the real world remains costly, time-consuming, and difficult to reproduce, particularly for tasks involving deformable objects. Simulation provides a scalable and systematic alternative, yet existing simulators often fail to capture the coupled visual and physical complexity of soft-body interactions. We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos and renders robots, objects, and environments with photorealistic fidelity using 3D Gaussian Splatting. We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing, demonstrating that simulated rollouts correlate strongly with real-world execution performance and reveal key behavioral patterns of learned policies. Our results suggest that combining physics-informed reconstruction with high-quality rendering enables reproducible, scalable, and accurate evaluation of robotic manipulation policies. Website: https://real2sim-eval.github.io/

I. INTRODUCTION

The paper addresses costly, poorly reproducible real-world policy evaluation by combining photorealistic Gaussian Splatting with physics-informed soft-body digital twins. It positions this framework as a way to narrow visual and dynamics gaps and improve simulation’s reliability as an evaluation proxy.

I. INTRODUCTION

  • Real-world policy evaluation is slow, expensive, and difficult to reproduce, creating a bottleneck for rapid robotics development and benchmarking.Source: §I · p. 1
  • The framework combines Gaussian Splatting for photorealistic scene rendering with PhysTwin-based deformable-object reconstruction for realistic dynamics.The simulator also incorporates automatic alignment and exposes the resulting environment through a Gym-style interface.Source: §I · pp. 1–2
  • The authors evaluate representative rigid- and soft-body tasks with ACT [1], Diffusion Policy, SmolVLA, and Pi-0, comparing simulated and real-world success rates.Source: §I · p. 2
  • The paper reports strong cross-task correlation, uses ablations to assess rendering and dynamics fidelity, and offers design guidance for simulation-based policy evaluation.Source: §I · p. 2

A. Robot Policy Evaluation

III. METHOD

The method formulates policy evaluation as matching simulated and real-world outcomes through accurate dynamics and appearance models. It combines PhysTwin system identification with Gaussian Splatting reconstruction and alignment to address both axes of simulation fidelity.

A. Problem Definition

  • Policy evaluation compares paired simulation and real-world performance across policies, with fidelity assessed through dynamics accuracy and rendered-observation realism.The simulator models state transitions as st+1 = f(st,at) and observations as ot = g(st).Source: A. Problem Definition · pp. 2–3

C. Real-to-Sim Gaussian Splatting Simulation

B. Preliminary: PhysTwin

  • PhysTwin represents objects as dense spring-mass systems and optimizes physical parameters from human-object interaction videos to match observed motion.For rigid objects, stiffness is fixed high enough to suppress deformation.Source: B. Preliminary: PhysTwin · p. 3

1) GS Construction:

The simulator builds Gaussian Splatting scenes, aligns reconstructed components to robot and object frames, matches rendered colors to real cameras, and propagates physics-based deformation into rendering.

1) GS Construction:

  • Scene construction scans the workspace and experimental objects, segments robot, objects, and background, and requires reconstruction only once per task.Source: 1) GS Construction · p. 3

2) Positional Alignment:

  • Positional alignment maps Gaussian reconstructions to robot, object, and background reference frames using relative 6-DoF transforms and ICP/RANSAC robot registration.Nearest-neighbor link assignment enables rendering of continuous arm and gripper motion.Source: 2) Positional Alignment · pp. 3–4

3) Color Alignment:

  • Color alignment optimizes a transformation between rendered Gaussian Splatting images and paired real-camera images to reduce camera-domain color mismatch.The method uses polynomial transformations and finds that a quadratic transform offers the best trade-off between expressivity and overfitting.Source: 3) Color Alignment · p. 4

4) Physics and Deformation:

  • The physics system extends PhysTwin with robot and environment collisions, frictional soft-body grasping, force-threshold gripper stopping, and deformation-aware kernel updates.Rigid kernels use rigid transforms, while deformable kernels use Linear Blend Skinning.Source: 4) Physics and Deformation · p. 4
  • The simulator runs at 5 to 30 FPS on a single GPU and evaluates policies several times faster than real-world execution.Source: 4) Physics and Deformation · p. 4

D. Policy Evaluation

The evaluation compares real-data-trained imitation policies in simulation and reality across plush-toy packing, rope routing, and T-block pushing. It uses controlled initial configurations, binary success rates, and training-curve comparisons to assess correlation and practical policy-selection value.

D. Policy Evaluation

  • Policies are trained exclusively on real demonstrations and evaluated in both domains from fixed initial object configurations to reduce variance.Source: D. Policy Evaluation · p. 4
  • Performance is measured with binary task success rates, using human judgments in reality and automatically computed privileged-state criteria in simulation.The simulator is packaged through the Gym environment API.Source: D. Policy Evaluation · p. 4

IV. EXPERIMENTS

  • The experiments ask whether simulation correlates with real performance, whether rendering and dynamics fidelity matter, and what practical benefits correlation provides.Source: §IV · p. 4

1) Tasks:

  • The benchmark covers plush toy packing, rope routing, and T-block pushing, spanning deformable and rigid manipulation with visually demanding success conditions.Packing and routing require tight geometric tolerances, while push-T requires visual pose inference for translation and reorientation.Source: 1) Tasks · pp. 4–5

IV. EXPERIMENTS

  • Figure 3 compares simulation and real-world success rates across multiple policies and checkpoints, while Figure 4 tracks corresponding performance curves throughout training.Source: Figure 3 · p. 5, Figure 4 · p. 5

2) Evaluation:

The evaluation uses controlled initial states, standard correlation metrics, and multiple imitation-learning policies to compare simulation with real-world performance. Across tasks and architectures, the simulator strongly tracks real outcomes, while remaining discrepancies reflect visual, dynamic, and sampling gaps.

2) Evaluation:

  • The evaluation compares four imitation-learning policies using fixed shared configurations and binary success criteria across three manipulation tasks.Evaluation sets contain 20, 27, and 16 states for toy packing, rope routing, and push-T, respectively.Source: 2) Evaluation · p. 5

C. Sim-and-Real Correlation

  • Simulation achieves Pearson correlation r > 0.9 for all policies, whereas IsaacLab reaches r = 0.649 on push-T and r = 0.237 on rope routing.The lower IsaacLab rope-routing correlation is attributed to a larger dynamics gap.Source: C. Sim-and-Real Correlation · p. 6
  • Policies with higher real-world success rates also achieve higher simulator success rates consistently across architectures and tasks.Table I compares MMRV and Pearson correlation, with lower MMRV indicating more accurate policy ranking and higher r indicating stronger correlation.Source: C. Sim-and-Real Correlation · p. 6, Table I · p. 6

D. Policy Performance Analysis

  • Simulation success rates generally follow real-world learning curves, including peaks at iteration 5,000 for toy packing-DP and around iteration 20,000 for rope routing-Pi-0.These curves support using simulation to monitor learning dynamics and select checkpoints for real-world testing.Source: D. Policy Performance Analysis · p. 6
  • When absolute simulation and real success rates differ, the simulator can still capture the correct performance trend, but residual appearance, dynamics, and episode-count gaps remain.The cited examples are toy packing-SmolVLA and rope routing-ACT, with 16–27 evaluation episodes per checkpoint.Source: D. Policy Performance Analysis · p. 6

E. Ablation Study

The ablation study shows that both appearance and dynamics fidelity are necessary for reliable policy evaluation. The conclusion reports strong real-to-sim correlation while identifying scale-up to broader task and policy sets as future work.

E. Ablation Study

  • Without physics optimization, object dynamics become inaccurate, whereas without color alignment, rendered images show clear color mismatches.The full method uses task-specific optimized dynamics and aligned appearance, producing rollouts closer to real outcomes.Source: E. Ablation Study · pp. 6–7
  • Dynamics mismatches can cause rope-routing failure, while color mismatches can alter perception and change how the robot pushes the block.The examples show that policies have different sensitivities to appearance and dynamics gaps.Source: E. Ablation Study · p. 7
  • The full method achieves the highest correlation values, outperforming variants without color alignment or physics optimization.Table I defines lower MMRV as fewer ranking errors and higher r as stronger statistical correlation.Source: E. Ablation Study · p. 7

V. CONCLUSION

  • Across representative deformable and rigid-body tasks, simulated policy success rates correlate strongly with real-world outcomes at r > 0.9.The analysis also supports simulation as a proxy for checkpoint selection and performance estimation.Source: §V · p. 7
  • Scaling simulation and evaluation to larger task and policy sets is identified as future work for deeper analysis of policy-evaluation design choices.The framework is also described as generalizable to more diverse environments and increasingly complex manipulation tasks.Source: §V · p. 7

APPENDIX

The appendix describes the robot, data-collection setup, and three manipulation tasks used to evaluate policies on rigid and deformable objects.

1) Robot Setup:

  • The setup uses a 7-DoF xArm 7 with interchangeable gripper or pusher end-effectors and fixed and wrist-mounted RealSense cameras.Source: 1) Robot Setup · p. 10

2) Data Collection:

  • GELLO tracks commanded joint angles with normalized joint-velocity control, and the same strategy is used in real and simulated evaluation.Source: 2) Data Collection · p. 10
  • Figure 6 visualizes training and evaluation state coverage; Table II summarizes simulation assets; Figure 7 shows color alignment; and Figure 8 presents representative PhysTwin training-video frames.Source: Figure 6 · p. 10, Table II · p. 10, Figure 7 · p. 11

3) Task Definition:

  • The datasets include 39 toy-packing, 56 rope-routing, and 60 T-block-pushing human demonstration episodes.Source: 3) Task Definition · pp. 10–11
  • Toy packing requires adapting a plush toy’s pose to place all limbs inside a compact box.Source: 3) Task Definition · p. 10
  • Rope routing depends on rope dynamics because the contact point with the holder is offset from the grasp location.Source: 3) Task Definition · p. 10
  • T-block pushing randomizes the block’s initial position and orientation while keeping the pusher height fixed and targeting a visual goal region.Source: 3) Task Definition · pp. 10–11

B. Simulation

The simulation combines Gaussian Splatting assets with physically simulated object representations, aligning robot and scene geometry in a unified coordinate frame.

1) Assets:

  • Each simulation asset is a Gaussian Splatting scan paired with a dynamics type and a 3D representation for physical simulation and rendering.Source: 1) Assets · p. 11

2) Positional Alignment:

  • Robot-scene alignment combines manual coarse registration, robot/scene separation, and ICP between robot Gaussian centers and URDF surface points.Source: 2) Positional Alignment · p. 11
  • The resulting rigid transformation is applied to the entire Gaussian Splatting reconstruction so robot and scene share one coordinate frame.Source: 2) Positional Alignment · p. 11
  • Algorithm 1 takes PhysTwin particle states, spring parameters, robot motion and mesh geometry, then advances them through spring, self-collision, robot-collision, fixed-mesh, and ground-collision steps.Source: 2) Positional Alignment · p. 11

3) Color Alignment:

The framework aligns rendered colors with RealSense images, trains deformable-object digital twins from RGB-D videos, and updates rendering within the simulation loop.

3) Color Alignment:

  • Robust IRLS with Tukey bi-weight estimates a color transformation from five 848×480 images, with RGB-norm weighting and 50 optimization iterations.Source: 3) Color Alignment · p. 11
  • Figure 7 compares real images, raw Gaussian Splatting renderings, and color-aligned renderings to show the alignment process.Source: 3) Color Alignment · p. 11

4) PhysTwin Training:

  • PhysTwin reconstructs object geometry and optimizes physical parameters from a single multi-view RGB-D video.Source: 4) PhysTwin Training · p. 12
  • For the rigid T-block, uniformly sampled mesh points are connected with springs using radius 0.5, at most 50 neighbors, and stiffness 3 × 10^4.Source: 4) PhysTwin Training · p. 12

5) Simulation Loop:

  • The simulation loop combines robot action processing, PhysTwin simulation, collision handling, and renderer updates.Source: 5) Simulation Loop · p. 12

1) Datasets:

  • Figure 6 visualizes the spatial coverage of initial states in the training and evaluation datasets.Source: 1) Datasets · p. 12

2) Normalizations:

The policy-training pipeline specifies normalization, augmentation, model configurations, and a protocol for matching simulated and real initial states during evaluation.

2) Normalizations:

  • Mean–std normalization standardizes features to zero mean and unit variance, while min–max normalization scales dimensions to [−1,1].Source: 2) Normalizations · p. 12, Table V · p. 12
  • VLA policies use relative SE(3) end-effector actions with rolling-window statistics computed over each action chunk.Source: 2) Normalizations · p. 12

3) Image Augmentations:

4) Hyperparameters:

1) Evaluation Protocol:

  • Evaluation rolls out policies from matched fixed initial states in simulation and reality, using rendered frames and a live overlay to guide manual physical-state adjustment.Source: 1) Evaluation Protocol · p. 12

2) Episode Settings:

Evaluation uses controlled initial configurations, automated task-specific success criteria, and expanded randomized trials to assess simulation reliability and uncertainty.

3) Success Criteria:

  • Automated success criteria replace tedious, potentially biased human scoring with simulation-state metrics for all three tasks.Source: 3) Success Criteria · pp. 12–13
  • Toy packing succeeds when over 3,050 of 3,095 mass particles occupy the box region for more than 30 final-window frames.Source: 3) Success Criteria · p. 13
  • Rope routing succeeds when more than 100 spring segments cross both clip openings for over 30 final-window frames.Source: 3) Success Criteria · p. 13
  • T-block pushing succeeds when the mean squared particle distance from the target state remains below 0.002 for over 30 final-window frames.Source: 3) Success Criteria · p. 13

A. Scaling up Simulation Evaluation

B. Replaying Real Rollouts

Replay evaluations reuse real-world commands and initial states to test outcome correspondence, while synchronized visualizations assess trajectory-level agreement across tasks.

B. Replaying Real Rollouts

C. Additional Qualitative Results

Loading 2511.04665v2…