Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators

1 Introduction

• Real-world evaluation is costly, and studies commonly report success rates from only 20–40 trials, limiting reliable assessment across new environment distributions.Robot evaluation requires physical interaction, time, and human effort.Source: §1 · p. 2
• Simulation-to-real mismatches in visual features and contact physics can bias performance bounds based only on large-scale simulation.Policies may be sensitive to discrepancies such as lighting, textures, and friction coefficients.Source: §1 · p. 2
• SureSim uses prediction-powered inference to combine simulation predictions with a comparatively small set of real-world evaluations.Real evaluations serve as gold-standard labels; simulation evaluations provide predictions, and paired trials rectify bias.Source: §1 · p. 2
• The framework provides finite-sample-valid confidence intervals for mean policy performance, with tighter bounds than real-only trials when simulation is sufficiently predictive.The approach scales simulation rather than costly hardware evaluations.Source: §1 · p. 2
• 20–25% fewer hardware trials achieve similar bounds on policy performance using state-of-the-art physics-based simulators.The evaluation includes a single-task diffusion policy and multi-task fine-tuned π0.Source: §1 · p. 2

2 Related Work

• Real-world evaluation remains costly and difficult to scale for broad statistical coverage, despite standardized benchmarks and cloud-based or distributed evaluation efforts.Hardware trials require limited-parallelism physical interaction, outcome logging, and environment resets.Source: §2 · pp. 2–3
• Simulation benchmarks and world models offer cheaper evaluation, but setup costs, hallucinations, and dynamics mismatch limit their reliability as proxies.Simulation-based rankings have been reported to correlate with real-world performance, while the simulation-to-real gap prevents rigorous inference from simulation alone.Source: §2 · p. 3
• Small real-world trial counts limit statistical policy comparisons, while existing confidence-interval approaches for non-binary metrics can require large real-evaluation budgets.Prior work includes sequential comparisons and confidence intervals for binary success criteria.Source: §2 · p. 3
• SureSim differs from related inference and control-variate work by targeting finite-sample-valid bounds for manipulation policies and combining simulation with real evaluation in robotics.The paper notes that manipulation policies can be sensitive to small environmental perturbations and that robot and environment states are tightly interdependent.Source: §2 · p. 3

3 Problem Statement

• Mean policy performance is the expected outcome of a bounded evaluation metric over the target distribution of real-world environments.The environment distribution can span objects, tasks, environmental factors, and variations in object and robot poses.Source: §3 · pp. 3–4
• The metric is bounded in [0, 1], and the analysis assumes only measurability and boundedness of outcomes.No distribution family, density, or other structural assumptions are required.Source: §3 · pp. 3–4
• The target is a confidence interval containing the true mean with high probability for any finite number of real-world evaluations.Such intervals support policy decision-making and comparison.Source: §3 · p. 4
• Real-only non-asymptotic intervals can be computed from finite evaluations, but obtaining tight bounds is costly because hardware trials are expensive.This motivates augmenting a small real-world sample with a larger set of scalable simulation evaluations.Source: §3 · p. 4

4 Approach: Simulation to Augment Real Tests via Prediction Powered Inference

• A real2sim function maps each real environment to a corresponding simulation environment designed to match its dynamics and visual features.The simulator then evaluates the policy in that translated environment.Source: §4 · p. 4
• SureSim treats real policy outcomes as ground truth and composed real2sim-plus-simulator outcomes as predictions for the corresponding real environments.This enables prediction-powered inference with paired real and simulation evaluations.Source: §4 · pp. 4–5
• The evaluation pairs real outcomes with simulation predictions for a subset of environments and uses the remaining environments for additional simulation evaluations.The paired data estimate simulation bias, while the additional simulation data extend the prediction sample.Source: §4 · p. 5
• The uniform PPI estimator uses a rectifier to adjust simulation-prediction bias, and non-asymptotic WSR mean estimation produces the confidence interval.SureSim-UB combines SureSim and Classical intervals using a union bound.Source: §4 · p. 5
• The two-stage PPI variant combines a rectifier interval with an interval for additional simulations; its small rectifier sample can bloat the resulting bounds.The method computes the component intervals at separate significance levels and combines them using their Minkowski sum.Source: §4 · p. 5
• SureSim and its variants return finite-sample-valid confidence intervals for the target mean.The validity result follows under the stated sampling setup and the cited PPI results.Source: §4 · p. 5
• Classical applies WSR directly to real evaluations, while Control Variate is discussed for sim2sim but not used as a hardware baseline because finite-sample Type-I control is not established.The paper notes that small-sample variance estimates may be biased and need not upper-bound the true variance.Source: §4 · p. 6

5 Robot Experiments

• About 120 real objects, mostly toy kitchen items, approximate the target object distribution for paired real–simulation evaluation.Source: §5 · pp. 6–7
• Rectifier variance reflects paired real–simulation mismatch; the experiments use matched random seeds and average 20 nearby simulated initial conditions to mitigate it.Source: §5 · pp. 7–8

5.1 Real2Sim Robot Experiments

• Over 25% hardware-evaluation savings versus real-only evaluation are achieved by the SureSim method family.Source: Figure 5 · p. 8, §5.1 · p. 9
• At moderate correlation (ρ = 0.59), SureSim converges by N = 500 additional simulations and reduces real trials by over 20% on average.Source: Figure 6 · p. 9, §5.1 · pp. 9–10
• SureSim and SureSim-UB approach the rectifier lower bound within 5,000 additional simulations, while SureSim (2-Stage) approaches it progressively up to 50,000.Source: Figure 7 · p. 9, §5.1 · p. 10
• With low correlation around −0.05, scaling simulations does not improve estimate variance, and the methods do not beat Classical.Source: §5.1 · p. 10, Figure 8 · p. 10

5.2 Sim2Sim Experiments

• In Sim2Sim tests with paired correlation around 0.97 and 400 heldout environments, all methods beat Classical at α = 0.1.Source: §5.2 · p. 11
• Empirical coverage for Control Variate can degrade as simulation samples increase, cautioning against interpreting narrow intervals without Type-1 error control.Source: §5.2 · p. 11, Figure 11 · p. 11

• SureSim augments large-scale simulation with fewer real-world evaluations to provide non-asymptotically valid inferences on policy performance.Source: §6 · p. 12
• The real2sim formalism frames combined real and simulation evaluations as prediction-powered inference and uses finite-sample-valid mean-estimation algorithms.Source: §6 · p. 12
• SureSim evaluates diffusion policy and π0 and saves over 20–25% of hardware evaluations on average versus hardware-only evaluation.Source: §6 · p. 12

7 Limitations and Future Work

• The simulation–real gap remains a challenge to the effectiveness of combining real and simulation evaluations.The section frames the following research directions around that gap.Source: §7 · p. 12
• Paired evaluations require real and simulation trials, yet real outcomes are difficult to predict and repeated trials can be noisy.Noise can reflect inherent stochasticity and policy sensitivity to hard-to-control perturbations such as lighting or object placement.Source: §7 · p. 12
• Physics-based simulations require substantial setup effort and are difficult to scale or adapt to new environments and tasks.The section proposes empirically assessing paired-evaluation correlation in action-conditioned video models as they improve.Source: §7 · p. 12
• Random batch sampling is the current approach; active sampling that prioritizes regions with larger real–simulation gaps is a proposed extension.Source: §7 · p. 12
• Scaling evaluation will require further investment in repositories of diverse manipulation tasks and environments, as well as scalable real2sim pipelines.Source: §7 · p. 12

A Summary Statistics from Experiments

• Table 1 reports paired-trial count n, additional simulation count N, Pearson correlation ρ, and sample real mean, averaged over 100 data draws.Source: Table 1 · p. 16

B Algorithms

• UniformPPI and 2-StagePPI use paired and simulation datasets to construct confidence intervals, while WSR supplies non-asymptotic mean intervals for bounded data.The algorithm excerpts also specify normalizing observations to [0, 1] and using c = 0.99 in the martingale computation.Source: B Algorithms · pp. 16–17

C Choosing the Confidence For Each Interval

• The risk allocation between rectifier and simulator intervals can be approximately optimized, with practical optimal allocations generally having δ ≈ 0.9α.The section describes finding an approximate equilibrium by binary search.Source: C Choosing the Confidence For Each… · p. 17

D Evaluation on Artificial Data

• The artificial-data evaluation uses simulated i.i.d. draws of scalar random variables with known statistical properties, rather than physics-based robot-simulator data.Source: D Evaluation on Artificial Data · p. 17

D.1 Value of Artificial Data and Research Questions

• Known parameter values in artificial data let researchers check whether generated confidence intervals contain the true mean, which cannot be verified on real data.The true labels also support hundreds or thousands of estimation problems across varying contexts.Source: §D.1 · p. 18
• Artificial data cannot represent and validate against every distribution practitioners may encounter, even with finite-time sampling.The passage identifies addressing this representativeness problem as important for effective characterization and evaluation.Source: §D.1 · p. 18
• The planned analyses cover data-generation generality, informative metrics, method effectiveness under parameter variation, and guidance on method strengths and use cases.Source: §D.1 · p. 18

D.2 The Key Parameters and Data Generation

• The artificial-data study varies significance level α, real sample size n, proxy sample size N, and proxy usefulness, measured through correlation ρ.The constructed real data are bounded in [0, 1], with a tunable population mean; proxy data are iteratively adjusted to target correlation.Source: §D.2 · p. 18
• Variance reduction in the unbiased mean estimator is presented as the key way proxy variables improve finite-sample confidence intervals.Pearson correlation ρ is used as an effectiveness measure because it directly relates real-to-proxy covariance to their geometric mean variance.Source: §D.2 · p. 18
• Interval width is the primary metric because estimation aims to minimize the uncertainty region containing the true parameter at a chosen error risk.Source: §D.2 · p. 19
• Estimator variance is interchangeable with interval width only when the estimator space is constrained to unbiased estimators.The passage cautions that variance minimization can be misleading when this constraint is not satisfied.Source: §D.2 · p. 19
• Bounded random variables are sub-Gaussian, so optimal interval widths across data scales should attain poly-logarithmic dependence on significance level.Source: §D.2 · p. 19

D.3 Results on Artificial Data

• Artificial-data means were set to 0.5 to reduce interval-truncation noise; these experiments test existential, not universal, algorithm behavior.Source: §D.3 · p. 19
• With fixed paired-trial n, more simulation tightens intervals but cannot eliminate rectifier uncertainty; SureSim recovers the paired-trial gap while two-stage variants remain inefficient at N = 10000.The intervals do not asymptote to zero as simulation data grows.Source: §D.3 · p. 19
• Control Variate intervals can be narrower at large α, but their widths are more sensitive to α, and that apparent advantage can coincide with miscoverage.Source: Figure 12 · p. 20, Figure 14 · p. 22
• Higher real–simulation correlation increases achievable interval tightening, while Figure 13 also shows Control Variate’s greater sensitivity to correlation and cost in low-correlation regimes.Source: §D.3 · pp. 19, 21, Figure 13 · p. 21
• Figure 14 shows valid nonasymptotic methods cover across regimes, whereas Control Variate miscovers as simulation grows because its estimator is biased.The reported miscoverage occurs as Control Variate intervals become narrower than the proposed methods’ intervals.Source: §D.3 · p. 21, Figure 14 · p. 22
• At α < 0.0001 and constrained n, Control Variate often yields vacuous intervals; Hoeffding-type tightening sacrifices generality and requires boundedness.Source: §D.3 · p. 22

D.4 Key Takeaways

• Control Variate is relatively stronger at higher α and ρ, but in small-n settings an optimistic coefficient choice can lead to miscoverage.Source: §D.4 · pp. 22–23

E Implementation Details

• Hardware experiments use a Franka Panda with 15 Hz joint-space control, a third-person webcam, a wrist camera, and VR teleoperation.Source: E Implementation Details · p. 23

E.1 Policy Implementation Details

• Diffusion Policy uses wrist and third-person images plus robot state, outputs joint-position and gripper targets, and is trained for 50,000 updates with batch size 64.The hyperparameter table is identified as Table 2.Source: §E.1 · p. 23, Table 2 · p. 23
• Multi-task π0 is fine-tuned from π0-base with a task instruction, freezing the VLM and training only the action expert for 15,000 steps with batch size 64.Training uses the instruction format “Put <object> into the box”.Source: §E.1 · p. 23

E.2 Paired and Additional Simulation Data

• Figure 15 identifies the objects used to train the multi-object policy, and Figure 16 presents simulation success scores for objects used in paired and additional simulations.Source: Figure 15 · p. 23, Figure 16 · p. 24
• Paired real evaluations and additional simulations are assumed i.i.d. from the same environment distribution, with real-evaluation objects treated as representative random samples of kitchen objects.The authors note that operationalizing this pipeline requires community-wide investment in large evaluation datasets.Source: §E.2 · p. 23