PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison

David Snyder, Apurva Badithela, Nikolai Matni, George Pappas, Anirudha Majumdar, Masha Itkina, Haruki Nishimura

arXiv:2603.13616v1cs.ROstat.AP

TL;DR

Policy comparisons in robotics need to make reliable decisions from costly, noisy evaluations, while common binary measures can be limited for informative comparisons. N-SCORE uses sequential safe, anytime-valid inference to compare policies across progress metrics; the paper reports evaluation-burden savings and identifies assumptions and risk-budget conservatism that bound its use.

Abstract

from arXiv · show

Generalist robot manipulation policies are becoming increasingly capable, but are limited in evaluation to a small number of hardware rollouts. This strong resource constraint in real-world testing necessitates both more informative performance measures and reliable and efficient evaluation procedures to properly assess model capabilities and benchmark progress in the field. This work presents a novel framework for robot policy comparison that is sample-efficient, statistically rigorous, and applicable to a broad set of evaluation metrics used in practice. Based on safe, anytime-valid inference (SAVI), our test procedure is sequential, allowing the evaluator to stop early when sufficient statistical evidence has accumulated to reach a decision at a pre-specified level of confidence. Unlike previous work developed for binary success, our unified approach addresses a wide range of informative metrics: from discrete partial credit task progress to continuous measures of episodic reward or trajectory smoothness, spanning both parametric and nonparametric comparison problems. Through extensive validation on simulated and real-world evaluation data, we demonstrate up to 70% reduction in evaluation burden compared to standard batch methods and up to 50% reduction compared to state-of-the-art sequential procedures designed for binary outcomes, with no loss of statistical rigor. Notably, our empirical results show that competing policies can be separated more quickly when using fine-grained task progress than binary success metrics.

I. INTRODUCTION

Robot policy design choices require empirical comparison, but hardware evaluation is costly, binary success can hide meaningful differences, and outcomes are noisy. N-SCORE addresses gaps in prior comparison methods by combining statistical rigor, sample efficiency, and support for general performance measures.

I. INTRODUCTION

  • Design changes require rigorous empirical comparison because their effects cannot be determined from first principles, while hardware evaluation is costly and binary success can conceal partial task progress.Observed performance is also noisy because environment configuration and policy action selection are uncertain.Source: §I · p. 1
  • N-SCORE is introduced as a sequential policy-comparison method for general progress metrics, designed to combine rigor, sample efficiency, and generality.The paper reports Type-1 Error control and evaluation-burden savings of up to 70% over batch methods and 50% over methods using binary success measures.Source: §I · pp. 1–2

A. Robot Policy Evaluation

  • Policy evaluation methods span hardware, simulation, and indirect signals, but the cited approaches do not provide rigorous decision-making for policy comparisons across general metrics.Simulation is an imperfect proxy because of the sim-to-real gap, and statistically rigorous comparison remains distinct from efforts to increase available evaluation data.Source: A. Robot Policy Evaluation · p. 2, §II · p. 2

B. The Parametric Policy Comparison Problem

  • Existing parametric tests face a trade-off between generality and sample efficiency, while asymptotic approaches lose rigorous assurances that matter in low-data robotics settings.Extending both parametric generality and efficiency can require an exact Wald-Bellman equation solution that becomes intractable for complex families.Source: B. The Parametric Policy Comparison… · pp. 2–3

C. Nonparametric Methods and Safe, Anytime-Valid Inference

Prior nonparametric approaches offer generality but face challenges in sequential use and finite-sample validity. The paper frames policy comparison as a statistically assured comparison of means over bounded progress measures, with decisions made from noisy evaluation data.

C. Nonparametric Methods and Safe, Anytime-Valid Inference

  • Nonparametric methods trade exploitable structure for generality, but face limits in representing distributions and generalizing to sequential settings; KDE adapts to low-dimensional structure but is not generally finite-sample valid alone.SAVI supports sequential estimation and decision-making; N-SCORE combines it with KDE ideas tailored to policy comparison.Source: C. Nonparametric Methods and Safe,… · p. 3

III. PRELIMINARIES

  • The comparison problem is to quickly and accurately compare the means of two distributions over a bounded progress metric, without other structural assumptions on that metric.The generalized progress metric is a bounded real-valued random variable in [0, 1].Source: §III · p. 3

IV. PROBLEM FORMULATION

  • Because true policy means are unknown and evaluation outcomes are stochastic, comparisons must bound the chance of incorrect inference while reaching decisions efficiently.N-SCORE uses sequential evaluation to adapt decisions to accumulated evidence and minimize unnecessary trials.Source: §III · p. 3

A. The Formal Hypotheses

The paper formalizes policy comparison as testing whether a new design improves mean performance, then evaluates procedures by correctness and decision speed. It prioritizes control of false positives while trading off expected decision time and false negatives.

A. The Formal Hypotheses

  • The testing problem is formulated in frequentist terms using skeptical and desired hypotheses that partition possible policy-comparison outcomes.In the nonparametric case, the hypothesis space comprises measurable distributions on [0, 1].Source: A. The Formal Hypotheses · p. 4
  • The null represents cases where the innovation is not better, while the alternative represents cases where it is better.A significant decision aims to support confidence that the true state is not in the null set.Source: A. The Formal Hypotheses · p. 4

B. Measuring the Quality of an Evaluation Algorithm

  • Evaluation quality is measured by correct decisions and decision speed, with Type-1 Error denoting a false positive and Type-2 Error a false negative.The expected number of evaluations, E[N], measures sample efficiency.Source: B. Measuring the Quality of an… · p. 4

C. Efficient Tests as a Multi-Objective Optimization

  • An optimal evaluation procedure would minimize Type-1 Error, Type-2 Error, and expected evaluations, but these objectives cannot all be minimized simultaneously.The paper therefore adopts the Neyman-Pearson approach, treating the Type-1 Error bound α ≤ α* as a hard constraint.Source: B. Measuring the Quality of an… · p. 4, C. Efficient Tests as a… · p. 4
  • The optimization searches decision rules Γ subject to α(γ) ≤ α*, with λ trading off expected time to decision against false-negative rate.The paper states that efficiently solving this objective yields a statistically rigorous procedure that detects performance changes quickly.Source: C. Efficient Tests as a… · p. 4

V. METHODOLOGY

  • The method distinguishes the hypotheses by constructing a scalar dynamical system that is stable under H0 and unstable under H1 based on observed evaluation evidence.Stability is assessed within the SAVI framework.Source: §V · p. 4

A. Constructing an ‘Evidence Integrator’

N-SCORE sequentially aggregates paired progress scores into an evidence process and stops when its running maximum crosses a threshold set by the desired Type-1 Error rate. Its theoretical properties support error control and an online choice of evidence-integration rate.

A. Constructing an ‘Evidence Integrator’

  • N-SCORE updates a multiplicative evidence process from the two policies’ observed progress scores, with positive evidence for H1 when r1,n exceeds r0,n.The update rate ξ can be selected online from past evidence, independently of the current observed scores.Source: A. Constructing an ‘Evidence Integrator’ · pp. 4–5
  • The procedure stops and rejects the null when the running maximum evidence reaches 1/α*, the threshold corresponding to the desired Type-1 Error rate.If the threshold is not reached by the evaluation limit, the protocol fails to reject the null.Source: A. Constructing an ‘Evidence Integrator’ · p. 5
  • Algorithm 1’s protocol is described as balancing time-to-decision and correctness while controlling Type-1 Error at a tunable, pre-specified level.Its stated purpose is to minimize evaluation burden while supporting statistically assured policy comparisons.Source: A. Constructing an ‘Evidence Integrator’ · p. 5

B. Theoretical Properties of Algorithm 1

  • Under the null, the evidence process contracts in expectation for any ξn in [0, 1], a stability property used to establish Type-1 Error control.The paper connects null stability to preventing null evidence increments from driving large process growth.Source: B. Theoretical Properties of Algorithm 1 · p. 5
  • Theorem 1 addresses Type-1 Error control for Algorithm 1 using the process defined in Equation (3); enforcing it ensures confidence at level 1 − α∗ that reported significant innovations and improvements to the state-of-the-art are strictly separated from inherent robotic-evaluation noise.Source: B. Theoretical Properties of Algorithm 1 · p. 5
  • N-SCORE optimizes the evidence-integration rate ξn online using nonparametric representations motivated by kernel density estimation and discrete partial-credit structure.The paper reports that this optimization yields a state-of-the-art near-optimal solution for general progress metrics.Source: B. Theoretical Properties of Algorithm 1 · pp. 5–6

VI. EXPERIMENTAL RESULTS

The experiments assess sequential evaluation, the value of informative metrics beyond binary success, and N-SCORE’s technical advantages over statistically valid sequential comparison methods.

VI. EXPERIMENTAL RESULTS

  • RQ1 asks whether sequential evaluation improves sample efficiency.Source: §VI · p. 6
  • RQ2 asks whether metrics more informative than binary success improve sample efficiency.Source: §VI · p. 6
  • RQ3 asks what benefits N-SCORE offers over statistically valid sequential policy-comparison approaches.Source: §VI · p. 6

A. Baseline Methods

Across binary and richer metrics, sequential evaluation reduces the number of evaluations needed for policy comparisons. N-SCORE also supports nonparametric comparisons, where the reported results show improved sample efficiency over WSR.

A. Baseline Methods

  • The baselines span STEP for binary success, θ-SAVI for parametric comparisons, and WSR for nonparametric metrics.STEP and θ-SAVI do not extend to nonparametric performance metrics.Source: A. Baseline Methods · p. 6

B. RQ1 and RQ2: Sequential Evaluation with Informative Metrics

1) Results on Artificial Bernoulli Sequences:

1) Simulated Nonparametric Data:

1) Results on Artificial Bernoulli Sequences:

3) Results on Simulated Nonparametric Data:

  • For simulated nonparametric distributions with mean-performance gaps of at least 0.01, average time-to-decision was low relative to the 1,000-sample allocation.Only N-SCORE and WSR apply to this analytically opaque distribution family.Source: 3) Results on Simulated Nonparametric… · p. 7
  • Approximately 70% fewer simulation evaluations than batch methods and over 50% fewer than sequential binary methods were needed with partial-credit metrics.On hardware, the reported reductions were approximately 45% versus batch and 24%–30% versus STEP [68].Source: 4) Results on LBM 1.0 Partial Credit… · p. 7, Table II · p. 7

2) Simulated Bernoulli Data:

Across simulated and real-world evaluations, N-SCORE matches θ-SAVI in parametric settings and improves sample efficiency over WSR in nonparametric settings. The results support sequential evaluation with informative metrics as a way to reduce evaluation burden.

2) Simulated Bernoulli Data:

  • On simulated Bernoulli data, θ-SAVI and N-SCORE performed nearly identically, while STEP was optimal and WSR took substantially longer to decide.Source: 2) Simulated Bernoulli Data · p. 8

3) LBM 1.0 Success and Partial Credit:

4) Results on RoboArena:

  • At least 450 fewer trials (1419 vs. 1881) were required overall by N-SCORE than WSR on RoboArena, while distinguishing all four policies.N-SCORE required over 200 fewer trials for π0; WSR could not separate π0 and PG-Diff after all 641 available trials.Source: 4) Results on RoboArena · p. 8

5) Real-valued Continuous Metrics:

  • On InvertedPendulum-v4, N-SCORE saved over 400 trials when comparing PPO with DDPG, while neither method separated SAC and TD3 on the same benchmark.SAC and TD3 returned the maximum possible reward in each rollout.Source: 5) Real-valued Continuous Metrics · p. 8
  • N-SCORE matches θ-SAVI’s sample efficiency in parametric settings and improves on WSR in nonparametric settings.The authors attribute the RoboArena efficiency to optimizing ξn with available data and note that WSR is not optimized for policy comparison.Source: 5) Real-valued Continuous Metrics · p. 8

VII. LIMITATIONS AND FUTURE WORK

  • SAVI-based procedures can leave some Type-1 error risk budget unused, explaining a gap to STEP on binary metrics; richer KDE tools may improve efficiency and generalization to unbounded measures.Source: §VII · p. 8

VIII. CONCLUSION

  • Validation across simulated and real-world data supports N-SCORE as a synthesis of two state-of-the-art sequential procedures that can reduce evaluation burden.The authors frame reliable performance improvements and lower evaluation burden as potential benefits for progress in the field.Source: §VIII · pp. 8–9

APPENDIX A ANALYTICAL RESULTS

The appendix establishes the validity of the evidence process under the null and uses Ville’s inequality to bound false rejection, including when the decision time is selected adaptively.

A. Proof of Lemma 1

  • Under the null, bounded progress metrics make the evidence increment bounded in [1−ξ, 1+ξ] and give it well-defined moments; the supermartingale result extends to binary, discrete, and continuous bounded metrics.Source: A. Proof of Lemma 1 · p. 13

B. Proof of Theorem 1

  • The theorem proof uses the nonnegative supermartingale property on the null set as the basis for its error-control argument.Source: B. Proof of Theorem 1 · p. 13

1) Verifying Ancillary Conditions:

2) Ville’s Inequality:

  • Ville’s inequality links a nonnegative supermartingale to an upper bound on its realized quantiles, and this remains valid for an optionally selected decision time.Source: 2) Ville’s Inequality · p. 13

3) Time-Varying ξn:

The proof permits ξn to depend on past data while preserving Type-1 error control, and describes how a discretized representation can guide its selection without changing the validity guarantee.

3) Time-Varying ξn:

  • With ξn chosen from past data, independence from new evaluation outcomes preserves the null supermartingale property.Source: 3) Time-Varying ξn · p. 13

4) Completing the Proof:

  • The evidence process and stopping rule bound the probability of falsely rejecting any true null by α*.Ville’s inequality bounds the probability that the process ever exceeds 1/α*, which yields the stated uniform false-rejection bound.Source: 4) Completing the Proof · p. 14

C. Proof of Remark 1

  • N-SCORE can discretize observed scores to choose ξn, but uses exact scores to construct increments for Type-1 error control; discretization can affect efficiency, not validity.The KDE-inspired binning approximates outcome distributions; increasing the bin count improves representation accuracy while increasing computational burden.Source: C. Proof of Remark 1 · p. 14

1) Intuition for Optimizing

The choice of ξn affects efficiency under the alternative, not Type-1 error under the null. The optimization balances positive evidence against losses from outcomes that cancel one another.

1) Intuition for Optimizing

2) Signal Effect:

  • Positive score differences provide a signal for the alternative and push ξn upward; N-SCORE estimates the relevant asymmetry from observed data.Source: 2) Signal Effect · p. 15

3) Hysteresis Effect:

  • Opposite outcomes can cancel the performance gap while reducing the evidence process; this hysteresis effect grows quadratically with ξ and regularizes its choice downward.Source: 3) Hysteresis Effect · p. 15

4) The Optimization Problem:

The optimization chooses a betting coefficient to maximize expected evidence growth using current empirical distributions. For comparison, WSR builds confidence sequences over policy-score differences and stops when they exclude zero.

4) The Optimization Problem:

  • The optimization uses empirical bin frequencies to estimate expected multiplier value, then selects ξ_n to maximize expected log-growth while balancing signal and hysteresis.A root-finding procedure on [0, 1] implements the choice using first-order optimality conditions.Source: 4) The Optimization Problem · p. 16

A. Waudby-Smith Ramdas Confidence Sequences

  • WSR compares policies by tracking a confidence sequence for their score difference and stops when the interval excludes zero; its sign indicates which policy has higher evidence.Source: A. Waudby-Smith Ramdas Confidence… · p. 16
  • WSR required more trials than N-SCORE uniformly across Mujoco pairwise comparisons, especially for PPO versus DDPG on InvertedPendulum-v4.Blank decisions count as 1,000 trials; each pairwise-comparison column represents 14,000 batch trials.Source: A. Waudby-Smith Ramdas Confidence… · p. 17
  • On InvertedPendulum-v4, WSR needed nearly three times the evaluation effort of N-SCORE to reach a decision.Source: Figure A.1 · p. 17

APPENDIX C ADDITIONAL EXPERIMENTAL RESULTS

Across the appendix experiments, N-SCORE reduces evaluation effort on continuous RL rewards and continuous task-progress scores. Continuous progress also separates all RoboArena policies in fewer trials than binary-score procedures that fail to distinguish them all.

A. Real Valued Continuous Metrics (RL Baselines)

  • About 5,600 evaluations let N-SCORE make all Mujoco comparisons, versus about 7,100 for WSR and a 28,000-rollout batch burden.N-SCORE used about 20% fewer trials than WSR across the RL evaluation process.Source: A. Real Valued Continuous Metrics (RL… · p. 17

B. Binary vs. Continuous Valued Metrics

  • N-SCORE on continuous progress used 1,419 trials, saving 560 versus STEP on Bernoulli scores while separating every policy; STEP failed to separate two.The partial-credit and binary measures came from the same policy rollouts.Source: B. Binary vs. Continuous Valued Metrics · p. 18, Figure A.3 · p. 19
  • Continuous task-progress scores provide more informative comparisons than Bernoulli scores, which can shrink the observed performance gap and require more trials.For the same performance gap, the paper attributes faster distinction to a higher signal-to-noise ratio, since continuous-score variance is no greater than the corresponding Bernoulli-score variance.Source: B. Binary vs. Continuous Valued Metrics · p. 19
  • For comparisons with performance gaps around 10% or lower, N-SCORE on continuous task progress distinguished policies fastest, while WSR on binary metrics was slowest.In two comparisons, neither N-SCORE nor WSR on binary metrics distinguished policies within 641 trials, though N-SCORE p-values were closer to the α threshold.Source: B. Binary vs. Continuous Valued Metrics · p. 19
  • Bonferroni correction across pairwise tests reduces statistical power; the authors identify efficient risk-budget allocation as future work.Source: B. Binary vs. Continuous Valued Metrics · p. 20

C. Data Independence in Robot Policy Evaluation

Environment-sampling protocols differ in their independence assumptions, which affect how policy comparisons should be interpreted.

C. Data Independence in Robot Policy Evaluation

  • Evaluation protocols range from repeating one fixed environment, to sharing each sampled environment across policies, to independently sampling environments per policy.The shared-environment scheme is used by RoboArena; independent per-policy sampling gives i.i.d. data but may produce unequal initial-condition frequencies across policies.Source: C. Data Independence in Robot Policy… · p. 20
  • Repeatedly evaluating policies on one nominally fixed environment is theoretically valid, but real resets are imprecise because of small reset and camera-pose errors [41].This setup is useful for comparing which policy succeeds more consistently in that particular setting.Source: C. Data Independence in Robot Policy… · p. 20
  • Stratification produces exchangeable but non-i.i.d. samples, fixing context counts; its effects are practically negligible when L is small relative to N.For example, stratification can require exactly 50 of 100 evaluations in each of two task contexts, unlike i.i.d. sampling.Source: C. Data Independence in Robot Policy… · p. 20
  • Shared environment draws can induce cross-policy correlations that inflate performance-difference variance and risk Type-I Error violations, especially when L≈N.The authors do not correct for this phenomenon in their evaluation algorithms.Source: C. Data Independence in Robot Policy… · p. 20
  • A post-hoc correction for shared-environment evaluations randomly retains one policy’s score per environment, yielding an average effective sample size of N/K per policy.The reduced data can then be evaluated with N-SCORE or WSR; the correction is generally less efficient than resampling an environment for each policy.Source: C. Data Independence in Robot Policy… · pp. 20–21

Related papers

Loading 2603.13616v1…