PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Learning as an Empirical Science: Best Practices for Policy Evaluation

Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kuppuswamy, Paarth Shah, Phoebe Horgan, Gordon Richardson, Siyuan Feng, Benjamin Burchfiel

arXiv:2409.09491v2cs.RO

TL;DR

Robot-policy evaluations often rely on success rate while omitting conditions, failure descriptions, and statistical analysis. The paper proposes evaluation practices that include setup guidance, complementary metrics, and statistical analysis. Its examples show that success percentage alone may not distinguish policy effectiveness; the authors believe broad adoption of their practices could reduce noise and improve progress.

  • Problem

    Robot-policy evaluations often report success rate without enough information about conditions, run counts, success criteria, failures, or statistical analysis.

    Source: §1 · p. 1

  • Method

    The paper proposes policy-evaluation practices covering experimental setup, complementary metrics, and statistical analysis.

    Source: §1 · p. 1, §4.2 · p. 8, §5 · p. 9

  • Results

    Success percentage alone does not support in-depth policy comparison; in one Bayesian example, policy A succeeded 13/20 (0.65) times and policy B 14/20 (0.7), but neither could be said to outperform the other.

    Source: §6.3 · p. 11, §6.4.2 · p. 13

  • Takeaways & Limitations

    The authors believe widespread adoption of their practices will reduce noise and improve the pace of progress in robot learning.

    Source: §5 · p. 9

    Learned-model and visual-language-model methods for automatically assessing semantic information and failures are not yet reliable for physical-system rollouts.

    Source: §3.1.1 · p. 5

Abstract

from arXiv · show

The robot learning community has made great strides in recent years, proposing new architectures and showcasing impressive new capabilities; however, the dominant metric used in the literature, especially for physical experiments, is "success rate", i.e. the percentage of runs that were successful. Furthermore, it is common for papers to report this number with little to no information regarding the number of runs, the initial conditions, and the success criteria, little to no narrative description of the behaviors and failures observed, and little to no statistical analysis of the findings. In this paper we argue that to move the field forward, researchers should provide a nuanced evaluation of their methods, especially when evaluating and comparing learned policies on physical robots. To do so, we propose best practices for future evaluations: explicitly reporting the experimental conditions, evaluating several metrics designed to complement success rate, conducting statistical analysis, and adding a qualitative description of failures modes. We illustrate these through an evaluation on physical robots of several learned policies for manipulation tasks.

1 Introduction

Robot learning has advanced, but evaluations often provide too little information beyond success rate to assess policies or compare findings. The paper proposes evaluation best practices and illustrates them with physical-robot manipulation tasks.

1 Introduction

  • Evaluations often report success rate without enough detail on conditions, run counts, criteria, failures, or statistical analysis to assess findings.The authors say this limits researchers’ ability to identify the state of the art and understand the conditions under which policies may fail.Source: §1 · p. 1
  • The paper proposes best practices for policy evaluation, including guidance on experimental setup, metrics, and analysis.The practices focus on physical robots and are also presented as applicable to simulation.Source: §1 · p. 1
  • The examples use physical-robot manipulation tasks and behavior-cloning policies that share demonstrations but differ in architecture, observation space, and hyperparameters.The tasks are bowl pushing, pancake flipping and serving, and shirt folding.Source: §1 · p. 2

1.1 Related Work

  • Related guidelines cover human-robot interaction experiments and statistical analysis for deep reinforcement learning [10] [11].Robotics metrics discussed in prior work include success rate, goal distance, task-specific measures, and sub-goal achievement [23].Source: §1.1 · p. 2

2 Experiment Setup

Physical evaluations cannot exactly reproduce conditions across runs, and policy performance depends on those conditions. The section recommends clear success criteria, controlled initial conditions, and consistent, less biased evaluation processes.

2 Experiment Setup

  • Physical evaluations cannot reproduce identical conditions across runs, so best practices aim to reduce confounds and improve confidence in comparisons.Policy performance is highly dependent on experimental conditions.Source: §2 · p. 3

2.1 Success criteria

  • A clear, detailed success definition helps prevent evaluator ambiguity and bias from distorting results.In one pouring example, three authors disagreed on whether the episode counted as a success.Source: §2.1 · p. 3

2.2 Initial conditions

  • Initial conditions include object types and locations, lighting, and camera positions, and can change unnoticed in ways that affect policy performance.The authors note that empirical work does not commonly control or explicitly describe these conditions.Source: §2.2 · p. 3
  • 72% vs 80% success for policies A and B on shirt, yet success depended strongly on initial condition and unmatched conditions could skew the comparison.After a lab reorganization, policy A succeeded 0/8 times (0%), compared with 13/20 (65%) previously on the same initial conditions, indicating a suspected distribution shift.Source: §2.2 · p. 4

2.3 Experimental Process Best Practices

  • Recommended practices include predefining criteria, matching initial conditions, interleaving blinded A/B rollouts, and standardizing evaluator roles.Matching conditions can use image overlays or scene markings, with policies evaluated in the same session where possible.Source: §2.3 · p. 4

3 Evaluation metrics

The section pairs binary measures of task correctness with continuous measures of behavior quality, including descriptive rubrics and computable temporal-logic metrics. Examples show how these metrics reveal partial progress, specification violations, and motion-quality differences that overall success alone does not capture.

3.2 Performance metrics

  • The section distinguishes binary semantic correctness from continuous performance quality, with task-agnostic measures such as trajectory smoothness alongside task-specific metrics.Source: §3 · p. 4, §3.2 · p. 6

3.1.1 Rubrics

  • Policy C completed spatula grasping and pancake flipping 100% of the time despite only 17% overall success, showing partial progress hidden by the aggregate outcome.The rubric records task components as well as overall completion.Source: Table 1 · p. 5, §3.1.1 · p. 5

3.1 Semantic information

  • Rubrics provide task-specific, fine-grained Yes/No measures of subgoals and failure modes, while computable semantic metrics can be based on logic.The evaluator fills out the rubric during physical-robot evaluation, adding work but providing useful information.Source: §3.1 · p. 5, §3.1.1 · p. 5

3.1.1 Rubrics

  • Learned-model and visual-language-model automation for semantic assessment is promising, but is not yet reliable for physical-system rollouts.Source: §3.1.1 · p. 5

3.1.2 Signal Temporal Logic

  • STL supports Boolean and quantitative evaluation of temporal behaviors, from simple obstacle-distance requirements to timed, conditional multi-arm behavior.Given a formula and robot/environment state, STL metrics can be computed automatically; physical-robot use requires state information such as proprioception and classifiers.Source: §3.1.2 · pp. 5–6

3.2.1 Signal Temporal Logic robustness

  • Robustness scores show two push-bowl behavior modes: near 80 indicates no contact, while near 0 indicates contact; negative values indicate violation of the z > 0.25 requirement.Policy 5 most often avoids contact, while policy 1 most often violates the z-value requirement.Source: §3.2.1 · p. 6, Figure 3 · p. 7
  • STL robustness indicates not only whether a potentially complex objective is satisfied but also how close a policy is to succeeding or failing.Source: §3.2.1 · p. 6

3.2.2 Smoothness metrics

  • For policy 2, the two successful bowl-contacting extreme rollouts had SPARC values of −2.72 (smoothest) and −11.13 (least smooth), completing the task in 4 and 15 seconds, respectively.Source: §3.2.2 · p. 7

4 Analysis and Reporting

The section recommends reporting experimental parameters, statistical analysis, and failure modes, and illustrates how analysis can reveal policy differences despite similar success rates.

4 Analysis and Reporting

  • Figure 4 plots pre-contact SPARC for bowl-contact rollouts, with one point per rollout and color-coded policies; more negative values mean less smooth motion.Rollouts in which the robot did not touch the bowl are excluded.Source: Figure 4 · p. 7
  • The evaluation report illustrates that policy differences can be identified even when success rates are similar.The section organizes reporting around experimental parameters, statistical analysis, and failures.Source: §4 · pp. 7–8

4.1 Experiment parameters

  • Report success criteria, evaluation counts and timing, and initial conditions, including their relation to training conditions.Initial conditions can be described visually or narratively, and evaluations should clarify whether conditions are in-distribution or intended to test out-of-distribution behavior.Source: §4.1 · p. 8

4.2 Statistical Analysis

  • Point estimates can mislead; statistical analysis can estimate uncertainty using frequentist intervals or bounds, or Bayesian parameter distributions.For the Pancake example, the estimated probability that policy B outperforms A is .11.Source: §4.2 · p. 8

4.3 Failure modes

  • Report failure categories, descriptions, frequency, and visual examples; in the Bowl example, policy 5 had 9 failures of missing the bowl, while policies 1, 2, and 3 had none.The authors frame failure descriptions as useful for setting expectations, identifying research needs, and providing a baseline for progress.Source: §4.3 · p. 8

5 Discussion

The discussion advocates task- and domain-suited metrics beyond success rate, applies the proposed practices to simulation, and encourages releasing evaluation rollouts. It notes that physical evaluation remains difficult and that the practices are not a panacea.

5 Discussion

  • Choose metrics suited to the approach, task, and domain that provide more nuanced information than success rate alone.Some metrics are task-specific, including subgoal achievement and STL specifications.Source: §5 · p. 9
  • The best practices also apply in simulation, where identical initial conditions and environmental states and automated metrics may be easier to ensure, but the sim-to-real gap remains relevant.The passage identifies the sim-to-real gap as an active research area.Source: §5 · p. 9
  • Release evaluation rollout data openly to support nuanced post hoc analysis, recognizing that useful metrics may depend on the application.Automation applications may favor consistency, while human-interaction settings may favor motion smoothness.Source: §5 · p. 9
  • Physical evaluation must contend with confounding variables, resource trade-offs, equipment malfunction, and human subjectivity; the proposed practices are not a panacea.The authors believe widespread adoption will reduce noise and improve the pace of progress.Source: §5 · p. 9

6 Appendix

The appendix explains STL-based behavior metrics, visualization of initial conditions, and Bayesian analysis, then demonstrates a physical-robot evaluation with explicit conditions and complementary outcome measures. Although the two policies had similar success rates, their failure patterns and trajectory metrics differed.

6.1 STL syntax and semantics

  • STL is defined over continuous-valued signals, and recursive formulas combine predicates with Boolean and temporal operators over intervals.For finite trajectories, the interval endpoint is finite; the appendix illustrates the initial location of the manipulated object as one aspect of an initial condition.Source: §6.1 · p. 10, §6.2 · p. 10

6.2 Visualizing initial conditions - additional example

  • Visualizing bowl locations showed that the training distribution was nonuniform and evaluation conditions differed across three policies despite similar success rates.The figure visualizes object locations; semantic segmentation, blob filtering, and image-centroid calculation were used to extract them.Source: §6.2 · pp. 10–11, Figure 5 · p. 10

6.3 Bayesian analysis

  • Bayesian success-rate distributions show that a fixed success percentage alone does not resolve policy effectiveness; with more evaluations, the example favored policy A over policy B.At 150/30 successes/failures for A and 110/60 for B, zero was no longer in the support of the difference distribution.Source: §6.3 · p. 11, Figure 6 · p. 11, Figure 7 · p. 12

6.4.1 Experiment parameters

  • The example report evaluated two behavior-cloned policies in blinded A/B trials across two days, with 20 evaluations per policy over 10 matched initial conditions.Each policy ran 12 times on the first day and 8 times four days later; each initial condition was evaluated twice per policy.Source: §6.4 · p. 12, §6.4.1 · p. 12, Figure 8 · p. 13

6.4.2 Results

  • Policy A succeeded 13/20 times (0.65) and policy B 14/20 (0.7); Bayesian analysis could not establish that either outperformed the other.Policy A failed to grasp or prematurely dropped the bar in 6/7 failures; policy B picked it up in 20/20 runs but failed to move or place it correctly in 6/6 failures.Source: §6.4.2 · pp. 13–15, Figure 11 · p. 15
  • Policy B produced smoother trajectories by both metrics, while its STL robustness values indicated a tendency to grasp lower than policy A.Smoothness comparisons include successful runs only; the STL formula checks that the end effector is below z = 0.25 when the gripper closes.Source: §6.4.2 · pp. 13–14, Figure 10 · p. 14
  • The policies failed on different initial conditions: both failed on IC 6, while policy A also failed on ICs 4, 7, 8, and 9 and policy B on ICs 2 and 9.Policy A failed on both runs of IC 4 and one run each of ICs 7, 8, and 9; policy B failed on both runs of ICs 2 and 9.Source: §6.4.2 · p. 14

Related papers

Loading 2409.09491v2…