PaperlayerView exact revision on arXiv

Source-linked AI summary

Position: Benchmarking is Limited in Reinforcement Learning Research

Scott M. Jordan, Adam White, Bruno Castro da Silva, Martha White, Philip S. Thomas

arXiv:2406.16241v1cs.LGstat.ME

TL;DR

RL research relies heavily on benchmark performance, but rigorous benchmarking can be costly and may show only that an algorithm works, not why. The paper analyzes these costs and advocates scientific testing as a complementary experimentation paradigm. It concludes that benchmarking can require hundreds to thousands of seeds and should be supplemented by controlled experiments that reveal algorithm behavior.

  • Problem

    Rigorous benchmarking can produce misleading or unsupported claims because it is costly, sensitive to methodological choices, and limited in explaining why algorithms succeed.

    Source: §1 · p. 1

  • Method

    The paper analyzes the trials and confidence intervals required for rigorous benchmarking and uses exploration algorithms to illustrate scientific testing’s additional insights.

    Source: §1 · p. 2, §2 · p. 3, §4 · p. 6

  • Results

    Benchmarking can require hundreds to thousands of seeds for each algorithm-environment pair to identify the top-performing algorithm reliably.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Benchmarking should remain a sanity or scaling check after scientific testing, rather than serving as the sole evidence for algorithmic claims.

    Source: §1 · p. 2, §5 · p. 8

    Bootstrap confidence intervals can be too narrow and fail to provide the desired coverage, especially with few samples.

    Source: §2 · p. 3, §3.1 · p. 4

Abstract

from arXiv · show

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking.

1. Introduction

RL research commonly treats benchmark performance as its primary experimental evidence, but rigorous benchmarking can be computationally prohibitive and cannot explain why algorithms succeed. The paper therefore advocates complementing benchmarking with scientific testing.

  • 91% of empirical NeurIPS 2022 RL papers used performance evaluation, reinforcing a standard protocol of proposing an algorithm and demonstrating benchmark superiority.Source: §1 · p. 1
  • Existing rigorous practices address reproducibility, statistical analysis, and sensitivity to hyperparameters, normalization, and task weighting, but their complexity and cost limit adoption.Source: §1 · p. 1
  • Rigorous benchmarking may require too many trials to reliably detect statistically significant performance differences, making strong claims difficult to support through benchmarking alone.Source: §1 · p. 1
  • Scientific testing uses carefully controlled experiments to understand how algorithms work, while avoiding benchmarking’s requirement to run many trials across many environments.Source: §1 · p. 1
  • The paper supports combining benchmarking with scientific testing because the two styles together provide a more complete understanding of proposed algorithms.Source: §1 · p. 2

2. Background

The paper models evaluation as estimating aggregate algorithm performance across environments while accounting for normalization, weighting, and uncertainty. It uses equilibrium-based weighting and bootstrap intervals, while noting that bootstrap coverage can be unreliable.

  • Evaluation estimates each algorithm’s aggregate performance across environments from samples of a specified performance metric, here the final policy’s return after a fixed episode budget.Source: §2 · p. 2
  • The evaluation metric is J(π_final), where π_final is the policy returned after a fixed number of episodes, and the conclusions likely extend to metrics with similar cross-algorithm performance.Source: §2 · p. 2
  • Performance ratios normalize scores relative to baselines, while a weighted combination of algorithms avoids favoring one algorithm through a single baseline choice.Source: §2 · p. 2
  • The aggregate measure uses a statistical summary and environment weights; the paper focuses on the mean while noting that lower-tail-sensitive metrics better reflect reliability across problems.Source: §2 · pp. 2–3
  • Because equilibrium-derived weights are unknown, typical concentration inequalities cannot construct the desired confidence intervals, so the study uses bootstrap intervals as an approximate lower bound on required samples.Source: §2 · p. 3

3. Performance Evaluation

The evaluation studies how seeds, weighting schemes, and benchmark composition affect uncertainty in aggregate RL performance. It finds that reliable rankings often require computationally prohibitive benchmarking, motivating complementary scientific testing.

3. Performance Evaluation

  • Unless thousands of cores and very short algorithm runs are available, rigorous benchmarking requires too many trials to be practical.The study evaluates eight algorithms across fourteen environments using resampled empirical distributions and bootstrap confidence intervals.Source: §3 · p. 3

3.1. Understanding the Failures of Confidence Intervals

  • At 100 seeds, adversarial weighting produces average confidence interval width 0.183 versus 0.150 for uniform weighting, making aggregate performance harder to estimate.Both weighting methods retain large uncertainty at small sample sizes, potentially requiring hundreds to thousands of seeds for significant differences.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
  • Scaling common evaluations from 1–5 seeds to reliable sample sizes requires at least 20–50 times more compute, often becoming impractical for environments taking hours per run.The authors note that blindly trusting bootstrap intervals with few samples is unwise.Source: §3.1 · p. 4
  • Bootstrap confidence intervals reach the target failure rate only above 10, 50, or 100 samples per pair, depending on the weighting configuration.The paper concludes that fewer than 100 seeds per algorithm-environment pair are insufficient for trusted general benchmarks.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2. Impact of Number of Algorithms and Environments

  • When algorithms perform similarly, many seeds are needed; reducing algorithms can help when the leader is separated, while adding environments reduces aggregate uncertainty.Because no algorithm dominates across all environments, identifying the top algorithm can still require hundreds or thousands of seeds, and the necessary count is hard to predict.Source: §3.2 · p. 5, Figure 4 · p. 5
  • The results indicate that rigorous RL benchmarking is expensive and should be supplemented with targeted experiments that reveal how algorithms work.The paper recommends softening ranking claims and combining benchmarking with scientific testing for a fuller picture of algorithmic properties.Source: §3.2 · pp. 5–6

4. Scientific Testing

Scientific testing asks how algorithms work, complementing benchmarking’s focus on ranking performance. Exploration experiments illustrate that controlled, targeted tests can reveal algorithm behavior with fewer computational resources.

4. Scientific Testing

  • Scientific testing investigates how an algorithm works, whereas benchmarking seeks to order algorithms or identify the best performer.It can examine performance sensitivity or properties orthogonal to performance, such as learned value-function features.Source: §4 · p. 6
  • In a NeurIPS 2022 survey, 131 of 144 non-theory papers used benchmarking, while 51 included scientific experiments.The paper argues that scientific testing should become a primary experimentation paradigm rather than a minority addition.Source: §4 · p. 6

4.1. Exploration Algorithms

  • Count-based intrinsic motivation adds novelty-based rewards, while restart-based exploration changes start states toward less frequently visited regions.The restart distribution samples encountered states inversely proportional to visitation frequency and ignores unseen states.Source: §4.1 · pp. 6–7

4.2. Scientific Testing Experiments

  • The experiments use discrete four-rooms MDPs to isolate fundamental algorithm properties and reduce confounding factors.The study uses Sarsa(λ) with intrinsic motivation, restart-based exploration, or both, and examines policy improvement across the state space.Source: §4.1 · p. 7, §4.2 · p. 7

4.3. Experiments Investigating Policy Optimization Over All States

  • The policy analysis measures distance between vπ and the ϵ-greedy optimal value function, while Figure 5 plots start-state return and this distance over episodes.Each plotted value averages 100 trials, with shaded standard-deviation regions and colors representing algorithms.Source: §4.3 · p. 7, Figure 5 · p. 8
  • Restart-based methods optimize policies across larger state-space regions and behave similarly to uniformly sampled starts, but none approaches the optimal value function.The authors hypothesize that the gap may partly reflect the absence of a decaying step size, leaving that test for future work.Source: §4.3 · p. 7, Figure 5 · p. 8

4.4. Scientific Testings and the Computational Demands of Experiments

  • Scientific testing can provide more information than benchmarking with fewer computational resources, although larger environments may make many algorithm trials infeasible.When only one run is feasible, tests can examine properties during learning by changing exploration briefly, measuring an outcome, and resetting the algorithm.Source: §4.3 · p. 7, §4.4 · pp. 7–8

5. Discussion

The discussion argues that performance-only evaluation can conflate algorithmic ideas with function-approximation effects. Scientific testing should build understanding, while benchmarks remain useful as weaker sanity and scaling checks.

5. Discussion

  • Experiments should disentangle algorithmic ideas from function approximation rather than attributing performance differences to the algorithm alone.Questions about state-density estimates, intrinsic rewards, and representation failures can expose their interaction.Source: §5 · p. 8
  • Scientific testing can reveal algorithm properties that support informed predictions on novel problems and guide better algorithm design.Its value is understanding behavior, not merely reporting which method performs best.Source: §5 · p. 8
  • Benchmarks should supplement scientific testing as sanity or scaling checks, with weaker claims about whether controlled-experiment ideas transfer elsewhere.Reporting both successes and failures helps define an algorithm’s applicability scope.Source: §5 · p. 8
  • Better experiments are intended to reduce reliance on waiting for an algorithm to outperform others and increase what researchers learn from experiments.The discussion connects this broader evidence base to faster progress in knowledge and algorithm development.Source: §5 · p. 8

6. Conclusion

The conclusion presents two claims: rigorous benchmarking may require hundreds to thousands of seeds, while scientific testing can provide an alternative source of insight into algorithm behavior.

6. Conclusion

  • Benchmarking can require hundreds to thousands of seeds per algorithm-environment pair to identify the top-performing algorithm reliably.The authors characterize this burden as potentially computationally infeasible and without shortcuts.Source: §6 · p. 8
  • Scientific testing offers an alternative experimental paradigm that provides insight into algorithm behavior and can improve experimental practice.The authors hope it motivates more informative experiments beyond standard benchmarking.Source: §6 · p. 8

Impact Statement

The paper surveys RL experimentation and specifies the algorithm and environment sets used in its empirical studies. It frames experimentation quality as a field-wide concern while noting no specific societal consequences requiring emphasis.

Impact Statement

  • The paper states that its goal is to enhance the quality of experimentation in reinforcement learning research.Source: Impact Statement · p. 9

A. Survey of RL Papers

  • The survey covers 144 non-theory RL papers from NeurIPS 2022 after removing papers whose main objective was theoretical contributions.Source: A. Survey of RL Papers · p. 12
  • The survey distinguishes benchmarking, ablations, qualitative illustrations, and scientific testing as four experiment types.Source: A. Survey of RL Papers · p. 12

B. Algorithm and Environment Sets

C. Benchmarking Configurations

This section varies normalization and weighting choices in the benchmarking procedure, comparing performance ratios and percentiles across alternative aggregate-weighting configurations.

C. Benchmarking Configurations

D. Alternative Confidence Intervals Techniques

The alternative-interval analysis replaces bootstrap intervals with t-distribution-based procedures and examines multiple-comparison correction. These choices produce wider intervals, while the correction scales failure probabilities across algorithms and environments.

D. Alternative Confidence Intervals Techniques

E. Correcting for Multiple Comparison in Aggregate Performance

F. Scientific Experiment Details

The scientific-testing experiments fix step-size and policy-randomness parameters while adding further experiments beyond those described in the main paper.

F. Scientific Experiment Details

G. Scientific Testing Experiments

G.1. Intrinsic Motivation Experiments

Increasing the intrinsic-reward scale β raises visited state-action entropy, but its effects weaken over time and can either suppress extrinsic rewards or trap exploration. These experiments show why understanding β’s mechanism matters beyond benchmarking a single tuned value.

Experimental setup

  • The experiment tests β ∈{0, 0.3, 0.5, 1.0, 1.5, 5.0} using entropy from exponentially averaged per-episode state-action counts.Both intrinsic motivation and restart distribution with intrinsic motivation are evaluated in the four rooms domain.Source: §G.1 · p. 17

Results

  • Higher β increases state-action entropy initially, but the effect weakens over time as intrinsic rewards decay toward a deterministic optimal policy.At β = 5.0, agents practically ignore extrinsic rewards until intrinsic rewards decay; very low β can trap agents near the nearby goal.Source: §G.1 · pp. 17–18

Experimental setup

  • Figure 13 reports return, visited state-action entropy, and the correlation between β and entropy, averaged over 100 trials with uncertainty bands.The correlation plot compares Intrinsic Motivation and restart distribution with intrinsic motivation; shaded regions denote pointwise 95% confidence intervals.Source: Figure 13 · p. 18

G.2. Distance From Start State Experiments

The second experiment tests whether restart-based exploration reaches states farther from the start than intrinsic motivation. Restart-based exploration spends more time in the far room across β values and random transition probabilities.

Experimental setup

  • The experiment varies random transition probability to control chance access to the far room and evaluates β = 1.5 and β = 5.0.States in the far room are grouped, and time spent there is measured with an exponential moving average of state visits.Source: §G.2 · p. 18

Results

  • Restart-based exploration spends more time in the fourth room regardless of β or random transition probability.This suggests adapting the start-state distribution is effective for exploring environments where high-reward states are far from the start.Source: §G.2 · p. 18

Experimental setup

  • Figure 14 shows return and the proportion of time spent in the Far state group for Intrinsic Motivation and restart distribution with intrinsic motivation.Each line averages 100 trials, with colors denoting algorithms and line styles denoting random transition probabilities.Source: Figure 14 · p. 19

Related papers

Loading 2406.16241v1…