Deep Reinforcement Learning at the Edge of the Statistical Precipice

Rishabh Agarwal; Max Schwarzer; Pablo Samuel Castro; Aaron Courville; Marc G. Bellemare

Source-linked AI summary

Deep Reinforcement Learning at the Edge of the Statistical Precipice

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare

arXiv:2108.13264v4Aug 30, 2021cs.LGcs.AIstat.MEstat.ML

TL;DR

Deep RL benchmark comparisons often rely on few-run point estimates that ignore substantial statistical uncertainty. The paper develops uncertainty-aware evaluation tools, applies them across major benchmarks, and finds that statistical issues can materially change reported comparisons. It recommends more rigorous reporting and releases rliable to support that methodology.

Problem
Deep RL evaluations commonly compare aggregate point estimates from finite, often small numbers of training runs, leaving uncertainty in benchmark comparisons insufficiently represented.
Method
The paper uses interval estimates, performance profiles, and robust aggregate metrics to evaluate benchmark results with a handful of runs.
Results
Statistical issues substantially influence reported results across Atari 100k and other widely used RL benchmarks, especially when point estimates or inconsistent protocols are used.
Takeaways & Limitations
Reliable deep RL evaluation should report uncertainty, account for performance variability, and provide results for all runs using tools such as rliable.
Takeaways & Limitations
The paper acknowledges that statistical sophistication can introduce new abuses and that accounting for uncertainty is not a panacea for reliable evaluation.

Abstract

from arXiv · show

Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field.

1 Introduction

Deep RL evaluation commonly relies on a small number of runs despite substantial performance variability, making aggregate point estimates statistically uncertain. The paper proposes uncertainty-aware evaluation because unreliable comparisons can distort conclusions about algorithm quality and field progress.

Motivation: Small run counts combined with high performance variability lead to substantial statistical uncertainty in reported point estimates.
Motivation: 3-10 runs are prevalent in deep RL because evaluating more runs is often computationally prohibitive.Five runs across more than 50 Atari 2600 games require over 1000 GPU training days.
Motivation: Ignoring uncertainty can create a false impression of rapid progress and steer researchers toward superficially beneficial methods.Less favorable random conditions can cause better methods to be neglected or rejected early.
Contributions: The paper finds unreliable point-estimate comparisons and non-standard evaluation protocols in recent deep RL work, including its Atari 100k case study.It extends the analysis to the ALE, Procgen, and DeepMind Control Suite, where prior comparisons also show discrepancies.
Contributions: The proposed methodology reports interval estimates, performance profiles, and robust aggregate metrics to evaluate performance with only a handful of runs.The paper specifically highlights interquartile mean scores as an efficient alternative for reducing uncertainty.

2 Formalism

The formalism models normalized scores from multiple independent runs across multiple tasks as random quantities and treats aggregate metrics as finite-sample estimates. It motivates confidence intervals and compares the uncertainty of median and IQM-based evaluation as run counts vary.

Score model: The evaluation setting contains M tasks and N independent runs per task, each producing a scalar normalized score.Normalized scores are obtained by linearly rescaling task scores using two reference points.
Score model: Run scores are modeled as realizations of task-specific random variables because task, learning, initialization, software, and hardware randomness affect outcomes.GPU and machine-learning-framework non-determinism can remain even when random seeds are fixed.
Aggregate metrics: Aggregate performance maps all normalized task-run scores to a scalar, with mean and median treated as prevalent aggregate metrics.
Aggregate metrics: Sample mean and median are point estimates of corresponding aggregate random variables, whereas true metrics represent the unlimited-run limit as N →∞.
Uncertainty: A 95% confidence interval estimates plausible values for a true score under repeated experiments and interval construction.
Uncertainty: Figure 2 shows substantial median uncertainty even with 50 runs, while IQM has much smaller confidence intervals.The figure uses subsampled run sets and compares sampling distributions and confidence intervals across run counts.

3 Case Study: The Atari 100k benchmark

The Atari 100k case study shows that few-run point estimates and inconsistent evaluation protocols can produce unreliable or incomparable algorithm comparisons. Statistical analyses reveal substantial variability and bias in reported sample medians, while maximum-based protocols can reverse conclusions about algorithm quality.

Few-run variability: Evaluating five algorithms with subsampled runs revealed substantial variability in reported sample medians and potential for erroneous conclusions from point estimates alone.The analysis used 100 independent runs per algorithm and subsampled 3–100 runs with replacement.
Few-run variability: DER may be better than OTR despite reported point estimates suggesting the opposite.
Median bias: The sample median is biased, with SPR’s 5-versus-100-run score difference of +0.03 points equaling about 36% of its +0.08-point mean improvement over DrQ(ε).The bias’s magnitude and sign depend strongly on the evaluated algorithm.
Run requirements: 95% confidence intervals for Atari 100k sample medians suggest that roughly 50–100 runs are needed, making conventional few-run evaluation statistically inadequate.The paper characterizes this run count as computationally infeasible for most research projects.
Evaluation protocols: Maximum-based evaluation protocols produce substantially different Atari 100k results from end-performance protocols and are generally incomparable.Applying CURL’s protocol to DER produced scores far above CURL’s reported scores, and protocol changes explained much of SUNRISE’s improvement over DER.

4 Recommendations and Tools for Reliable Evaluation

In the few-run regime, reliable deep RL evaluation requires uncertainty-aware intervals, performance profiles, and robust aggregate metrics rather than point estimates alone. These tools address variability, outliers, and statistical inefficiency while supporting comparisons with limited runs.

4.1 Stratified Bootstrap Confidence Intervals: Stratified bootstrap confidence intervals replace point estimates with uncertainty ranges for aggregate performance in small-sample evaluations.Percentile confidence intervals provide good interval estimates, though three-run bootstrap intervals may underestimate true 95% coverage.
4.2 Performance Profiles: Performance profiles represent the full distribution of normalized scores across tasks and runs, revealing variability that aggregate intervals and large tables can obscure.Score distributions are recommended for the few-run regime because they are robust to outliers and have smaller confidence bands than average-score distributions.
4.2 Performance Profiles: Score distributions are unbiased for the underlying score distribution, while an extreme outlier run changes their value at any threshold by at most 1/(MN).Average-score distributions instead estimate a biased quantity based on per-task averages.
4.3 Robust and Efficient Aggregate Metrics: IQM confidence intervals are typically narrower than median intervals and can detect a given improvement with fewer runs.IQM is therefore more statistically efficient in the few-run regime, although its performance should still be reported with uncertainty intervals.
4.3 Robust and Efficient Aggregate Metrics: IQM trims the bottom and top 25% of combined runs, retaining the middle 50% to balance robustness to outliers with lower bias than the median.The optimality gap offers a robust alternative to the mean when performance beyond a chosen target γ is less important.
4.3 Robust and Efficient Aggregate Metrics: Average probability of improvement measures how likely one algorithm is to outperform another on a randomly selected task, but it ignores the size of improvement.The paper notes that the best aggregate metric remains dependent on the underlying normalized score distribution.

5 Re-evaluating Evaluation on Deep RL Benchmarks

Re-evaluations across ALE, DeepMind Control Suite, and Procgen show that uncertainty and metric choice can substantially alter conclusions about algorithm performance. Interval estimates, performance profiles, and probability-of-improvement analyses expose overlaps, ranking changes, and weakly supported improvement claims.

Arcade Learning Environment: DreamerV2 exhibits substantial uncertainty in aggregate ALE scores, while performance-profile comparisons can reverse across score thresholds.IQN is better than Rainbow for τ ≥2 but worse for τ < 2; DreamerV2 and M-IQN profiles intersect repeatedly for τ < 8.
Arcade Learning Environment: Aggregate-metric choice changes algorithm ordering because median and IQM capture different aspects of performance across tasks and runs.Optimality gap and IQM/median rankings can differ, revealing nuances that a single aggregate metric misses.
DeepMind Control Suite: Most DeepMind Control Suite algorithms do not consistently rank above methods they claimed to improve upon, while 95% confidence intervals overlap substantially.The evaluation covers six tasks at 100k and 500k steps.
Procgen benchmark: Procgen PPO-normalized scores are typically heavy-tailed, making mean performance highly dependent on a small fraction of tasks.The paper recommends normalization using estimated minimum and maximum scores instead.
Procgen benchmark: 40–50% is the probability that UCB-DrAC improves upon PLR, while several reported Procgen improvements are only 50–70% likely.The probability-of-improvement analysis treats claimed improvements as probabilistic rather than binary.

6 Discussion

The paper finds that statistical uncertainty and inconsistent evaluation protocols materially influence reported deep RL results, despite earlier calls for experimental rigor. It recommends fuller reporting and open tools, while acknowledging that reproducibility remains complex and rigorous evaluation may discourage stronger claims.

6 Discussion: Statistical issues substantially influence reported results when evaluations rely on point estimates or inconsistent protocols.The authors find this pattern in both the Atari 100k case study and analyses of other widely used RL benchmarks.
6 Discussion: Fixing random seeds does not solve reproducibility because it cannot establish performance under different seeds and may benefit algorithms unequally.The paper distinguishes confidence in reported results from missing information about experimental procedures.
6 Discussion: The authors released rliable, a Colab notebook, and individual-run data to support fuller reporting of bias and uncertainty.The resources implement the paper’s recommendations and support replication of its analyses.
6 Discussion: A barrier to adopting rigorous evaluation is that greater rigor generally produces more nuanced and tempered claims.The paper points to scientific quality as one incentive for adoption.
6 Discussion: The authors argue that increased computation is unlikely to resolve statistical problems in future RL benchmarks, making experimental discipline important.This conclusion is based on statistical effects observed in Atari games and low-dimensional robotic simulations.

Societal Impacts

The paper presents statistical sophistication as a foundation for more trustworthy few-run evaluation, while acknowledging that it is not a complete solution. It also identifies potential computational and environmental benefits from reducing the need for many runs.

Societal Impacts: Statistical sophistication can introduce new forms of statistical abuse, so monitoring the literature remains necessary.The authors describe reliable evaluation and reproducibility as an ongoing process that the paper only partly addresses.
Societal Impacts: The methodology may reduce computational burden and be greener than evaluating a large number of runs.The paper frames this as a potential positive societal impact rather than a guaranteed outcome.

A.1 Open-source notebook and data

The paper supports reproducible analysis through an open notebook, released individual runs, and publicly available benchmark scores and data resources.

Open-source notebook and data: A Colab notebook implements performance profiles, robust aggregate metrics, and interval estimates based on stratified bootstrap confidence intervals.The notebook also supports replication of the paper’s results.
Open-source notebook and data: The authors released 100 runs per game for six Atari 100k algorithms in a public cloud bucket.These individual runs support the Atari 100k case study.
Open-source notebook and data: Individual runs and final scores for ALE, Procgen, and DeepMind Control were released using benchmark-specific public data sources.The ALE data combine Dopamine baselines with scores obtained from corresponding authors for several methods.
Open-source notebook and data: A dedicated website provides additional access to the paper’s materials.The site is listed as agarwl.github.io/rliable.

A.2 Atari 100k: Additional Details and Results

The Atari 100k analysis examines implementation details, run variability, uncertainty in comparisons, and the statistical efficiency of IQM. Results show that few-run conclusions can be unstable, while IQM offers lower bias and better efficiency than median in the reported analyses.

Run variability: Fixed random seeds did not ensure deterministic outcomes because GPU and framework nondeterminism can produce different runs.The analysis compared two sets of 100 DER runs using the same seeds for Python, NumPy, and JAX.
IQM analysis: Median requires many more runs than IQM to achieve small uncertainty in score comparisons.The comparison is shown using 95% confidence intervals for differences in median and IQM scores.
Experimental setup: 18,200 runs were trained and evaluated across Atari 100k experiments, including 100 runs per game for six algorithms and additional DER runs.The case study used 26 games and Tesla P100 GPUs, with each run lasting approximately 3–5 hours.
Statistical comparisons: 95% confidence intervals for score differences reveal uncertainty that point estimates can obscure, including intervals spanning both improvement and decline.With 5 runs, the median improvement from SPR over DrQ lay within (−0.09, 0.18).
IQM analysis: IQM provides good statistical efficiency among trimmed estimators and considerably smaller bias than median on Atari 100k.Expected IQM differences between 3 and 100 runs were typically an order of magnitude smaller than expected median differences; for SPR, the comparison was 0.006 versus 0.05 points.

A.3 Related work on rigorous evaluation in deep RL

The related-work discussion distinguishes this paper’s focus on reliable benchmark evaluation from work on single-task significance testing, algorithm stability, and complete-algorithm evaluation. It also identifies limitations of an alternative probability-based performance profile.

Scope relative to prior work: This paper focuses on reliable comparisons across task suites, whereas prior work emphasizes single-task significance testing and minimum-run determination.The paper mainly recommends stratified bootstrap confidence intervals for aggregate metrics with few runs.
Bootstrap comparisons: Single-task bootstrap confidence intervals may require 20–30 runs per task for near-95% coverage, compared with 5–10 runs for stratified bootstrap aggregate metrics.The comparison is reported for Atari 100k and aggregate median, mean, and IQM scores.
Performance profiles: Performance profiles can expose return variability across tasks and runs while remaining robust to outlier runs and small performance changes.They provide a visual alternative to overwhelming per-task tables.
Evaluation regimes: Game-theoretic evaluation recommends 1,000–10,000 runs per task for complete algorithms, while this paper targets post-tuning evaluation with only a handful of runs.Run-score distributions across hyperparameter configurations may still reveal tuning sensitivity.
Profile limitations: A probability-of-beating-the-best profile is biased, less robust to outliers, insensitive to difference magnitude, dependent on Gaussian assumptions, and dependent on the comparison set.These limitations apply to the alternative profile based on Welch’s t-test.

A.4 Non-standard Evaluation Protocols Involving Maximum

Maximum-based evaluation protocols can bias scores and make comparisons with end-performance results incomparable. The Atari 100k case study shows that protocol differences can explain substantial portions of reported algorithm improvements.

Protocol bias: Maximum operations across or within runs introduce positive bias relative to standard evaluation without the maximum.This remains a concern even when an adequate number of runs is used.
Protocol bias: Maximum-during-training results are generally incomparable with end-performance results because learning curves are not generally monotonic.Taking maxima also adds another source of positive statistical bias.
Atari 100k case study: CURL’s maximum-over-evaluations protocol produced scores far above those reported for CURL when applied to its DER baseline.Natural variability during evaluation and training contributes to overestimation under this protocol.
Atari 100k case study: Much of SUNRISE’s reported improvement over DER can be explained by taking the maximum across eight hyperparameter configurations.The analysis simulated the procedure on DER using a dummy hyperparameter and three runs per configuration.

A.5 Bootstrap Confidence Intervals

The paper develops bootstrap-based uncertainty estimates and complementary aggregate metrics for few-run, multi-task RL evaluation. It also addresses normalization, visualization, and metrics that emphasize difficulty or superhuman performance.

Bootstrap confidence intervals: Stratified bootstrap confidence intervals are used to quantify uncertainty in aggregate metrics across tasks and runs.The paper evaluates basic, percentile, bias-corrected, and bias-corrected accelerated bootstrap intervals, finding percentile methods more effective in the few-run regime.
Bootstrap confidence intervals: Bootstrap intervals for per-game means require many more runs than intervals for aggregate scores to approach nominal 95% coverage.Aggregate scores use samples across tasks and runs, whereas per-game intervals use fewer observations.
Bootstrap confidence intervals: With only 1–2 runs per task, stratified bootstrapping over tasks estimates sensitivity to task selection and expected performance on a larger task population.This complements stratification over both tasks and runs when more runs are available.
Normalization: The paper recommends Min-Max normalized scores for Procgen instead of PPO normalized scores to enable comparisons beyond PPO-based methods.Normalization choices also include human-normalized Atari scores and Min-Max normalized DeepMind Control scores.
Visualizations and metrics: Performance profiles visualize score distributions across tasks and runs, while alternative x-axis scaling focuses attention on regions containing most runs.The paper also presents aggregate metrics including Difficulty Progress, Superhuman Probability, optimality gap, and probability of improvement.
Alternative aggregate metrics: Difficulty Progress measures mean scores among the bottom 25% of runs, emphasizing performance on difficult tasks.It is proposed alongside optimality gap as a measure focused on lower-performing portions of a benchmark.
Alternative aggregate metrics: Superhuman Probability reports P(X > 1), the fraction of runs exceeding average human performance, instead of counting games above that threshold.This metric is presented as an alternative aggregate measure for ALE.