How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

Cédric Colas; Olivier Sigaud; Pierre-Yves Oudeyer

Source-linked AI summary

How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer

arXiv:1806.08295v2Jun 21, 2018cs.LGstat.ML

TL;DR

Deep RL comparisons need reliable statistical significance checks because few seeds and unstable experimental outcomes can produce misleading conclusions. The paper analyzes t-tests and bootstrap confidence intervals, derives seed-count guidance, and examines assumption violations. It concludes that bootstrap tests need at least N = 20 samples and that larger samples than prescribed may be necessary.

Problem
Few random seeds and experimental variability can make statistically different-looking results arise even when algorithm performances are the same.
Method
The paper analyzes t-tests and bootstrap confidence-interval tests, derives sample-size guidelines for type-I and type-II errors, and evaluates deviations from their assumptions.
Results
The bootstrap test should not be used with fewer than N = 20 samples, and sample sizes should exceed power-analysis estimates when standard-deviation estimates may be inaccurate.
Takeaways & Limitations
Use larger seed samples than prescribed when needed to reduce type-II errors and compensate for inaccurate standard-deviation estimates.
Takeaways & Limitations
Statistical-test error guarantees can be inaccurate when their assumptions do not hold, including with insufficient bootstrap sample sizes.

Abstract

from arXiv · show

Consistently checking the statistical significance of experimental results is one of the mandatory methodological steps to address the so-called "reproducibility crisis" in deep reinforcement learning. In this tutorial paper, we explain how the number of random seeds relates to the probabilities of statistical errors. For both the t-test and the bootstrap confidence interval test, we recall theoretical guidelines to determine the number of random seeds one should use to provide a statistically significant comparison of the performance of two algorithms. Finally, we discuss the influence of deviations from the assumptions usually made by statistical tests. We show that they can lead to inaccurate evaluations of statistical errors and provide guidelines to counter these negative effects. We make our code available to perform the tests.

1 Introduction

Deep RL reproducibility is undermined by implementation differences, unstable random-seed outcomes, and practices that can make equal algorithms appear different. This paper presents statistical testing and sample-size guidance for more reliable comparisons.

Different implementations of the same deep RL algorithm and hyperparameters can produce drastically different results.
The authors provide code for performing the proposed statistical tests.
Using only a few seeds, split averages, or the best-performing runs can falsely suggest that two algorithms differ.The paper motivates larger samples but asks how to choose among 5, 10, or 100 seeds.
The paper studies how to test performance differences and choose sample sizes that control statistical errors.
With five seeds on Half-Cheetah, confidence intervals may suggest Algo1 outperforms Algo2 even though their performances are actually the same.

2 Definition of the statistical problem

The paper formalizes algorithm performance as a random variable and frames comparison as testing whether the mean difference is zero. It then explains significance decisions, confidence intervals, and the type-I and type-II errors that govern sample-size choices.

2.1 First definitions: 2.1 First definitions: Algorithm performance is modeled as a random variable, with repeated seeds producing samples whose empirical mean and standard deviation become more reliable as N grows.
2.1 First definitions: 2.1 First definitions: Comparing two algorithms is equivalent to testing whether the mean of their performance difference, µ_diff, equals zero.
2.1 First definitions: 2.1 First definitions: In the five-seed Half-Cheetah example, learning curves and confidence intervals appear to favor Algo1, but statistical tests are required before concluding a difference.
2.2 Comparing performances with a difference test: 2.2 Comparing performances with a difference test: Difference testing defines H0 as no performance difference and Ha as a difference, with one-tail alternatives possible when the better algorithm is specified in advance.
2.3 Statistical errors: 2.3 Statistical errors: At α = 0.05, the false-positive probability is 1 in 20 under test assumptions, increases linearly across experiments, and may require multiple-comparison correction.
2.2 Comparing performances with a difference test: 2.2 Comparing performances with a difference test: A result is significant when the p-value or confidence-interval criterion rejects H0 at the chosen significance level α.
2.3 Statistical errors: 2.3 Statistical errors: Type-I error is a false positive, whereas type-II error is failing to detect a real difference.

3 Choice of the appropriate statistical test

The paper recommends choosing between Welch’s t-test and bootstrap confidence intervals according to distributional and variance assumptions, while recognizing that violations can distort error rates. It also explains how significance thresholds, sample size, and bootstrap procedures affect statistical conclusions.

3.1 T-test and Welch’s t-test: Welch’s t-test is preferred over the equal-variance t-test when comparing algorithms with unequal variances, while both tests coincide when standard deviations are equal.The tests require continuous, ordinal, representative, independent, and approximately bell-shaped measurements.
3.1 T-test and Welch’s t-test: A t-test rejects H0 when the statistic reaches the critical value tα, but failing to reject H0 does not establish that the algorithms have equal performance.The failure may reflect insufficient evidence caused by noise or a type-II error.
3.1 T-test and Welch’s t-test: Increasing sample size N reduces type-II error β at fixed α by making the estimated performance difference more precise.Reducing α raises the critical t-value and increases β, whereas larger samples narrow the sampling distributions.
3.2 Bootstrapped confidence intervals: Bootstrap confidence intervals avoid assumptions about the distribution of Xdiff by resampling observed differences and testing whether the resulting interval contains 0.The method assumes the sample is large enough to represent the underlying distribution, and bootstrap sample counts should be large, such as B > 1000.
3.2 Bootstrapped confidence intervals: A bootstrap test declares a statistically significant difference when its 100 · (1 −α)% confidence interval excludes 0, indicating a positive or negative difference.The interval is computed from bootstrap means using percentile bounds, such as the 2.5th and 97.5th percentiles when α = 0.05.
3.2 Bootstrapped confidence intervals: With five seeds per algorithm, both tests reported a significant difference at α = 0.05, yet the comparison was a type-I error because both algorithms were identical DDPG implementations.The example illustrates that the nominal α controls false positives only when the test assumptions hold.
3 Choice of the appropriate statistical test: The t-test analytically evaluates p-values and confidence intervals under t-distribution assumptions, whereas bootstrap methods estimate confidence intervals empirically without assuming a performance distribution.Welch’s test relaxes the equal-variance assumption made by the standard two-sample t-test.

4 In theory: power analysis for the choice of the sample size

The paper presents a three-step power-analysis workflow: estimate variability in a pilot, choose seeds to control type-II error, then apply statistical tests. In the example, increasing from 5 to 10 seeds makes the algorithm difference detectable with tighter confidence intervals.

4 In theory: power analysis for the choice of the sample size: Power analysis chooses N so the probability of detecting an effect size ϵ is 1−β while significance level α remains fixed.For Welch’s t-test, β depends on α, ϵ, empirical standard deviations, and N.
4.1 Step 1 - Running a pilot study: A pilot study runs both algorithms for n seeds to estimate empirical means and standard deviations needed for power calculations.In the example, five seeds produce (x1, x2) = (3523, 4905) and (s1, s2) = (1341, 990).
4.1 Step 1 - Running a pilot study: The N = 5 pilot yields conflicting tests: Welch’s t-test has p = 0.1, while the bootstrap interval [795, 2692] excludes zero.The t-test therefore does not reject H0, whereas the bootstrap test does.
4.2 Step 2 - Choosing the sample size: With N = 5, detecting the observed effect ϵ = 1382 has β = 0.51; increasing to N = 10 lowers β to 0.19, meeting the β = 0.2 requirement.The calculation uses α = 0.05 and empirical standard deviations (1341, 990).
4.3 Step 3 - Running the statistical tests: With N = 10, both tests reject H0, with p = 0.0037 for Welch’s t-test and a bootstrap interval of [732, 2612].Compared with N = 5, the larger sample makes the difference more visible and produces tighter confidence intervals.

5 In practice: influence of deviations from assumptions

Empirical checks show that deviations from statistical-test assumptions can misestimate both type-I and type-II error rates. Small samples distort bootstrap false-positive estimates and standard-deviation estimates can substantially understate the sample size needed for reliable power.

5.1 Empirical estimation of the type-I error: For small N, empirical false-positive rates exceed α = 0.05; the bootstrap test should not be used with fewer than 20 samples.The bootstrap test is especially sensitive to small samples, while the Welch test can slightly under-evaluate type-I error for non-normal data.
5.1 Empirical estimation of the type-I error: At N = 5, the bootstrap test’s false-positive probability is around 10%, while the Welch test’s is above 5%.The Welch test is harder to pass at small N, but non-normal data may still cause slight under-evaluation of the empirical false-positive rate.
5.2 Influence of the empirical standard deviations: Low-sample standard-deviation estimates are variable and underestimate σ = 1 on average, producing lower β estimates and an underestimated required N.With n = 5 preliminary samples, the reported average error is −0.059 and a larger possible error is −0.40.
5.2 Influence of the empirical standard deviations: Underestimated standard deviations can reduce the prescribed sample size from N = 17 to N = 7 for detecting ϵ = 0.9 at β = 0.2.The comparison assumes correctly estimated versus under-evaluated standard deviations in Figure 8.
5.2 Influence of the empirical standard deviations: Increasing the preliminary-study size reduces standard-deviation underestimation and improves estimates of β and the required experiment size.The authors also recommend choosing N systematically larger than the power-analysis prescription.
5 In practice: influence of deviations from assumptions: Statistical-test error rates should be empirically checked because assumptions about distributions and sample sizes may be unreliable.The paper specifically identifies inaccurate α estimates and underestimated N for type-II-error requirements as practical risks.

6 Conclusion

The paper formalizes statistical comparison of deep-RL algorithms and explains how to choose random-seed sample sizes for both error types. Its conclusion emphasizes empirical safeguards against assumption violations and inaccurate power estimates.

6 Conclusion: The paper defines type-I and type-II errors, proposes statistical tests for performance differences, and details how to select random-seed sample sizes.The stated goal is to satisfy requirements for both error types when comparing two RL algorithms.
6 Conclusion: The paper challenges Welch-test and bootstrap assumptions, finding inaccurate false-positive rates and strong dependence of power-based N on standard-deviation accuracy.It recommends at least N = 20 for bootstrap testing, tighter α, and larger N than power analysis prescribes.
6 Conclusion: Welch’s t-test is recommended over the bootstrap confidence-interval test.This recommendation appears among the paper’s practical conclusions.
6 Conclusion: Using α < 0.05 is recommended to keep the empirical type-I error below 0.05.The tighter threshold compensates for possible underestimation caused by violated test assumptions.
6 Conclusion: Multiple-comparison correction is recommended to avoid false positives growing linearly with the number of experiments.This recommendation addresses repeated statistical testing across experiments.
6 Conclusion: A pilot study should use at least n = 20 samples to estimate both algorithms’ standard deviations robustly.The recommendation targets inaccuracies in standard-deviation estimates used for power analysis.
6 Conclusion: The experimental sample size N should exceed the power-analysis prescription to compensate for standard-deviation-estimation inaccuracies and reduce type-II errors.This directly addresses the observed tendency of inaccurate variance estimates to understate the needed N.