PaperlayerView exact revision on arXiv

Source-linked AI summary

Rejection Odds and Rejection Ratios: A Proposal for Statistical Practice in Testing Hypotheses

M. J. Bayarri, Daniel J. Benjamin, James O. Berger, Thomas M. Sellke

arXiv:1512.08552v1stat.ME

TL;DR

The paper addresses how to interpret evidence from hypothesis tests when p-values alone can be misread and may omit power and prior odds. It proposes pre- and post-experimental rejection odds, reporting that the post-experimental measure has frequentist justification and is more accurate than the pre-experimental measure.

  • Problem

    P-values can be misinterpreted as the probability of the observed data under the null, while common interpretations may leave power and prior odds out of the evidential assessment.

    Source: §1 · p. 3, §2.1 · p. 8, §3 · p. 13

  • Method

    The paper proposes odds of correct versus incorrect null rejection, using power and Type I error before an experiment and prior odds times a Bayes factor afterward.

    Source: §1 · p. 3

  • Results

    The post-experimental odds have full frequentist justification and are described as much more accurate than the pre-experimental measure.

    Source: §1 · p. 4, §6 · p. 30

  • Takeaways & Limitations

    Researchers can adopt modifications using standard tests and basic tools, with options ranging from p-values and power calculations to Bayesian analysis.

    Source: §1 · p. 2

    The paper focuses on precise hypotheses, where the null is a lower-dimensional subspace of the alternative; its highlighted p-value problem is muted for directional comparisons such as θ < 0 versus θ > 0.

    Source: §1 · p. 2

Abstract

from arXiv · show

Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 years. We propose, as an alternative, the use of the odds of a correct rejection of the null hypothesis to incorrect rejection. Both pre-experimental versions (involving the power and Type I error) and post-experimental versions (depending on the actual data) are considered. Implementations are provided that range from depending only on the p-value to consideration of full Bayesian analysis. A surprise is that all implementations -- even the full Bayesian analysis -- have complete frequentist justification. Versions of our proposal can be implemented that require only minor modifications to existing practices yet overcome some of their most severe shortcomings.

1 Introduction

The introduction proposes reporting odds that distinguish correct from incorrect rejection, with pre-experimental and post-experimental forms. It presents this framework as a practical response to shortcomings in hypothesis testing, including p-value misinterpretation and ignored power, prior odds, and stopping rules.

1 Introduction

  • For precise tests of a point null against a broader alternative, the paper proposes odds of correct versus incorrect rejection with a dual frequentist/Bayesian interpretation.The authors define precise testing as a lower-dimensional null within the alternative; their chemotherapy-plus-steroids example allows the effect to be essentially zero.Source: §1 · p. 2
  • The proposal targets omitted power and prior probabilities, p-value misinterpretation, and ignored optional stopping, while setting aside effect-size questions.The paper focuses on the testing problem itself rather than whether effect size is more important than effect existence.Source: §1 · p. 3
  • The pre-experimental rejection ratio is power divided by the significance threshold, intended to represent evidence for H1 over H0 before data collection.The authors say it addresses the failure to incorporate power when interpreting evidence.Source: §1 · p. 3
  • Pre-experimental rejection odds multiply that ratio by prior odds, implying more stringent thresholds when the alternative has low prior probability.The introduction also links low prior odds to generally larger sample sizes and warns that conventional 0.05 testing can mislead when power or prior odds is low.Source: §1 · p. 3
  • After data collection, the post-experimental rejection ratio is the Bayes factor; the paper argues it avoids common misreadings of p-values as the probability of the null or observed data.The post-experimental odds are the prior odds times the Bayes factor.Source: §1 · pp. 3–4
  • The authors claim a frequentist justification for the Bayes factor: its prior prioritizes where to maximize power while the procedure retains frequentist error control.They describe this result as novel to their paper and as a bridge between frequentist and Bayesian testing.Source: §1 · p. 4
  • The recommendations span practice from p-values and power calculations to prior-based Bayesian analysis, with a p-value-only Bayes factor bound among the options.The authors say the bound needs no specified prior and can be calculated from a standard test's p-value; the recommended measures are the pre-experimental ratio for designs and post-experimental ratio for results.Source: §1 · pp. 2, 5, 6
  • The post-experimental odds approach is presented as resistant to optional stopping because the stopping-related likelihood factor cancels in the ratio.The authors state that the post-experimental odds are unaffected by optional stopping and retain their frequentist justification.Source: §1 · p. 5

2 The Pre-Experimental Odds Approach: Incorporating the Anticipated Effect Size and Prior Odds in Interpretation of Statistical Significance

The section proposes interpreting statistical significance through rejection odds that combine power and Type I error with prior odds. It shows how this framework informs study planning and significance thresholds.

2.1 Pre-Experimental Rejection Odds: the Correct Way to Combine Type I Error and Power

  • Pre-experimental odds combine prior odds with the rejection ratio, separating prior belief in H1 from the experiment’s evidentiary contribution.The rejection ratio is the probability of rejecting H0 under H1 divided by that probability under H0.Source: §2.1 · pp. 7–8
  • A rejection ratio of 5 : 1 offers limited confidence when power is 0.25 and α = 0.05.The authors argue that power remains relevant when interpreting a statistically significant result.Source: §2.1 · p. 8
  • Meta-analytic effect sizes can be biased upward by publication bias, winner’s curse, or questionable research practices, and can also be shifted toward the null by unusually noisy outcomes.The authors caution against relying uncritically on meta-analyses when choosing an anticipated effect size.Source: §2.1 · pp. 9–10
  • A 16 : 1 rejection-ratio planning threshold aligns with conventional 80% power at α = 0.05.The authors recommend planned sample sizes sufficient to achieve an adequate rejection ratio.Source: §2.1 · pp. 10–11
  • 280 participants per condition are needed for 80% power at effect size r = 0.21, despite typical samples often being below 50 per condition.The power calculation assumes the appropriate anticipated effect size for the research area.Source: §2.1 · p. 11, Table 2 · p. 11

2.2 Setting the Significance Threshold α

  • When prior odds are below 1 : 20, a rejection at α = 0.05 cannot make H1 more likely than H0, even at maximum power.The maximum possible rejection ratio at α = 0.05 is 20 : 1.Source: §2.2 · p. 12
  • Low prior odds call for a more stringent significance threshold; the section notes psychology domains may need thresholds below 0.05 and larger samples.The authors describe the parapsychology and genome-wide association examples as extreme relative to experimental psychology.Source: §2.2 · pp. 12–13
  • A genome-wide association study proposed α = 5 × 10^-7, reporting 21 genome/disease associations, virtually all replicated; the later convention became α = 5 × 10^-8.The cited subsequent work reports a growing number of robust findings at the tighter threshold.Source: §2.2 · pp. 12–13

3 The Post-Experimental Rejection Odds Approach: Finding the Rejection Odds Corresponding to the Observed Data

This section presents post-experimental rejection odds and ratios as data-dependent reports of how strongly observed data support the alternative relative to the null, in contrast to p-values. It describes the rejection ratio as a frequentist report that depends on a prior, with each prior inducing a procedure whose post-experimental ratio equals the pre-experimental ratio in expectation.

3 The Post-Experimental Rejection Odds Approach: Finding the Rejection Odds Corresponding to the Observed Data

  • Post-experimental rejection odds compare the hypotheses’ probabilities given the observed data, with the rejection ratio (Bayes factor) reflecting the data rather than the rejection region.The approach can also be implemented in fully frequentist and default versions that do not require specifying prior distributions.Source: §3 · p. 14, §3.1 · p. 14

3.1 Post-Experimental Odds

  • For the vaccine example, the observed z = 2.06 indicates odds of 4 : 1 or 5 : 1 for H1, while no prior choice can produce a rejection ratio above 8.35.The pre-experimental ratio was 9 : 1, so it did not represent the evidence in the observed data.Source: §3.1 · pp. 15–16

3.2 A Simple Bound on the Post-experimental Rejection Ratio (Bayes Factor), Requiring Only the p-value

  • When p ≤1/e and the p-value is proper, a p-value-only method gives an upper bound on the post-experimental rejection ratio, not the ratio itself.The bound is useful when a Bayes factor cannot be computed, but its assumptions and upper-bound status matter for interpretation.Source: §3.2 · pp. 17–18
  • A p-value of 0.05 could correspond to a post-experimental rejection ratio of at most 2.44 : 1, while p = 0.01 could correspond to at most 8.13 : 1, below 16 : 1.For the low-powered example, p = 0.001 gives Rpost(x) ≤ 52.9, but this upper bound alone does not allow a strong significance claim; in low-powered studies, the Bayes factor bound may be far too high.Source: §3.2 · pp. 17–18

3.3 The Surprising Frequentist/Bayesian Synthesis

  • Under H0, the long-run average post-experimental rejection ratio over rejected cases equals the pre-experimental ratio, giving Bayes factors a frequentist accuracy interpretation.Under H1, an analogous identity applies to reciprocal ratios, but it averages across the marginal alternative rather than a fixed true alternative; the equivalence also extends to composite nulls with suitable shared invariance.Source: §3.3 · pp. 21–23
  • Each prior distribution induces an Rpost(x) procedure whose post-experimental rejection ratio equals Rpre in expectation, making it a valid frequentist procedure with prior-dependent power characteristics.A prior can be centered on an alternative value where power is desired, including a value viewed as practically different from the null.Source: §3.3 · p. 23
  • Rpre is the uniformly worst valid conditional-frequentist report among nondecreasing rejection-ratio curves because its constant value does not distinguish data with different evidential strength.Rpre cannot be obtained from any prior distribution; decreasing curves are nonsensical because they report lower ratios for more extreme data.Source: §3.3 · pp. 23–24
  • Rpost(x) is a superior frequentist report to Rpre because it reflects the strength of evidence in the actual data rather than averaging over all possible data in the rejection region.Result 2 establishes that a frequentist is entitled to report Rpost(x) as well as Rpre.Source: §3.3 · pp. 23–24
  • In the GWAS example, Bayes factors Rpost(x) for 21 claimed associations ranged from 1/10 to 1068; post-experimental odds ranged from 1 to 10 against H1 to overwhelming odds in favor.The authors say these data-dependent odds seem preferable to always reporting 10 to 1, especially because reporting Rpost(x) is every bit as frequentist as reporting the pre-experimental rejection ratio.Source: §3.3 · p. 24

4 Choosing the Priors for the Post-Experimental Odds

The paper compares ways to choose priors for post-experimental rejection odds, from existing power assumptions and objective priors to simpler p-value bounds. It warns that the last two approaches can overstate support for the alternative hypothesis.

Prior options

  • The prior-choice question arises because post-experimental odds require prior odds and a prior distribution for θ under H1.The authors frame the section as addressing which priors researchers should choose.Source: §4 · p. 24
  • Researchers may use a subjective prior when available, or use the prior implicit in the experiment’s power calculation.Power calculations may use a weight function or a point effect size.Source: §4 · pp. 24–25
  • The intrinsic prior is constructed from an objective estimation prior and yields a proper prior for the testing problem.The supplied passages introduce the construction but do not include its full formula.Source: §4 · p. 25

Choosing among approaches

  • When researchers can specify a prior distribution or power weight function, Option 1 is the preferred choice and remains fully frequentist.The prior or weight function indicates where high power is desired.Source: §4 · p. 25
  • Option 2 uses the power assumptions from experiment design, but point-based power can leave the study underpowered when the true effect differs from the chosen point.If the actual effect is near the chosen point, post-experimental power is maximized.Source: §4 · p. 25
  • When neither a subjective prior nor power-based prior is available, the authors regard the intrinsic prior approach as attractive.They point to an existing literature discussing its virtues.Source: §4 · p. 26
  • The last two approaches report larger rejection ratios in favor of H1 than the data support, from both Bayesian and frequentist perspectives.The authors note that these approaches are easy to implement, especially the p-value bound, but describe their answers as biased in favor of H1.Source: §4 · p. 26
  • The practical prior-choice problem is not a compelling reason to reject post-experimental odds, because options range from modeling priors to a p-value bound that avoids specifying π(θ).The bound is an upper bound on post-experimental rejection odds and is described as reasonably accurate for well-powered experiments.Source: §4 · pp. 25–26

5 Post-Experimental Rejection Ratios Are Immune to Optional Stopping

Optional stopping can inflate the apparent significance of repeated p-value checks, while the Bayes factor does not depend on the stopping rule. The authors argue that post-experimental rejection ratios therefore remain interpretable even when stopping is undisclosed.

Optional stopping and p-values

  • Ignoring optional stopping while repeatedly checking unadjusted p-values can make significance likely and can eventually guarantee p = 0.05 even when H0 is true.The example describes sequential additions to a sample and repeated checks, though its probability value is incomplete in the supplied passage.Source: §5 · p. 26

Frequentist analysis

  • A frequentist must include the stopping rule when evaluating Type I error, so the pre-experimental rejection ratio depends on that rule.Power is defined similarly, and the rejection ratio is denoted Rpre.Source: §5 · p. 27

Post-experimental odds

  • The Bayes factor is unchanged by optional stopping because the stopping-rule factor appears in both likelihoods and cancels.The likelihood ratio is the same for stopped data and data treated as a predetermined fixed sample.Source: §5 · p. 27
  • Bayes factors permit optional stopping without penalty, including when there is no sampling plan or the analyst does not know it.The authors state that undisclosed optional stopping cannot fool someone using the Bayes factor to assess significance.Source: §5 · pp. 27–28

Example

  • With p = 0.05, the example’s conventional objective Bayesian analysis gives a Bayes factor such as Rpost = 2, which the authors say does not imply strong evidence.The passage presents this as an optional-stopping example with the stopping rule unreported.Source: §5 · p. 28

Frequentist justification

  • Ignoring the stopping rule when reporting a Bayes factor is valid as a frequentist report because the post-experimental ratio conditional on rejection does not depend on that rule.The authors distinguish this result from calculating the pre-experimental ratio, which requires knowing the rejection region and stopping rule.Source: §5 · p. 28

Practical boundary

  • If optional stopping is undisclosed, the rejection region may be unknown, preventing calculation of the pre-experimental ratio; post-experimental ratios reduce this potential to mislead.The authors describe this as a minor issue when post-experimental ratios are reported.Source: §5 · p. 28

6 Summary: Our Proposal for Statistical Hypothesis Testing of Precise Hypotheses

The authors recommend reporting rejection ratios at design and results stages, with choices ranging from power-based calculations to Bayesian analyses. They also recommend preregistering key parameters and adjusting thresholds to account for prior odds.

Recommendations

  • Report the pre-experimental rejection ratio with the design and the post-experimental rejection ratio with the results.The paper presents these as its two central recommendations, with implementations ranging from minor changes to full Bayesian inference.Source: §6 · p. 28

Pre-experimental ratio

  • The pre-experimental rejection ratio is power divided by Type I error and summarizes the expected persuasiveness of significant results.The authors recommend reporting it when presenting the design.Source: §6 · p. 29

Prior odds

  • Lower prior odds for the alternative call for a higher pre-experimental rejection ratio, which at a 0.05 threshold requires a higher-powered experiment.The authors recommend reporting prior odds or a reasonable range of them.Source: §6 · p. 29

Significance thresholds

  • A 0.05 Type I error threshold should not be universal; hypotheses with higher prior odds should face a more stringent threshold.The appropriate threshold can be calculated given researchers’ prior odds.Source: §6 · p. 29

Post-experimental ratio

  • After observing the data, report the Bayes factor as the post-experimental counterpart to the pre-experimental rejection ratio.The Bayes factor compares the likelihood of the observed data under the alternative with its likelihood under the null.Source: §6 · pp. 29–30

Post-experimental options

  • The p-value bound 1/[−ep log(p)] offers a simple upper bound on post-experimental rejection odds and is reasonably accurate for well-powered experiments.It requires only the p-value from a standard statistical test.Source: §6 · p. 30
  • Researchers can calculate an actual Bayes factor using a statistical model and a prior for effect size, including a prior justified by power calculations.The authors recommend reporting prior odds or ranges where subjective priors are acceptable and identify post-experimental odds as their top pick.Source: §6 · p. 30

Preregistration

  • Preregister anticipated effect sizes, significance thresholds, pre-experimental ratios, and prior odds to help avoid hindsight bias and ex-post parameter changes.The authors say preregistration would make data analysis more credible.Source: §6 · p. 30

Appendix 1: Proof of Result 2

The appendix begins a proof by considering an expected value under the null and then states a rejection-region probability in terms of power.

Proof

  • The proof begins with the expected value under the null hypothesis.The supplied passage does not show the expression that follows.Source: Appendix 1: Proof of Result 2 · p. 35
  • The appendix says the result follows from additional steps and concludes that the desired result follows.Those intermediate expressions are not stated in the supplied passages.Source: Appendix 1: Proof of Result 2 · pp. 35–36
  • The rejection-region integral is given as 1 − β(θ), relating that probability to power under the alternative.The passage introduces testing H0 against an alternative but does not preserve the full notation.Source: Appendix 1: Proof of Result 2 · p. 35
Loading 1512.08552v1…