Hypothesis testing with e-values

Standard Probability Notation; Standard Mathematics Notation; Notation Specific to the Book; Abbreviations; Conventions

• The reference material defines probability, mathematics, and book-specific notation, lists abbreviations, and sets conventions for symbols, vectors, inequalities, and expectations.The authors recommend writing “e-value” with a lowercase, non-italic e, while noting that journal styles may differ.Source: Standard Probability Notation · p. 9, Standard Mathematics Notation · p. 10, Notation Specific to the Book · p. 11

List of examples of e-values

• The examples list spans e-values for likelihood-ratio and mean testing, sequential inference, multiple testing, and risk measures.The list includes examples across chapters, from simple hypothesis tests to value-at-risk and expected-shortfall tests.Source: List of examples of e-values · p. 13

Introduction

• E-values are nonnegative statistics with expectation at most one under the null, and larger values indicate more evidence against it.For a simple null, e-values are likelihood ratios or dominated by them; composite-null e-values generalize likelihood ratios but can require nontrivial construction.Source: Introduction · p. 15

1.1 The three roles of e-values

• E-values serve methodological, technical, and foundational roles, from constructing tests and inference procedures to underpinning FDR control and risk-controlled decisions.The book also connects them to reverse information projection and nonnegative supermartingales, and characterizes when powered tests and bounded powered e-variables exist.Source: §1.1 · p. 16

1.2.1 E-values and p-values are statistical cousins

• E-values and p-values are statistical cousins: their classes of tests and variables exist under the same conditions, with p-variable existence equivalent under a mild assumption.Calibration can convert between them; e-to-p calibration preserves threshold decisions, and a particular p-to-e calibrator preserves decisions at a fixed level.Source: §1.2.1 · p. 17
• E-values support data-dependent testing decisions, optional continuation with multiplicative combination, and inference when the sampling scheme is unknown.The section contrasts these capabilities with limitations of p-values, whose validity may depend on a fully specified design and which are difficult to merge across sequential studies.Source: §1.2.1 · pp. 17–18
• Compound e-values underpin multiple-testing results, including representations of FDR-controlling procedures and admissible merging of dependent p-values.For fixed alternatives, a log-optimal e-value exists for any composite null, whereas uniformly most powerful p-values are limited to special cases.Source: §1.2.1 · p. 18

1.2.2 Power and e-power

• Power comparisons depend on the conversion: tests from an e-value and its calibrated p-value agree, as do p-value tests and their all-or-nothing e-value at level α, while another calibrator can reduce rejection probability.E-values and p-values do not have power by themselves; power is the rejection probability under a specified alternative.Source: §1.2.2 · p. 18
• Any level-α test can be represented by thresholding an e-variable at 1/α, so using e-values does not by itself lose expressibility or power.The book prefers log-optimal e-values, which maximize E_Q[log E] and avoid zero; logarithmic growth also determines how quickly products of sequential e-values accumulate evidence.Source: §1.2.2 · pp. 18–20
• Power-optimal all-or-nothing e-values have e-power minus infinity, while log-optimal e-values instead maximize expected log evidence under the alternative.The logarithm is motivated as a natural evidence measure that is easy to update as evidence accumulates.Source: §1.2.2 · p. 19

1.3 Mathematical notation and conventions

• The notation uses P and Q for null and alternative distribution sets, P† for the data-generating distribution, and Xt for the first t observations.The text also specifies standard measure, distribution, and set notation, while noting that observations need not be iid.Source: §1.3 · p. 20

1.4 Definitions of e-variables, p-variables, and tests

• An e-variable has null expectation at most 1, a p-variable has null probability P(P ≤ α) at most α, and a level-α test has null rejection probability at most α.A realized large e-value is evidence against the null; the book treats e-values as usable without relying on p-values and notes their role in sequential testing.Source: §1.4 · pp. 21–22

1.5 Betting interpretation of e-values

• An e-variable is a valid bet against the null: Skeptic stakes one dollar and receives E(ω) if outcome ω occurs.Under an alternative Q, Skeptic suggests the bet when E_Q[E] > 1; the realized e-value is also called a betting score.Source: §1.5 · pp. 22–23

1.6 A first example: likelihood ratio

• Likelihood ratios provide e-variables for simple alternatives, and the normal and Bernoulli examples yield exact e-values that increase with the evidence favoring the alternative.The normal example’s corresponding Neyman–Pearson p-value decreases with the sample sum; the Bernoulli likelihood-ratio e-values also form e-processes.Source: §1.6 · pp. 23–24
• With unknown but non-cherry-picked sampling design, the likelihood-ratio e-variable remains valid, whereas the same data can yield different standard p-values under different designs.For the displayed data, the standard p-value is 0.035 if eight observations were planned and 0.078 if collection stopped at five heads; e-process validity does not cover cherry-picking or selective reporting.Source: §1.6 · p. 24

An illustration

• At n=500, adaptive estimates and a mixture are more powerful than misspecified or random alternatives and have the best performer’s asymptotic log-scale slope.The comparison is between methods (iv) and (v) and (vi), versus methods (ii) and (iii); Figure 1.6 plots the processes on a logarithmic scale.Source: Figure 1.6 · p. 25, An illustration · p. 25

1.7 A second example: the soft-rank e-value

• The soft-rank e-value transforms exchangeable test statistics into nonnegative scores and uses their normalized score sum, yielding a valid e-variable whose reciprocal is a smoothed rank.The direct p-value is always smaller, so the soft-rank p-value offers no advantage at the same single-test threshold; the text points to later advantages for e-values in multiple testing and confidence intervals.Source: §1.7 · pp. 26–27

1.8.1 Who invented e-values?

• The book treats e-values as likelihood ratios in simple testing and as generalizations for composite settings, while noting that generalized likelihood ratios and some Bayes factors are not e-values.The book focuses mainly on the non-sequential setting and reserves Chapter 7 for a glimpse of the sequential framework.Source: §1.8.1 · pp. 27–28

1.8.2 Ville, Wald, Kelly, Robbins, Cover, and Vovk

• Ville connected martingales with testing events of measure zero; Wald developed sequential analysis, while Kelly advanced log-optimality and Robbins linked e-values to confidence sequences.The authors distinguish their focus on evidence processes and guarantees at any stopping time from Wald’s single stopping rule and Robbins’ level-α tools.Source: §1.8.2 · p. 28
• Cover developed log-optimal portfolios and regret bounds comparing adaptive rebalancing with the best constant rebalanced portfolio in hindsight.The book gives Cover less central attention because it only provides glimpses of the sequential setting.Source: §1.8.2 · pp. 28–29

1.8.3 Recent progress on e-values

• The 2018–2020 resurgence included e-processes and time-uniform bounds for sequential tests and confidence sequences, betting and log-optimality, and game-theoretic foundations.The overview names Howard, Ramdas, McAuliffe, and Sekhon [2020, 2021], Kaufmann and Koolen [2021], Shafer [2021], and Shafer and Vovk [2019], while warning that its list is incomplete and biased.Source: §1.8.3 · p. 29
• The 2019 work identified an e-process definition, advocated log-optimality and reverse information projection, and highlighted meta-analysis; the book collects these and later results.Grünwald, De Heide, and Koolen [2024a] described an e-process as a nonnegative process that is an e-value at any stopping time. The book’s bibliographic notes are relevant but not exhaustive.Source: §1.8.3 · pp. 29–30
• Recent work established results on e-value combination and calibration, universal inference, dependent p-value merging, e-BH, SAVI admissibility, and nonparametric betting.Vovk and Wang [2021] proved results on combining e-values and converting between e-values and p-values; Wasserman, Ramdas, and Balakrishnan [2020] developed universal inference. Vovk, Wang, and Wang [2022] linked admissible p-value merging to e-values, and Wang and Ramdas [2022] showed e-BH controls FDR under arbitrary dependence. Ramdas, Ruf, Larsson, and Koolen [2020] gave SAVI admissibility conditions; Waudby-Smith and Ramdas [2024] developed betting methods for bounded-mean estimation.Source: §1.8.3 · p. 30

1.8.4 Unfortunate name clashes: other e-values in the literature

• The book recommends distinguishing its expectation-based e-values from other uses of “e-value,” noting that “e-variable” and “e-process” have no known clashes.The literature also uses the name for BLAST’s expected hits [Altschul et al., 1990], a sensitivity-analysis measure [VanderWeele and Ding, 2017], and a Bayesian significance measure.Source: §1.8.4 · pp. 30–31

1.9 Road map

• Chapters 2–4 introduce null validity and conversion tools, alternative-hypothesis power and log-optimality, and the role of e-values in post-hoc testing and decision making.Chapter 3 identifies the likelihood ratio as the log-optimal e-variable for a simple null against a simple alternative; Chapter 4 contrasts what e-values and p-values can do.Source: §1.9 · p. 31
• Chapters 5–9 cover universal inference, the unique log-optimal numeraire and reverse information projection, anytime-valid inference, e-value merging, and compound e-values with e-BH.The numeraire is also the only e-variable that is a likelihood ratio between the alternative and an element of the effective null; Chapter 9 shows every FDR-controlling procedure can be represented through e-BH on compound e-values.Source: §1.9 · pp. 31–32
• Chapters 10–13 bridge asymptotics and nonasymptotic methods, develop dependent e-value and p-value combinations, and address e-confidence intervals and false coverage after selection.Chapter 11 presents e-weighted BH; Chapter 12 uses optimal transport duality and random or exchangeable improvements of Markov’s inequality; Chapter 13 discusses majority-vote merging of dependent confidence intervals.Source: §1.9 · p. 32
• Chapters 14–16 examine p-value/e-value duality and powered exact values, improvements to Markov’s inequality, and risk measures with nonparametric testing of statistical functions.The functions include the mean, variance, and quantiles.Source: §1.9 · p. 32

2.1 Testing using Markov’s inequality

• With an e-variable E, rejecting when E≥1/α gives a level-α test, but its type-I error is usually below α and it generally differs from the test based on a p-value.The type-I error reaches α only when E is supported on 0 and 1/α with mean 1 under the null.Source: §2.1 · p. 33

2.2 Testing using distributional knowledge

• When the null distribution of E is known, a level-α cutoff can be lowered using its tail probability; discontinuities may require strict rejection or randomization.When the null distribution is difficult to compute, partial distributional information can still bound the cutoff.Source: §2.2 · pp. 34–35

2.3 Relating p-values and e-values using calibrators

• Among nonrandomized transformations, the e-to-p calibrator is essentially unique, f(t)=min(1, 1/t), whereas p-to-e calibrators form a much richer class.For an atomless null, every e-variable can also be represented as a calibrator applied to an exact p-variable; if the e-variable is exact, the calibrator can be admissible.Source: §2.3 · pp. 35–37
• Bidirectional calibration can lose substantial evidence: p=0.01 becomes e=9 and then p′=1/9.Other admissible calibrators penalize very small p-values more, but may be useful when multiplying independent e-values.Source: §2.3 · p. 37

2.4 Randomized tests and calibrators

• Randomization yields a typically strictly smaller p-value, U/E, than 1/E, and thus a typically more powerful test.For bounded Y, the randomized inequality holds with equality when α≤1/C; Markov’s inequality has equality only for distributions supported on {0, 1/α}.Source: §2.4 · pp. 38–39

2.5 Admissible e-variables

• An e-variable is admissible if no other e-variable can dominate it by being at least as large everywhere and strictly larger with positive probability under a null.Source: §2.5 · p. 39

2.5.1 Admissible e-variables for a simple null are likelihood ratios

• For a simple null, every admissible e-variable is a likelihood ratio, and exact e-variables are admissible.The likelihood ratio is the density of a distribution absolutely continuous with respect to the null.Source: §2.5.1 · p. 39

2.5.2 Characterization of admissibility

• For general null families, admissibility has a necessary exactness condition and a full characterization using expectations under nulls that assign positive probability to events.The characterization is stated in terms of expectations over the sets Pδ_A.Source: §2.5.2 · pp. 39–40

2.5.2 Characterization of admissibility; 2.5.3 Admissible e-variables on product spaces

• Strong admissibility is stricter than admissibility for infinite null classes, coincides with it for finite classes, and is preserved by products of admissible e-variables when both null classes are finite.For product spaces, the product of strongly admissible e-variables is strongly admissible.Source: §2.5.2 · p. 41, §2.5.3 · p. 41

2.6.1 Monotone likelihood ratio

• Under a monotone likelihood ratio condition, the same e-variable covers several one-sided testing configurations, including normal mean tests; likelihood ratios also yield log-optimal e-variables in one-parameter exponential families.The cited examples include testing {N(θ, 1): θ ≤ 0} against {N(θ, 1): θ > 0}.Source: §2.6.1 · p. 42

2.6.2 Testing symmetry

• For symmetry tests, exact e-variables are admissible; multiple observations permit product, sign-based, and signed-rank constructions, while the one-observation e-variable is at most 2.The signed-rank construction uses the null’s uniform distribution over subsets of positive-observation ranks.Source: §2.6.2 · p. 43

2.6.3 Testing the mean and the variance; 2.6.4 Testing sub-Gaussian means

• Mean and variance constraints yield e-variables even with dependent observations, while zero-mean 1-sub-Gaussian nulls additionally permit Z^2, unlike one-sided sub-Gaussian nulls.For nonnegative observations with common marginal mean at most µ, (X1 + ··· + Xn)/(nµ) is an e-value under any dependence structure.Source: §2.6.3 · p. 44, §2.6.4 · p. 45

2.6.5 Constrained hypotheses; 2.6.6 Testing likelihood ratio bounds; 2.6.7 Testing exchangeability

• For integral-constraint nulls, finite constraint sets characterize e-variables by quasi-sure domination, while infinite sets use a weakly closed cone construction; likelihood-ratio bounds and exchangeability also admit e-variable constructions.For likelihood-ratio bounds, scaling dQ/dP0 by γ gives an e-variable; under exchangeability, q(Z)/q(Zπ) is exact and admissible.Source: §2.6.5 · pp. 45–46, §2.6.6 · p. 46, §2.6.7 · p. 47

2.7 Informal grids for realized e-values

• Jeffreys-style evidence grids suit natural likelihood-ratio-like e-values but not all-or-nothing e-variables, and p-values and e-values can point in opposite directions.The text cautions that p-values and e-values have no one-to-one correspondence; for δ = 5 and X = 2, it gives P ≈ 0.023 and E ≈ 0.08.Source: §2.7 · p. 47
• Calibration can often preserve direction, but p-values and e-values constructed directly from data need not have natural concordance.Consequently, the resulting methods are not always directly comparable.Source: §2.7 · p. 48

Bibliographical note

• The chapter combines results from important e-value papers with additional results derived for the book to provide a coherent treatment.Examples include the bounded-mean, sub-Gaussian, and exchangeability cases, whose prior work is attributed to Waudby-Smith and Ramdas [2024], Howard et al. [2020], Ramdas et al. [2020], and Vovk [2021], among others.Source: Bibliographical note · p. 48

3.1 Powered tests, e-values, and p-values

• For H0: µ ≤ 0 against H1: µ > 0, exp(X1 + ··· + Xn − n/2) is a powered e-variable but not uniformly e-powered; it is uniformly powered against µ ≥ ε for any ε > 0.If the alternative instead includes the boundary µ = 0 while the null is µ < 0, no powered e-variable or test exists.Source: §3.1 · p. 50
• For any α ∈ (0, 1), [0, 1/α]-valued e-variables correspond one-to-one with level-α tests; if a problem is testable (a powered test exists), it is testable with e-values.Source: §3.1 · pp. 50–51

3.2 A unified existence result

• Under Assumption (U), the chapter connects existence conditions for powered e-variables, powered p-variables, and powered tests.Condition (U) matters for constructing powered p-variables: on finite sample spaces, powered e-variables can exist even though powered p-variables cannot.Source: §3.2 · pp. 51–53

3.3 E-power and powered e-values

• For iid observations from N(µ, 1), the likelihood-ratio e-value has e-power nµ^2/2, and positive e-power yields asymptotic power-1 tests in the simple iid setting.E-power is the expected log e-value; for iid e-variables, it is also the process's almost-sure asymptotic log-growth rate.Source: §3.3 · pp. 53–54
• Positive e-power implies a powered e-variable, but the converse need not hold; for a simple alternative, a powered e-variable can be transformed to have positive e-power.The transformation is E → 1 − λ + λE for a sufficiently small λ > 0.Source: §3.3 · p. 54
• No bounded e-variable can be powered against Q, even though P lies in the closed convex hull of Q but not in Q or its convex hull.The example uses Qn approaching δ0 in total variation; averaging δ0 with Q1 yields P.Source: §3.3 · p. 55
• E-power can be counterintuitive: one e-variable has high Q-probability of large values but e-power −∞, while another exceeds 1 with probability 1/600 and has infinite e-power.For iid copies of the latter variable, the product diverges Q-almost surely.Source: §3.3 · p. 56

3.4 Conditions for the existence of powered e-variables

• The existence of powered e-variables and p-variables is characterized using total-variation separation of the closed convex hulls of the null and alternative distributions.For finite hypothesis sets, the stated condition reduces to disjoint convex hulls; without a common reference measure, the separation condition need not suffice.Source: §3.4 · pp. 56–57

3.5 Log-optimality and likelihood ratios

• When Q≪P, the likelihood ratio dQ/dP is the unique log-optimal e-variable for testing simple P against simple Q.In the Gaussian location example, θ=μ maximizes the e-power of En(θ).Source: §3.5 · p. 58

3.6 Conditional log-optimality and log-optimal calibrators

• For an exact p-variable uniform on [0, 1] under P, a log-optimal calibrator produces an e-variable at least as powerful as any other calibrator.The comparison is among e-variables calibrated from that p-variable for a given simple alternative Q.Source: §3.6 · p. 59

3.7 Asymptotic log-optimality: methods of mixtures and plug-in

• Mixture and plug-in methods can asymptotically attain the e-power of an oracle that knows the generating alternative.The mixture method requires full support over Q for asymptotic log-optimality and an easily evaluated integral; a plug-in method can use the posterior mean under a prior.Source: §3.7 · p. 60
• A non-scale-invariant e-variable can have positive e-power against some alternatives even with n=1.The example is an e-variable for the null family N(0, σ²), and its e-power is greater than one for Q=N(a, 1) when a is sufficiently large.Source: §3.7 · p. 61

3.8 Axiomatic justification of the e-power

• The axiomatic treatment ranks e-variables by performance under the alternative and requires comparison consistency under common scaling and a Kelly-style asymptotic power condition.The setup restricts attention to variables bounded above and away from zero and assumes an atomless alternative.Source: §3.8 · pp. 61–62
• Axioms 3.26 and 3.27 characterize e-power as a strictly increasing transform of EQ[log E], preserving the same ordinal comparisons.The geometric mean exp EQ[log E] is one such transform; it is nonnegative and exceeds 1 when e-power is positive.Source: §3.8 · pp. 62–63
• Unlike test power, which is the probability of rejection under the alternative, e-power’s value represents an asymptotic growth rate in a multiplicative setting.It also serves as a comparative notion for determining which e-variables are more powerful.Source: §3.8 · p. 63
• For an e-variable E, mixing toward 1 preserves positive or nonnegative e-power under the stated conditions, and finite e-power yields real-valued mixtures for λ < 1.If ΛQ(E) > 0, then ΛQ(1 − λ + λE) > 0 for λ ∈ (0, 1]; if ΛQ(E) ≥ 0, it remains ≥ 0 for λ ∈ [0, 1]. If ΛQ(E) < ∞, the mixture’s e-power is real for λ ∈ [0, 1). The mapping extends to nonnegative random variables when EQ[log E] is well-defined.Source: §3.8 · pp. 63–64

• Koning [2024] discusses the connection between tests and bounded e-variables, Vovk and Wang [2024b] define e-power, and Wang [2024a] discusses Example 3.16.Source: Bibliographical note · p. 64
• Kelly [1956] proposed log-optimality for betting and investment; Larsson et al. [2025a] established existence and uniqueness of a log-optimal e-variable without conditions on P or Q.Source: Bibliographical note · p. 64
• The result in Section 3.5 essentially dates to Breiman [1961], while its presentation follows Shafer [2021]; mixture and plug-in methods for composite alternatives were mentioned by Wald [1947].Robbins and Siegmund [1974] later derived strong connections between the mixture and plug-in methods.Source: Bibliographical note · p. 64
• Lai [1976], Pérez-Ortiz et al. [2024], and Wang and Ramdas [2025] derive and study the t-test e-variable; the latter paper gives the e-variable for n = 1.Source: Bibliographical note · p. 64
• The asymptotic optimality of the method of mixtures is related to redundancy and description length, with further optimality results inferable from Grünwald [2007].Source: Bibliographical note · p. 64
• The changepoint e-value in Example 3.25 is borrowed from Shin et al. [2024].Source: Bibliographical note · p. 64

4.1 Testing at data-dependent error levels

• An e-variable generates post-hoc-valid test families, and every such family is dominated by one generated from an e-variable.With the stronger post-hoc p-variable condition over α in (0, ∞), post-hoc p-variables are precisely reciprocals of e-variables.Source: §4.1 · pp. 66–67

4.2 Post-hoc decisions using p-values and e-values

• P-values are safe for binary decisions in fixed scenarios, but not for non-binary decisions or data-dependent scenarios where e-values provide the desired risk guarantee.The section frames decision making more broadly than binary acceptance or rejection, allowing the possible decisions or scenario to depend on data.Source: §4.2 · pp. 68–69

4.2.1 Decision rules based on p-values and e-values

• In ternary significance decisions, the p-value rule does not satisfy the risk constraint, whereas the e-value rule always does.The rules assign significant and very significant labels using thresholds α1 and α2, with 0 < α2 < α1 < 1.Source: §4.2.1 · p. 69

4.2.2 Binary decisions in fixed scenarios

• For binary decisions in a fixed scenario, p-value thresholding is the standard rule, while e-value thresholding is typically more conservative.The rules reject for P(X) ≤ γ and E(X) ≥ 1/γ, respectively.Source: §4.2.2 · p. 70

4.2.3 Non-binary decisions or data-dependent scenarios

• For non-binary decisions or data-dependent scenarios, e-value rules satisfy the risk constraint, while p-value rules may fail it.Under a strictly increasing loss under the null, the e-value rule is optimal when the scenario is known and at least as large as the optimizer under a worst-case constraint.Source: §4.2.3 · p. 72

4.3 Optional continuation of experiments

• After a likelihood-ratio test fails to reject, optional continuation is presented as a setting where an e-value approach is possible.The example notes that the original test's type-I error budget is exhausted, then introduces summarizing the data with a likelihood-ratio e-value from the start.Source: §4.3 · p. 73
• Scientists may repeat e-value batches indefinitely, multiply successive batch e-values, and monitor cumulative evidence, rejecting once it exceeds 1/α; Ville’s inequality controls type-I error, and at data-dependent stopping time τ, evidence is E(τ).Source: §4.3 · p. 73

4.4 Incentive-compatible principal-agent hypothesis testing

• When ineffective drugs are twenty times as numerous as effective ones and 5% of ineffective drugs are approved, at least half of approved drugs will be ineffective.An e-value-based statistical contract can discourage participation by malicious agents while making license profit depend on evidence; an inverse-p-value license has infinite expected profit under the null, whereas C/(2√P) is incentive compatible.Source: §4.4 · pp. 75–76

Bibliographical note

• Grünwald et al. [2024a] coined “optional continuation”; Ter Schure and Grünwald [2022] detailed its meta-analysis application, and Bates et al. [2022] studied incentive-compatible principal-agent testing.Shafer [2023] further explored optional continuation, while the incentive-compatible testing problem in Section 4.4 is attributed to Bates et al. [2022].Source: Bibliographical note · p. 76

The universal inference e-value

• Universal inference constructs an e-variable or e-process for composite nulls against composite alternatives without regularity conditions when maximum likelihood under the null can be calculated efficiently.The method is described as a reduction from hypothesis testing to maximum likelihood estimation under the null.Source: The universal inference e-value · p. 78, §5.1 · p. 78

5.1 Motivation: Irregular models

• Universal inference addresses irregular tests whose usual asymptotic guarantees may fail, including Gaussian-versus-mixture and log-concavity testing.The passages describe the Gaussian-mixture problem as having unidentified parameters and no straightforward classical asymptotic control; they also note polynomial-time computation of the maximum-likelihood log-concave density.Source: §5.1 · p. 78

5.2 Split likelihood ratio e-variable

• The split likelihood-ratio e-variable uses independent data splits to select an alternative on one split and evaluate its likelihood ratio against the null maximum likelihood estimate on the other.The alternative-selection method can vary without affecting validity, though it affects e-power; swapping split roles and averaging yields the cross-fit likelihood-ratio e-variable, and mixture alternatives can replace plug-in alternatives.Source: §5.2 · pp. 79–80

5.3 Subsampled likelihood ratio e-variable

• Averaging split likelihood-ratio e-variables preserves or improves e-power, and running averages can be tested sequentially with type-I error control, including termination at a data-dependent B.Averaging removes variability introduced by random splitting; the first sequential step coincides with the split LRT.Source: §5.3 · pp. 80–81

5.4 Exchangeable Markov’s inequality

• The exchangeable Markov inequality provides a nonasymptotic bound for exchangeable variables and extends to arbitrarily dependent variables by uniformly permuting them.Uniform sampling, with or without replacement, also induces the exchangeability needed to apply the inequality.Source: §5.4 · pp. 81–82

5.5 Universal confidence sets

• Inverting the universal inference test gives a (1 − α) confidence set without regularity conditions, and subsampling can reduce its expected diameter.The construction tests each point null and retains the parameter values not rejected.Source: §5.5 · pp. 82–83

An example: multivariate Gaussian data with identity covariance

• As α tends to zero at fixed dimension d, the optimal split proportion p* converges to 1, suggesting that nearly all data should be used for likelihood estimation; its effect on squared radius is greater in higher dimensions.Source: An example: multivariate Gaussian data… · p. 83, §5.7 · p. 87

5.6.1 Handling nuisance variables

• For a target function of nuisance parameters, profiling the likelihood on the testing data yields a (1 − α) confidence set for that target.The construction follows by mapping the confidence set for the full parameter to the target parameter.Source: §5.6.1 · p. 83

5.6.2 Smoothed likelihood

• When the null likelihood is unbounded, a smoothed likelihood can replace it; the resulting smoothed split LRT controls type-I error.The validity argument checks that the smoothed universal statistic is an e-variable.Source: §5.6.2 · pp. 84–85

5.6.3 Relaxed, powered and conditional likelihood

• Likelihood relaxations allow inference by upper-bounding the null maximum; power likelihoods support robustness to misspecification, and conditional likelihoods permit non-iid sampling when computable.The conditional likelihood need only be calculable on one data split given the other.Source: §5.6.3 · pp. 85–86

5.7 Numerical studies

• Subsampling yields confidence regions with smaller volume than split LRT regions, while the classical LRT regions are smallest; power ranks classical, subsampling, cross-fit, then split.For large B, subsampling regions are approximately spherical. At d = 1, subsampling and cross-fit power are nearly identical; their gap increases with d.Source: §5.7 · pp. 86–87, Figure 5.11 · p. 86
• Universal inference is valid without regularity conditions, whereas the LRT serves as a benchmark only when the Gaussian mixtures’ mixing proportions are known.The test concerns whether a mixture has one versus two components.Source: §5.7 · pp. 87–88
• The comparison includes split-LR UI and UMI-UI and subsampling-LR SUI, UMI-SUI, EMI-SUI, and EUMI-SUI, using standard, randomized, or exchangeable Markov inequalities.UMI denotes uniformly randomized Markov’s inequality; EMI-SUI and EUMI-SUI use exchangeable and exchangeable uniformly randomized versions, respectively.Source: §5.7 · p. 88
• UMI-SUI and EUMI-SUI increased absolute power over SUI by on the order of 15% for some µ values, and randomized Markov variants dominated their counterparts.The comparisons used 500 samples of size n = 500 across ten equally spaced µ values.Source: §5.7 · p. 88
• Split-LRT methods were markedly more conservative than the LRT, but universal inference remains usable for arbitrary mixture models where the LRT cannot easily yield a valid test.Source: §5.7 · pp. 87–88

Bibliographical note

• Wasserman et al. [2020] developed universal inference and showed universal confidence-set width scaling as d/n; Ramdas [2023] derived the exchangeable Markov inequality, whose testing implications were detailed by Ramdas and Manole [2025].The smoothed MLE idea is related to Seo and Lindsay [2013].Source: Bibliographical note · pp. 88–89
• Dunn et al. [2023] found universal sets match standard-set scaling, with radii less than twice as large; Strieder and Drton [2022] found their split-LRT ratio converges to 1/2 for a fixed parameter count, otherwise varying with tested-hypothesis dimension.Dunn et al. [2025] applied the method to log-concavity testing, for which no other valid test is known; Shi and Drton [2024] found the split likelihood-ratio statistic achieves the optimal (log log n)/n detection rate. Tse and Davison [2022] study power improvements for high-dimensional universal inference with many nuisance parameters. The Gaussian-mixture-model-selection problem was previously studied by [Ghosh and Sen, 1984, Chen and Li, 2009], and the limiting distributions used can be found in Goffinet et al. [1992].Source: Bibliographical note · p. 89

information projection

• The chapter’s duality centers on the numeraire e-variable and the reverse information projection, extending beyond point alternatives and composite nulls to general P and Q.The universal-inference e-variable is known to be asymptotically, but not finite-sample, log-optimal; the log-optimal e-variable exists uniquely and dominates it.Source: information projection · p. 90

6.1 The numeraire e-variable

• The numeraire is unique up to Q-nullsets, is equivalent to log-optimality, and for a simple null with Q ≪ P is the likelihood ratio dQ/dP.The chapter states that a numeraire exists without assumptions on P or Q; under the stated weak condition Q ≪ P, it is Q-almost surely finite.Source: §6.1 · pp. 90–91

6.2 Strong duality for finite P

• For finite P, the RIPr minimizes KL divergence to Q over Conv(P), and the numeraire is dQ/dP*.The finite-case argument obtains the minimizer over the probability simplex and establishes the strong-duality result.Source: §6.2 · p. 93

6.3 Reverse information projection

• The RIPr P* is uniquely determined up to Q-nullsets by dP*/dQ = 1/E*, and can be a sub-probability measure rather than a probability measure.The chapter says P* lies in the bipolar of P even when it does not lie in Conv(P).Source: §6.3 · p. 94, §6.3.1 · p. 95

6.3.1 The effective null hypothesis is the bipolar of P

• The bipolar P◦◦ is the effective null: it is the largest family of distributions with exactly the same e-variables as P.It can strictly extend Conv(P), and its members have total mass at most one.Source: §6.3.1 · pp. 94–95

6.3.2 A general strong duality result

• Under Q ≪ P, the numeraire E* and RIPr P* satisfy a general strong-duality relation, whose quantities may be infinite.The chapter also states characterizing properties of the RIPr.Source: §6.3.2 · p. 96
• The general strong-duality result identifies P* as the RIPr.Source: §6.3.2 · p. 96

6.4 Numeraire is the only e-variable that is a likelihood ratio

• A positive e-variable is the numeraire exactly when its reciprocal likelihood-ratio measure belongs to the effective null; this measure is maximal among equivalent null measures.The measure has expectation 1 under the numeraire, and no other equivalent effective-null measure can dominate it on every event with strict inequality somewhere.Source: §6.4 · p. 97

6.5 Numeraire dominates universal inference

• The numeraire dominates universal inference, strictly with positive probability for nondegenerate composite nulls, but universal-inference e-variables are coherent while numeraires generally are not.The numeraire also needs weaker assumptions, since universal inference requires a reference measure to define likelihoods; this creates a tradeoff between e-power and coherence or interpretability.Source: §6.5 · pp. 98–99

6.6 Connections between e-values and Bayes factors

• The numeraire is always an e-variable, even without reference measures, and dominates universal inference; when it is a Bayes factor, it is the unique such factor for the chosen alternative mixture.With neither hypothesis simple, Bayes factors are sometimes but usually not e-variables, and the numeraire is sometimes but usually not a Bayes factor; when only the null is simple, the Bayes factor, universal-inference e-variable, and numeraire coincide.Source: §6.6 · p. 100

6.7.1 Exponential family with one-dimensional sufficient statistic; 6.7.2 Monotone likelihood ratio

• In one-dimensional exponential families with θ1 > θ0, the null’s largest parameter θ0 gives the RIPr, and the likelihood ratio pθ1/pθ0 is the numeraire; the same conclusion holds under a monotone likelihood ratio.For the exponential-family result, the null parameter set is closed and has a largest element, and the RIPr condition follows from the stated moment inequality.Source: §6.7.1 · p. 101, §6.7.2 · p. 101

6.7.3 Testing symmetry

• For testing symmetry, the RIPr is the part of the symmetrized alternative absolutely continuous with respect to Q, and the resulting candidate is verified as the numeraire.The symmetrization itself need not be equivalent to Q; without symmetric support, the RIPr candidate may be a proper sub-probability measure, and the construction extends to alternatives without densities.Source: §6.7.3 · pp. 101–102

6.7.4 Testing bounded means

• For bounded-mean testing against a uniform alternative, the candidate numeraire is verified, and its RIPr belongs to the null.The log-optimal candidate is found at a unique interior λ* in (0, µ−1), characterized by a first-order condition.Source: §6.7.4 · pp. 102–103

6.7.5 Testing sub-Gaussian means

• For a 1-sub-Gaussian null with nonpositive mean and a unit-variance normal alternative of mean µ > 0, maximizing expected log e-value over the stated family gives λ* = µ.Source: §6.7.5 · p. 103
• For testing sub-Gaussian means, E* is the numeraire and its RIPr P* is standard normal; the example extends to σ-sub-Gaussian distributions for σ ≠ 1.Source: §6.7.5 · p. 103

6.7.6 Testing likelihood ratio bounds

• Under a likelihood-ratio bound γ, E* maximizes E_Q[log X] under the stated P* constraint and is therefore the numeraire for testing P against Q.Source: §6.7.6 · pp. 103–104

6.7.7 Testing exchangeability

• For exchangeability testing, q(Zπ) is an e-variable and the numeraire for P against Q; it is also the RIPr of q onto P.Source: §6.7.7 · p. 104

Bibliographical note

• The note traces RIPr from its first appearance to the unrestricted definition used in this chapter.Csiszár and Tusnády [1984] introduced it; Li [1999] proved uniqueness under assumptions and an inequality later yielding the numeraire e-variable property; Grünwald et al. [2024a] described its role in constructing optimal e-variables. Lardy et al. [2024] relaxed conditions, while Larsson et al. [2025a] removed all conditions by defining RIPr via the numeraire e-variable.Source: Bibliographical note · p. 104
• Grünwald et al. [2024a] propose GROW’s worst-case e-power criterion and REGROW, which resembles the chapter’s asymptotic log-optimality objective.The note characterizes GROW as pessimistic because worst-case tuning may not adapt to simpler alternatives with larger growth rates, while noting its joint information projection mathematics.Source: Bibliographical note · p. 104
• Numeraires for exchangeability, other group-invariant nulls, and one-parameter exponential families were derived across several works; the chapter follows Larsson et al. [2025a].Koning [2023b] and Larsson et al. [2025a] used different techniques for exchangeability; Pérez-Ortiz et al. [2024] treated other group invariance; Grünwald et al. [2024b] also derived the one-parameter exponential-family numeraire. Grünwald et al. [2024b] treat exponential families more generally; Csiszar and Matus [2003] discuss reverse information projection for them. The bounded-mean example varies one in Waudby-Smith and Ramdas [2024], with duality theory in Honda and Takemura [2010].Source: Bibliographical note · p. 104
• Theorem 6.13’s minimum KL divergence is related to but distinct from the bandit-literature KLinf metric, and their exact relationship remains unresolved.Agrawal et al. [2021] is cited as a recent example of the bandit-literature metric.Source: Bibliographical note · pp. 104–105
• Theorem 6.13’s strong duality also holds for some divergences beyond Kullback–Leibler, including a Rényi-divergence example in Larsson et al. [2025a, Section 6].The example uses ϕ(x) = x1−γ/(1 −γ) for γ > 1 instead of log E.Source: Bibliographical note · p. 105
• Bickel [2024] first observed that numeraire e-variables are generally logically incoherent, unlike universal inference.Source: Bibliographical note · p. 105

7.1 The problematic practice of peeking at p-values

• Under the null, repeatedly checking p-values and stopping at P_n ≤ α can lead almost surely to eventual rejection, making peeking a source of type-I errors.Because the data-collection protocol may not be reported, the issue can be difficult to detect from the data alone; the text identifies it as one factor in the reproducibility crisis.Source: §7.1 · pp. 106–107

7.2 Wald’s sequential probability ratio test

• 137.7 samples under the null and 139.3 under the alternative are the SPRT’s expected sample sizes, about half the LRT’s sample requirement.For the stated coin-testing setup, the LRT requires 280 samples to target type-I and type-II errors of 0.05; Table 7.1 compares the SPRT and LRT.Source: §7.2 · pp. 107–108, Table 7.1 · p. 108
• Wald optimizes error tradeoffs at a single stopping time, while SAVI provides evidence valid at any stopping time and emphasizes log-optimality.For simple nulls and alternatives, the likelihood-ratio martingale optimizes both objectives, though the text says this need not hold generally.Source: §7.2 · p. 108
• E-processes provide a framework beyond likelihood ratios: every sequential test, including one for a composite or nonparametric null, can be obtained using them.The framework focuses on sequentially updated evidence rather than primarily on binary decisions.Source: §7.2 · pp. 108–109

7.3 Test supermartingales, sequential tests, and e-processes

• An e-process has expected value at most one at every stopping time under every null distribution, equivalently being bounded by a family of test martingales.Test supermartingales are e-processes, but the converse does not generally hold.Source: §7.3 · p. 110

7.4 Optional stopping and Ville’s inequality

• Ville’s inequality may hold with equality or strict inequality, depending in general on whether the process overshoots 1/α at the stopping time.The text gives a continuous-time Brownian exponential martingale as an example where equality holds.Source: §7.4 · p. 111
• Ville’s inequality turns an e-process into a level-α sequential test by rejecting when it reaches 1/α, and every level-α sequential test has an e-process representation.The text presents e-processes as a complete framework for sequential testing and as interpretable evidence even without thresholding.Source: §7.4 · p. 111
• At any stopping time, switching between e-processes preserves e-process validity; Ville’s inequality also yields a p-process from an e-process via its running reciprocal minimum.The switch may depend on data or judgment, and the next process need not be specified until the switch occurs; switching may be repeated.Source: §7.4 · pp. 112–113

7.5.1 Log-optimality of likelihood ratio processes

• For a simple null P and alternative Q, the likelihood-ratio process is log-optimal among e-processes, including at stopping times finite Q-almost surely.At each fixed time it is also log-optimal among e-variables based on the first t observations.Source: §7.5.1 · pp. 113–114

7.5.2 Asymptotic log-optimality and the plug-in method

• The likelihood-ratio process grows asymptotically at rate KL(Q, P) > 0 under iid data, and no other e-process has a larger asymptotic growth rate.Adaptive plug-in processes are e-processes; in the normal model, the cited condition holds for the sample-mean estimator and for MAP with a normal prior.Source: §7.5.2 · pp. 114–116

7.6 Testing by betting

• In the betting view, requiring each bet to be a conditional e-variable makes wealth a test supermartingale; for a simple alternative, the likelihood-ratio bet is log-optimal.For composite nulls, the e-process interpretation takes the worst wealth across separate games, though the text notes practical and measure-theoretic difficulties with this interpretation.Source: §7.6 · pp. 117–118

7.7 The universal inference e-process

• Universal inference constructs an e-process using alternative predictions based on past data and a null maximum-likelihood estimate recalculated as observations arrive.The text states that its 1/α threshold does not depend on P and Q, unlike generalized likelihood-ratio methods, whose threshold must be adjusted for model complexity.Source: §7.7 · pp. 118–119

7.8 Sequential e-values and empirically adaptive e-processes

• Sequential e-values have conditional expectation at most 1 given the past, and their product process is an e-process.More generally, testing-by-betting constructions can build e-processes from sequential e-values.Source: §7.8 · p. 119
• Under iid alternatives, the empirically adaptive martingale is asymptotically Q-log-optimal, needs no specified Q, and retains level-α type-I control at threshold 1/α.When E_Q[E1] > 1, it grows to infinity; under approximately iid alternatives, it generally has good e-power.Source: §7.8 · pp. 120–121

7.9 Obtaining an e-process from e-values using time-mixture

• A probability-mass-weighted time-mixture turns sample-size-specific e-variables into an increasing e-process whose asymptotic growth matches the numeraire sequence.Source: §7.9 · pp. 121–122

7.10 E-processes avoid reasoning about hypothetical worlds

• Fixed-sample p-values can be invalidated by hypothetical data-dependent stopping, whereas an e-process value remains valid at any stopping time and permits continued sampling.Source: §7.10 · p. 123

7.11 A pathological example for the e-power of e-processes

• E-power can mischaracterize sequential behavior: processes may have positive infinite e-power yet vanish almost surely, or negative infinite e-power yet diverge almost surely.One intuitively powerful process diverges in probability under Q even though its e-power is negative infinity, motivating caution when e-power calculations involve infinity.Source: §7.11 · pp. 123–124

Bibliographical note

• The note traces the sequential probability ratio test to Wald [1945, 1947], its optimality to Wald and Wolfowitz [1948], and e-process definitions and their equivalence to Howard et al. [2020], Grünwald et al. [2024a], and Ramdas et al. [2020].It also identifies universal inference with Wasserman et al. [2020, Chapter 8] and the adaptive likelihood ratio with Tartakovsky et al. [2014, §5.4.2], and traces testing by betting to Ville [1939] and the optional stopping theorem to Doob [1953].Source: Bibliographical note · pp. 124–125

Handling multiple e-values

• The chapter focuses on combining e-variables for a common hypothesis, with arbitrary dependence handled here and independence and specific dependence structures treated separately.Multiple e-values can arise in multiple testing or in single testing when computed from different data parts or through data splitting.Source: Handling multiple e-values · p. 126, §8.1 · p. 126

8.1 Merging e-values under arbitrary dependence

• Under arbitrary dependence, the arithmetic mean essentially dominates every symmetric e-merging function; the admissible functions are exactly the Mλ family, and every e-merging function is dominated by one of them.Every symmetric positively homogeneous e-merging function is dominated by the arithmetic mean, but the arithmetic mean does not dominate every symmetric e-merging function.Source: §8.1 · pp. 127–128

8.2 Independent e-values and their products

• For independent e-variables, products and U-statistic functions and their convex mixtures are admissible ie-merging functions; these methods also remain valid under negative lower orthant dependence.The product of independent e-variables is itself an e-variable, and the product and U-statistic functions remain valid under the stated dependence condition.Source: §8.2 · pp. 129–130

8.3 Sequential e-values and martingale merging functions

• Martingale merging functions are exactly the admissible se-merging functions, while convex mixtures of U-statistic functions are admissible for sequential e-values.Sequential validity requires the inputs to be sequential in only one of the K! possible orders; martingale merging functions need not be increasing in all arguments, but satisfy sequential monotonicity.Source: §8.3 · pp. 131–132
• The empirically adaptive martingale merging function can serve as a default se-merging choice without prior information about the e-values.The martingale merging function is uniquely determined by its parameter γ.Source: §8.3 · p. 132

8.4 Mean-variance trade-off

• Under Q, ΠK maximizes all moments for powered e-variables, but under the null it has the largest variance among exact independent se-merging functions.The product function is therefore an extreme choice: maximizing expected value need not maximize e-power, which is measured by expected log value.Source: §8.4 · pp. 134–135
• In the example, the Q-log-optimal process grows asymptotically at rate (4/3)^k, while the all-in process tends to 0 Q-almost surely.The all-in strategy has the largest expectation, but its distribution has a large chance of reaching 0.Source: §8.4 · p. 135

Summary

• For iid e-values, the empirically adaptive martingale merging function is asymptotically log-optimal.Source: Summary · p. 135

8.5 Family-wise error rate and merging e-values

• For FWER control, merging e-values for each hypothesis intersection yields an e-collection; rejecting k when min_{A∋k} E_A ≥ 1/α controls the error rate.Source: §8.5 · p. 136

Bibliographical note

• The chapter draws chiefly on Vovk and Wang [2021, 2024a]; related results cover negative lower orthant dependence, e-value merging, FWER procedures, and connections to closed testing.Negative lower orthant dependence was introduced by Block et al. [1982]; merging results are in Chi et al. [2024], FWER procedures in Vovk and Wang [2021] and Hartog and Lei [2025], and closed-testing connections in Xu et al. [2025] and Goeman et al. [2025].Source: Bibliographical note · p. 137

False discovery rate control using compound e-values

• The chapter’s main setting concerns testing K hypotheses using e-variables or, in comparison, p-values, with the e-BH procedure as a central method.The chapter’s purpose is to show that e-values are useful for FDR control and central to FDR-controlling procedures.Source: False discovery rate control using… · p. 138, §9.1.1 · p. 138

9.1.1 Compound e-variables

• Compound e-variables permit arbitrary dependence, and conditionally valid follow-up e-variables for selected hypotheses can be combined while preserving compound validity.In the multi-armed bandit example, complicated serial and cross-dependence makes useful p-values difficult to obtain, while products of sequential e-variables provide e-values.Source: §9.1.1 · pp. 139–140

9.1.2 FDR and the e-BH procedure

• Any self-consistent e-testing procedure, including e-BH, controls FDR at level α for arbitrary dependence among compound e-variables.The post-hoc guarantee also holds; for ordinary e-variables, its upper bound improves to K0/K, allowing α to depend on the data.Source: §9.1.2 · p. 142

9.1.3 Boosting the e-values with distributional information

• Distributional information can boost e-values while retaining FDR validity by choosing factors that make truncated, rescaled e-values valid; without such information, the factor must be 1.The truncation preserves the e-BH rejection set, so validity of the transformed e-values justifies boosting.Source: §9.1.3 · pp. 144–145

9.1.4 Connection between the e-BH and the BH procedures

• BH is a special case of boosted e-BH, while BHY is a special case of e-BH; unlike BH under arbitrary dependence, e-BH needs no ℓK correction.BH controls FDR at level (K0/K)α under independent or PRDS p-variables, whereas arbitrary-dependence validity requires the ℓK correction.Source: §9.1.4 · pp. 145–146

9.2 Universality of compound e-values and e-BH

• All FDR-controlling procedures can be represented as e-BH applied to compound e-values, and every admissible FDR procedure is e-BH.The chapter also discusses implications for combining and derandomizing multiple-testing procedures.Source: §9.2 · p. 147
• Any FDR-controlling procedure can be recovered by e-BH using compound e-values, which can be chosen tight when the procedure is admissible.This holds even when the procedure is tuned to a particular fixed set of hypotheses.Source: §9.2 · pp. 147–148
• Convexly combining compound e-values from multiple FDR-controlling procedures and applying e-BH controls FDR at level α.The construction can also reduce sensitivity to the randomness of randomized procedures; full derandomization occurs when the procedures’ random variables are iid and their number tends to infinity.Source: §9.2 · p. 148

9.3.1 Background: Robbins’ compound decision theory

• Robbins’ compound decision theory identifies the optimal simple separable estimator under a fixed sequence with the Bayes estimator under the sequence’s empirical distribution.The resulting estimator is an oracle benchmark: it is simple separable and knows the true parameter vector, unlike feasible estimators.Source: §9.3.1 · p. 149

9.3.2 The log-optimal simple separable compound e-values

• The fundamental theorem of compound decisions yields log-optimal simple separable compound e-values, and every tight e-value of this form has representation (9.9).The objective is bounded by the sum of the separate problems’ KL divergences, so it can be worse than optimizing each test individually; pooling may help approximate the construction when nulls are unknown.Source: §9.3.2 · p. 151

9.4 A minimally adaptive e-BH procedure

• With F=M_K, the minimally adaptive procedure dominates base e-BH while controlling FDR at α, with practical interest mainly for K≤10.Its improvement is negligible for large K.Source: §9.4 · pp. 151–152
• For boosted e-values, using S instead of their potentially invalid arithmetic average gives a procedure that dominates e-BH and controls FDR at most α.The procedure applies S to the boosted e-values before returning discoveries from base e-BH.Source: §9.4 · p. 152

9.5 Stochastic rounding of e-values and randomized e-BH

• External randomization can stochastically round e-values onto a closed grid using the adjacent grid points around each input.Inputs outside the grid’s range, already on the grid, or with an infinite upper neighbor are left unchanged.Source: §9.5 · p. 153
• Stochastic rounding preserves expectation while trading off e-power for power, and rounding upward with positive probability can increase e-BH discoveries without inflating expectation.Rounded e-values have e-power no greater than the original and power at a grid level at least as large; randomization is useful because e-values often fall between grid points.Source: §9.5 · pp. 153–155
• Ge-BH controls FDR under arbitrary dependence and includes e-BH’s rejections; it makes extra discoveries with positive probability exactly under the stated condition on unrejected e-values.That condition is often met, including when e-values are independent and continuously distributed over [0, K/α].Source: §9.5 · p. 155
• De-BH and Ue-BH control FDR under arbitrary dependence and weakly dominate Ge-BH and e-BH, respectively; randomized BHY likewise preserves FDR control while enlarging discoveries.De-BH first rounds on Gα and then on a data-dependent grid; Ue-BH applies e-BH to e-values divided by an independent uniform random variable.Source: §9.5 · pp. 155–156

Numerical experiments.

• For K = 100 hypotheses, Figure 9.30 shows De-BH and Ue-BH power improvements over e-BH across various µ and ρ values, with power averaged over 500 trials.The simulations set π0 = 0.3 and α = 0.05.Source: Numerical experiments · p. 156, Figure 9.30 · p. 157

9.6 The closed e-BH procedure

• Closed e-BH is presented as a procedure that usually strictly dominates e-BH and is universal and unimprovable for FDR control.The section also sets out goals of improving e-BH, recovering any FDR or post-hoc FDR procedure, and extending the techniques beyond FDR.Source: §9.6 · pp. 156–157

9.6.1 Improving the e-BH procedure

• Closed e-BH can reject a superset of e-BH and minimally adaptive e-BH discoveries, and closed BHY usually strictly improves BHY while controlling FDR at α.For e-values (60, 39, 11) at α = 0.05, e-BH rejects one hypothesis, minimally adaptive e-BH rejects two, and closed e-BH rejects all three.Source: §9.6.1 · pp. 159–160

9.6.2 Recovering any FDR procedure

• Every FDR-controlling procedure can be represented as closed e-BH applied to an appropriate e-collection, and a closed e-BH version using compound e-values can make more discoveries than e-BH.The section also notes that every FDR procedure can be represented both by e-BH with compound e-values and by closed e-BH with an e-collection.Source: §9.6.2 · p. 160

9.6.3 Recovering or improving any post-hoc FDR procedure

• Closed e-BH can recover any post-hoc FDR procedure, and its α-independent e-collection has values at least as large as the preceding construction, typically strictly larger.This construction strictly improves e-BH at some level α and assigns no zero e-values because every index eventually enters Rβ.Source: §9.6.3 · pp. 160–161

9.6.4 A general e-closure principle for arbitrary error metrics

• E-closure applies to arbitrary error metrics, and for FWER control its mean e-collection recovers the e-Holm procedure.The general principle also permits choosing the loss function and level α post hoc.Source: §9.6.4 · pp. 161–162

9.6.5 Numerical experiments

• Across simulations, closed e-BH improved over e-BH and minimally adaptive e-BH while controlling FDR below α, and closed BHY substantially increased power over BHY.The comparisons used independent Gaussian means for e-BH and correlated Gaussian means for BHY.Source: Figure 9.37 · p. 162, Figure 9.38 · p. 162, §9.6.5 · p. 163

9.6.4 A general e-closure principle for arbitrary error metrics

• Applying e-closure to the e-collection defined by L_A(R)/α can recover or improve any procedure that guarantees the target error bound.The loss function and α can then be chosen post hoc, including switching from FWER control at 0.01 to FDR control at 0.05.Source: §9.6.4 · p. 162

9.6.5 Numerical experiments

• Although e-value procedures often had less power than p-value procedures, they offer stronger post-hoc type-I control, can improve p-value procedures, and depend heavily on the chosen e-values.All p-value-based procedures can be recovered using e-value-based procedures.Source: §9.6.5 · p. 163

Bibliographical notes

• Benjamini and Hochberg [1995] introduced the FDR metric and BH procedure for independent p-values, while Benjamini and Yekutieli [2001] extended the result to PRDS.Under arbitrary dependence, BHY control equals αℓK, the Benjamini–Yekutieli correction.Source: Bibliographical notes · p. 163
• Wang and Ramdas [2022] proposed the e-BH procedure.They also provide simple code for it.Source: Bibliographical notes · p. 163
• Wang and Ramdas [2022] prove Theorem 9.13 and study other e-value boosting methods, including methods under PRDS.Blanchard and Roquain [2008] developed the proof technique in Remark 9.17 without using e-value terminology.Source: Bibliographical notes · p. 163
• Compound e-values were first used without being given a name in Wang and Ramdas [2022, Theorem 3], and the term compound e-values first appeared in Ignatiadis et al. [2024b].Source: Bibliographical notes · p. 163
• Ren and Barber [2024] used the combination and derandomization recipe for the model-X knockoff filter, and Banerjee et al. [2023] applied it to meta-analysis.Li and Zhang [2024] discuss aggregating e-values from different procedures; the chapter unifies and extends these observations.Source: Bibliographical notes · p. 164
• Ignatiadis et al. [2024b], Xu and Ramdas [2023], Xu et al. [2025], and Goeman et al. [2025] proposed methods underlying Sections 9.4–9.6.Online multiple testing with e-values was studied by Xu and Ramdas [2024], and Fischer et al. [2024] proposed an online analog of BH and e-BH.Source: Bibliographical notes · p. 164

Approximate and asymptotic e-values

• The chapter formalizes approximate and asymptotic e-values, compound e-values, and asymptotic compound e-values, and presents their versatility for multiple testing alongside optimal constructions and examples.Source: Approximate and asymptotic e-values · p. 166

10.1 Approximate p-values and e-values

• Approximate p- and e-variables allow multiplicative slack ε and failure probability δ; δ is more lenient, and positive ε can be absorbed into a δ′ strictly below ε + δ.Controlling δ more strongly leads to two distinct notions of asymptotic e-values and p-values.Source: §10.1 · pp. 167, 169

10.2.1 Definitions and conditions

• Convergence in distribution to valid p- or e-variables suffices for asymptotic validity, while expectation lim sup ≤ 1 characterizes strongly asymptotic e-variables and uniform integrability links the notions.Source: §10.2.1 · pp. 171–172

10.3 An asymptotic e-value via the central limit theorem

• With unknown finite positive variance, studentized-mean constructions are strongly asymptotic e-variables; under stated alternatives, they diverge almost surely.Using an estimated standard deviation yields asymptotic e-variables, and mixing over any compactly supported distribution also gives a strongly asymptotic e-variable. The example addresses unknown variance, for which the authors know no nonconstant nonasymptotic e-variable.Source: §10.3 · pp. 173–174

10.4 Approximate/asymptotic compound p-values and e-values

• The multiple-testing framework extends the approximate and asymptotic p- and e-value setting to compound values.Source: §10.4 · p. 175

10.4.1 Compound p-values and e-values

• Compound p-variables and e-variables can be transformed in either direction through calibration.Source: §10.4.1 · p. 175

10.4.2 Approximate compound p/e-values

• Assuming each null distribution is atomless permits uniform randomization and supports approximate compound p- and e-value definitions used for later approximate and asymptotic FDR control.Source: §10.4.2 · p. 176
• Approximate compound p-variables or e-variables can be recovered from per-hypothesis approximate values, and a common p-to-e calibrator maps compound p-variables to compound e-variables.For approximate compound p-variables, the text notes that a corresponding representation is more complicated and uses an event-based definition.Source: §10.4.2 · pp. 177–179

10.4.3 Asymptotic compound p/e-values

• The asymptotic setup covers testing with data size n and hypothesis count K growing separately or together, and assumes atomless distributions for the approximate-e-variable representation.The chapter relates growing n to Gaussian approximations for Z- or t-tests and growing K to empirical-Bayes sharing of information across hypotheses.Source: §10.4.3 · p. 180
• e-BH applied to asymptotic compound e-values has asymptotic FDR control, and a procedure satisfying the stated tail inequality has (ε, δ/α)-approximate FDR control.The setup allows n, K, or both to grow; it uses triangular-array asymptotics indexed by m with max{K, n} → ∞.Source: §10.4.3 · pp. 180–181

10.4.4 The simultaneous t-test sequence model

• In the simultaneous t-test sequence model, a conservative estimate replaces a construction that depends on the unknown distribution, yielding asymptotic compound e-variables under stated growth and distributional conditions.The asymptotic index has n=n(m), K=K(m), and max{n, K} → ∞; when K → ∞, each distribution must be null for infinitely many groups, with moment restrictions or normality imposed on null-group marginals.Source: §10.4.4 · p. 182

Bibliographical note

• The bibliographical note attributes much of the chapter to Ignatiadis et al. [2024a] and traces related e-variable, compound p-variable, approximate FDR, and multiple-t-test constructions to prior work.It also attributes standard examples to Van Der Vaart [1998] and Lehmann and Romano [2005], and Example 10.12 to Wang and Ramdas [2023].Source: Bibliographical note · p. 184

11.1 Optimal transport duality

• Optimal transport duality is the technical tool for proofs in Sections 11.2 and 12.2, with dual functions inheriting upper-semicontinuity and stated monotonicity and boundedness properties.For upper-semicontinuous F, the infimum is attained; when F is decreasing and nonnegative, or bounded in [0, 1], the dual functions can be chosen with the corresponding properties.Source: §11.1 · pp. 185–186

11.2 Admissible e-merging functions

• Every e-merging function is dominated by a function M_λ from the characterized class; in one dimension, this class is bounded by (1 − λ) + λx.For a supplied e-variable E, the affine form (1 − λ) + λE dominates other ways of transforming E.Source: §11.2 · pp. 186, 189

11.3 Cross-merging: merging an e-value and a p-value

• The quotient combiner P/E, capped at 1, is admissible for independent p-value P and e-value E, and provides a basis for using e-values as unnormalized multiple-testing weights.With independent randomization U, U/E is a p-variable and satisfies U/E ≤ 1/E.Source: §11.3 · p. 190
• Fisher combination is preferable under a priori dataset exchangeability, while P/E can often outperform it when the primary, more powerful dataset supplies P and the secondary dataset supplies E.When exchangeable datasets also have similar power, Fisher’s combination typically has higher power than P/E.Source: §11.3 · p. 191
• PE is asymptotically as efficient as full-data PZ when α→0 or m/n→0, where Y is substantially more informative than X.Here PE is the quotient combiner PY/EX, and PZ is the p-value based on the full dataset.Source: §11.3 · p. 191
• Simulations show that for small m/n, PE nearly matches PZ and both outperform PF; when m=n, PF slightly exceeds PE, while both trail PZ.When combining one p-value and one e-value, the primary dataset should generate the p-value.Source: §11.3 · p. 191, Figure 11.8 · p. 192

11.4 E-values as weights in p-value-based multiple testing

• In the multiple-testing setup, each hypothesis may have a p-value and an e-value; a missing value is set to 1, representing absent follow-up or preliminary analysis.The described context uses e-values from preliminary analysis and p-values from follow-up confirmatory research.Source: §11.4 · p. 192
• ep-BH controls FDR when p-values P are PRDS and e-values E are independent of P, even though these conditions need not make the adjusted p-values PRDS.The argument conditions on E, applies weighted p-BH to the PRDS p-values, then integrates over the randomness in the weights.Source: §11.4 · pp. 192–193
• ep-BH uses E as unnormalized weights, preserving evidence from preliminary analyses, and requires no assumption about dependence within E.An e-value above 1 strengthens its p-value through Pk/Ek; an e-value below 1 weakens it.Source: §11.4 · pp. 192–193

Bibliographical note

• The bibliographical note credits optimal-transport results to Kellerer [1984], Rachev and Rüschendorf [2006], Rüschendorf [2013], Villani [2009], Vovk and Wang [2021], and Wang [2025].Sections 11.3–11.4 and e-weighted BH are based on or introduced by Ignatiadis et al. [2024b], while deterministic BH weights were studied by Benjamini and Hochberg [1997].Source: Bibliographical note · p. 193

12.1.1 Average p-values

• For simple or composite nulls, average p-values include p-variables and means of p-variables, form a convex class, and twice any average p-variable is p-valid, with factor 2 sharp.Source: §12.1.1 · p. 195

12.1.2 Special cases: mid p-values and posterior predictive p-values

• Mid p-values from discrete statistics and Bayesian posterior predictive p-values are average p-values; discrete mid p-values have mean 1/2 but need not be p-variables.Source: §12.1.2 · pp. 196–197

12.1.3 Calibration between average p-values and e-values

• Convex p-to-e calibrators map average p-variables to e-variables, while (2E)^-1 ∧ 1 maps any e-variable to an average p-variable.Composing this intermediate calibration with p → (2p) ∧ 1 yields the unique admissible e-to-p calibration e → e^-1 ∧ 1.Source: §12.1.3 · pp. 197, 199

12.1.4 Tests based on average p-values

• For arbitrary dependent p-values, thresholding their arithmetic mean with independent randomization gives level-α tests; a nonrandomized alternative is valid but weaker in power.Source: §12.1.4 · pp. 199–200

12.2 Merging p-values under arbitrary dependence

• Admissible p-value merging under arbitrary dependence must proceed through e-value merging, with optimal-transport duality turning probability constraints into expectation constraints.Source: §12.2 · p. 200

12.2.1 P-merging functions and examples

• Admissible p-merging functions are precise and lower semicontinuous, stable under pointwise limits, and dominate any p-merging function; natural symmetric families use order statistics or generalized means.Source: §12.2.1 · p. 201
• Simes is the minimum of symmetric p-merging functions, but is itself not a p-merging function; it produces a p-variable for independent or PRDS inputs, while the penalized Hommel function is precise.The Simes function is multiplied by the factor ℓ_K = ∑_{k=1}^K 1/k to obtain the Hommel function.Source: §12.2.1 · p. 202

12.2.2 Using e-values to merge p-values

• When the type-I error level is fixed in advance, a homogeneous p-merging function suffices, and admissible homogeneous functions obtain calibrators across levels by rescaling.For symmetric homogeneous functions, the calibrators and weights can be chosen identical across inputs.Source: §12.2.2 · pp. 203, 205
• Admissible homogeneous p-merging functions can be represented using calibrators and e-value combinations, although the calibrators inducing a function need not be unique.The direct conversion from a merged e-value to a p-value is generally not admissible; for convex calibrators, f ≤ K on (0, 1] is necessary and sufficient for precision and sufficient for admissibility under Theorem 12.17.Source: §12.2.2 · pp. 202, 206

12.2.3 Examples

• Twice the arithmetic mean is precise but not admissible, while replacing its non-calibrator with (2 − 2p)+ yields a p-merging function that dominates it.The untruncated function f(p) = 2 − 2p is not a calibrator because it is nonzero for p > 1.Source: §12.2.3 · p. 207

12.3 Combining exchangeable p-values

• For exchangeable p-values, online combination can yield a valid p-value whenever stopped without fixing the number of inputs in advance, provided the calibrator does not depend on K.Symmetric merging functions cannot improve on the factor 2 before the arithmetic average, even under exchangeability.Source: §12.3 · p. 208

12.4 Randomized combinations of p-values

• Randomized Markov inequality enables improved merging for arbitrarily dependent p-values, and choosing a uniform randomizer gives a smaller p-value than the deterministic rule.When an input is independent of the others, it can serve as the randomizer; exchangeable combination can also be applied after randomly permuting the inputs.Source: §12.4 · p. 209

13.1 Defining and constructing e-CIs

• Every e-CI is an ordinary CI, and every ordinary CI can be represented as an e-CI; therefore, level-free e-CI families are the more interesting object.The chapter notes that e-CIs can also be constructed by universal inference, stopped confidence sequences, and calibrated CIs.Source: §13.1 · p. 212

13.1.1 E-CIs from stopped confidence sequences

• An e-process for each hypothesized parameter yields a confidence sequence that remains valid at any stopping time.The construction follows from Ville’s inequality.Source: §13.1.1 · p. 212

13.1.2 E-CIs by calibrating CIs

• Calibrating the p-values implicit in an ordinary CI converts it to an e-CI, which contains the original CI.The construction uses the calibrator’s right inverse and relies on the CI being decreasing and continuous from below.Source: §13.1.2 · p. 213

13.2.1 The goal of FCR control

• FCR control requires choosing confidence levels for selected parameters so that expected false coverage stays at most δ, regardless of the selection rule.The selection may depend on the data, and the rule may be unknown when only the selected set is observed.Source: §13.2.1 · p. 214

13.2.2 The e-BY procedure

• The e-BY procedure guarantees FCR ≤δ under any dependence structure and any selection rule, and its FCR control implies e-BH FDR control.The guarantee applies to level-free e-CI families.Source: §13.2.2 · p. 215

13.3 Combining CIs via majority vote

• A black-box majority-vote method combines arbitrarily dependent uncertainty sets; weighted averaging, randomization, and exchangeability offer extensions.The method aims to retain nearly the input sets’ error guarantee, while randomization can produce strictly smaller merged sets without changing the coverage guarantee.Source: §13.3 · p. 215

13.3.1 The majority vote procedure

• 2α is the stated majority-vote error guarantee; for odd K, a worst-case dependence structure attains αK/⌈K/2⌉.Under independence, the error can be exponentially small in K; under exchangeability, a tighter set is described as retaining the same 1 −2α coverage guarantee.Source: §13.3.1 · p. 217
• At τ = 1/2, the majority-vote set’s measure is at most twice the average input-set measure, and for intervals it is never wider than the largest input interval.The measure bound is essentially tight; majority vote can also be longer than randomly selecting an interval, as in the nested-width example where lengths are 8 versus an average of 7.Source: §13.3.1 · p. 218

13.3.2 Exchangeable, randomized and weighted variants

• For arbitrarily dependent sets, random permutation gives Cπ coverage 1−2α with Cπ ⊆ CM; for exchangeable or randomly ordered confidence sets, running majority votes form a (1−2α) confidence sequence.The running construction can combine arriving sets on the fly.Source: §13.3.2 · pp. 219, 221
• For K ≥ 2 different sets satisfying property (13.1), CR has coverage at least 1−2α and is never larger than CM, while CU has coverage at least 1−α and is never smaller than CR.The weighted-majority set’s measure is at most twice the average measure obtained by randomly selecting an interval with probabilities proportional to w. If the αk levels are known, weights can achieve target level 1−α whenever α/2 lies in the convex hull of (α1, …, αK).Source: §13.3.2 · pp. 220–221

13.3.3 Median of Midpoints and Median of Median of Means

• For equal-width input intervals, the median-of-midpoints set contains the majority-vote set and is no wider than an input interval; repeated independent MoM runs form a MoMoM estimator with guarantees at data-dependent stopping K.The underlying median-of-means estimator has a sub-Gaussian tail bound despite finite-variance data, and its randomness comes from splitting observations into buckets.Source: §13.3.3 · pp. 221–223

13.3.4 Empirical examples

• In simulations, Cπ had empirical coverage 0.93 and was smaller than majority vote, CU had empirical coverage 0.96 near nominal 1−α, and union outputs occurred in fewer than 1% of runs.The figure compares intervals from different λ values with CM(x), CR(x), CU(x), and Cπ(x).Source: Figure 13.23 · p. 223, §13.3.4 · p. 224
• On Parkinson’s data, all merging procedures had good length and coverage; random forest yielded the smallest sets, while randomized voting had good coverage and slightly greater empirical length than the neural net.The percentage of outputs that were unions of intervals or empty sets was nearly zero for all methods.Source: §13.3.4 · p. 224, Table 13.23 · p. 224
• In MoMoM simulations, estimates stabilized after about 50 iterations for one t-distribution run and 25 for a skew-t run; average successive-estimate differences showed an inflection around 25 for both distributions.Plots of successive estimates can be tracked on the fly, with the statistical guarantee retained at the adaptively chosen stopping K.Source: §13.3.4 · pp. 224–226, Figure 13.24 · p. 225

Bibliographical note

• The note credits randomized and exchangeable majority-vote extensions to Gasparin and Ramdas [2024b] and weighted online conformal aggregation to Gasparin and Ramdas [2024a].It also cites e-confidence interval and confidence-sequence work, the e-BY procedure, and a computationally intensive derandomized median-of-means method; Zhang et al. [2024] prove Theorem 14.6 using optimal transport under joint non-atomicity.Source: Bibliographical note · pp. 226, 231

Further contrasts between e-values and p-values

• The chapter strengthens e-value foundations with an e-value/p-value duality, conditional-expectation and probability representations, and existence criteria for powered exact values.Source: Further contrasts between e-values and… · p. 227

14.1 A duality between e-values and p-values

• For increasing e-variables and decreasing p-variables, Theorem 14.2 identifies the reciprocal of the best monotone p-variable as the supremum of monotone e-variables.In the simple-null case, this supremum is 1/S(X; P), while the best decreasing p-variable is S(X; P).Source: §14.1 · p. 228

14.2 Representations as conditional expectations and probabilities

• E-variables built from data are conditional expectations of likelihood ratios, while p-variables are conditional probabilities on the data.The likelihood-ratio representation generalizes the simple-null connection and is optimal for testing a simple null against an alternative.Source: §14.2 · p. 230

14.3 Existence of powered exact e-values and p-values

• Under condition (U) for finite hypotheses, powered exact p-values and e-values exist exactly when the alternative convex hull and null span are disjoint.The equivalent conditions include a pivotal exact p-variable and a pivotal, exact, bounded e-variable with positive e-power.Source: §14.3 · p. 231

Improved e-value thresholds under additional conditions

• Markov's bound P(E ≥1/α) ≤α cannot be improved for all e-variables, but shape or distributional restrictions can improve the strict-tail bound P(E >1/α).The section considers a simple null; for composite nulls, the corresponding shape information must hold for every null distribution.Source: Improved e-value thresholds under… · p. 232

15.1 Conditional e-to-p calibrators on a subset of e-values

• For a restricted class of e-variables, Rγ yields the smallest conditional e-to-p calibrator, which can improve on x 7→(1/x) ∧1 and applies to mixtures.These calibrators can convert class-specific e-values for procedures that take p-values as input, including Benjamini–Hochberg.Source: §15.1 · pp. 233–234

15.2 Comonotonic e-variables

• For comonotonic e-variables, the supremum's threshold test controls level α even when the supremum is not itself an e-variable.For comonotonic likelihood-ratio families, the supremum can have higher power than a mixture; in the normal-mean example it also retains the type-I error guarantee.Source: §15.2 · p. 235

15.3 Density conditions

• Decreasing-density e-variables have threshold Tα(ED)=1/(2α); for ED>1, Tα(ED>1)=1/(2α)+α/2, a near-twofold boost at small α.The density condition applies to the e-variables themselves; exponential and Pareto distributions are examples.Source: §15.3 · pp. 236–237

15.4 Conditions on log-transformed e-variables

• With P(E=0)=0, log-symmetric and log-unimodal classes cannot improve Markov’s bound for γ≤1/2, while specified decreasing-log-density and unimodal-symmetric classes allow roughly 1/e gains at small γ.The discussion notes that products of e-variables are common and that distributional properties of log E may be reasonable in that setting.Source: §15.4 · pp. 238–239

Bibliographical note

• The chapter largely follows Blier-Wong and Wang [2024], and cites work on probability bounds, dependence, and decision theory as related background.It notes improved thresholds for combined e-values and e-BH in Blier-Wong and Wang [2024], and identifies comonotonicity as relevant to dependence modeling and ambiguity.Source: Bibliographical note · p. 240

E-values and risk measures

• Risk measures are mappings from random variables or distributions to real values, encompassing statistical functions such as means, variances, and quantiles.The chapter focuses on distributions on R and treats these mappings as risk measures.Source: E-values and risk measures · p. 241

16.1 Risk measures and statistical functions

• Risk measures can map either random variables or distributions to extended real values; the chapter focuses on distributions on R and notes ES is coherent whereas VaR is not.It also gives an alternative proof that twice the average of arbitrarily dependent p-variables is a p-variable, using the convexity of ES.Source: §16.1 · pp. 241–242

16.2 Connecting e-values to coherent risk measures

• An e-variable for a null hypothesis is characterized by a coherent risk-measure constraint, and ES constraints correspond to nulls with likelihood ratios bounded by 1/(1 −β).For composite alternatives, maximizing relative e-power can learn the true data-generating distribution, unlike mixture- or worst-case e-power; the resulting problems are convex programs and are efficiently solvable in many cases.Source: §16.2 · pp. 243–244

16.3 E-values for testing risk measures

• The chapter develops e-values for testing the value of a given law-invariant risk measure.It uses the formulation mapping distributions to real values.Source: §16.3 · p. 244

16.3.1 E-statistics

• Sequential e-variables from risk forecasts can be combined into an e-process to test risk-measure hypotheses, even for dependent, non-iid observations.A backtest e-statistic has power when risk is underestimated regardless of whether an auxiliary risk forecast is truthful; monotonicity rewards reporting a higher risk value.Source: §16.3.1 · pp. 245–246

16.3.2 Mean, variance, quantiles, and expected losses

• For the mean, e(x, r) = x/r is a monotone backtest e-statistic, while for (variance, mean), e(x, r, z) = (x −z)2/r provides one using the mean as auxiliary information.For an expected loss bounded below by a, the proposed statistic is e(x, r) = (ℓ(x)−a)/(r −a).Source: §16.3.2 · p. 247
• For fixed ρ or (ρ, ϕ), linear combinations of a backtest e-statistic and 1 with weights between 0 and 1 are also valid and may suit different situations.Source: §16.3.2 · p. 247

16.3.3 E-statistics for testing ES

• On M1, the displayed construction defines a backtest e-statistic for (ESβ, VaRβ); the r < z case is irrelevant because ESβ(F) ≥ VaRβ(F).Source: §16.3.3 · p. 248
• The ES/VaR statistic is an e-variable for a larger null, but not for the joint null ESβ(X) ≤ r and VaRβ(X) ≤ z; such statistics are essentially the only choices up to transforms.Source: §16.3.3 · p. 248

Bibliographical note

• Coherent risk measures were introduced by Artzner et al. [1999], with representations developed by Delbaen [2002, 2012] and Jouini et al. [2006].Rockafellar and Uryasev [2002, Theorem 10] give formulas (16.1)–(16.2), calling ES CVaR; McNeil et al. [2015, Theorem 8.14] gives its dual representation, and Embrechts and Wang [2015] provide seven proofs of ES subadditivity.Source: Bibliographical note · pp. 248–249
• Super-expectations have a long history, appearing in Huber [1981, Chapter 10] and in decision theory, imprecise probabilities, nonlinear expectations, and game-theoretic probability.The cited works include Schmeidler [1989], Gilboa and Schmeidler [1989], Walley [1991], Peng [2019], and Shafer and Vovk [2001, 2019].Source: Bibliographical note · p. 249
• Section 16.3 is largely based on Wang et al. [2025], who showed that most backtest e-statistics there are essentially the only possible choices in a suitable sense.They also studied financial-data backtesting; related work covers forecast comparison [Henzi and Ziegel 2022], sequential quantile testing [Howard and Ramdas 2022], Bayes pairs [Embrechts et al. 2021], joint ES/VaR elicitability [Fissler and Ziegel 2016], and mean/variance testing [Fan et al. 2025].Source: Bibliographical note · p. 249

A.1 Atomless probability spaces

• A statistical model is rich when one random variable is uniform on [0, 1] under every model measure; adjoining an independent uniform component makes any model rich.For a model with a single probability measure, richness is equivalent to atomlessness.Source: §A.1 · p. 250
• The richness assumption is essential: with all point masses as model measures, maxima of e-variables remain e-variables, but the maximum function is not a valid e-merging function.Source: §A.1 · p. 251
• For e-merging functions, it suffices to check validity on one fixed atomless probability space; analogous statements apply to ie- and se-merging functions.The appendix also notes analogous results for calibrators, e-to-p calibrators, and combiners.Source: §A.1 · pp. 251–252
• On an atomless space, for any random variable X and p in (0, 1), an event A of probability p can be chosen between {X > x} and {X ≥ x}, with x the left (1−p)-quantile.A uniform random variable lets the construction split the mass at x to attain probability p.Source: §A.1 · p. 252

A.2 Quantile functions

• The left quantile is the infimum of x with P(X ≤ x) ≥ α for α in (0, 1], while the right quantile uses P(X ≤ x) > α for α in [0, 1).Source: §A.2 · p. 252
• The appendix gives translation rules between probability and quantile statements and states that the two quantile functions are almost everywhere equal.Source: §A.2 · pp. 252–253
• When the quantile level is uniform on [0, 1], random-level quantiles are interpreted pointwise; undefined endpoint values do not matter, and Lemma A.9 implies a well-known formula.Source: §A.2 · pp. 253–254
• For continuous X, U = F(X) provides the uniform variable used in the quantile representation; on atomless spaces, uniforms on the atoms handle discontinuous distributions when no independent uniform exists.The appendix presents the result in a different form from Ryff [1965, Theorem 1].Source: §A.2 · pp. 254–255