Source-linked AI summary
Anytime-valid simulation-based hypothesis testing
Patrick Forré, Lydia Brenner
TL;DR
When analytic densities are unavailable, simulation-based hypothesis testing must also address the fixed-sample limits of classical testing. The paper constructs a sequential e-test martingale from simulations, with anytime-valid type-I error guarantees and approximate growth optimality. It reports geometrically decaying type-II error bounds and asymptotic power one, while identifying broader hypothesis classes and dependent data as future extensions.
Problem
Analytic likelihoods may be unavailable in simulation-based testing, while classical p-value procedures generally require a sample size fixed in advance and do not allow optional stopping.
Source: §1 · p. 2Method
Results
The test has anytime-valid type-I error guarantees, approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one.
Source: §5 · p. 13Takeaways & Limitations
The construction provides a sequential testing procedure for simple hypotheses when only simulations from the competing distributions are available.
Source: §5 · p. 13The discussion identifies composite hypothesis classes, dependent data streams, and more realistic growth-optimality results for non-compact spaces as areas for future work.
Source: §5 · p. 14
Abstract
from arXiv · showhide
For a given data distribution $(X_t)_{t \in \mathbb{N}} \sim Q$ i.i.d., we investigate the hypothesis testing problem: $H_0: Q = P_0$ vs. $H_1: Q = P_1$, for two different model probability distributions $P_0$ and $P_1$. In contrast to the standard setting, where analytic densities $p_0$ and $p_1$ are given, here, we consider the density-free setting, where we only have access to i.i.d. simulations $(Z^0_t)_{t \in \mathbb{N}} \sim P_0$ and $(Z^1_t)_{t \in \mathbb{N}} \sim P_1$. For this simulation-based hypothesis testing setting, we construct an e-test martingale, resulting in a sequential test with anytime-valid type-I error guarantees, approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one. Most ingredients used in our constructions are variants of well known concepts. The value of this paper lies in the compact presentation of an effective, anytime-valid solution for the density-free simulation-based sequential hypothesis testing case.
1 Introduction
The paper addresses simulation-based hypothesis testing when analytic likelihoods are unavailable, alongside the fixed-sample limits of classical p-value testing. It proposes an anytime-valid sequential test for this density-free setting.
1 Introduction
- In realistic LHC analyses, detector-level likelihoods are often unavailable in closed form or computationally intractable, so simulations provide samples under competing hypotheses.The passage connects this difficulty to the multiple stages of particle production and detector response.Source: §1 · p. 2
- Classical p-value tests generally require a sample size fixed in advance, disallowing optional stopping or continuation.The cited passage also notes challenges aggregating evidence across experiments or conducting multiple tests.Source: §1 · p. 2
- The proposed sequential test combines anytime-valid type-I error guarantees with approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one.The test statistics also implicitly approximate the likelihood ratio.Source: §1 · p. 2
- The paper studies simple hypotheses H0: Q = P0 versus H1: Q = P1 using simulated samples rather than analytic densities.The introduction places this problem in the context of experimental particle physics, including searches and precision measurements at the LHC.Source: §1 · pp. 1–2
2 Anytime-valid hypothesis testing with conditional e-variables
The section develops sequential testing from conditional e-variables, multiplying them into a supermartingale. It establishes anytime-valid type-I control and, under additional conditions, geometrically decaying type-II error and asymptotic power one.
2 Anytime-valid hypothesis testing with conditional e-variables
- The section defines e-power as expected log evidence under H1 and relates e-variables to p-values through the Markov inequality.An e-variable directly supplies a rejection rule for every confidence level α.Source: §2 · p. 3
- For simple hypotheses with P1 absolutely continuous with respect to P0, the likelihood ratio is the e-variable with maximal e-power, equal to the KL divergence.The paper notes that this ratio is not analytically available in its setting, motivating e-variables based only on simulations.Source: §2 · p. 3
- Multiplying conditional e-variables constructed from past information yields a nonnegative supermartingale that is an e-variable under H0.The resulting process Wt is the basis for the section’s sequential decision rule.Source: §2 · p. 4
- Ville’s inequality gives the sequential test an anytime-valid type-I error bound.The test rejects H0 when Wt exceeds 1/α.Source: §2 · p. 4
- With bounded logarithms and sufficiently large conditional e-power, the test has geometrically decaying type-II error bounds and asymptotic power one.When the bounds are constant and the conditional e-power lower bound is positive, the type-II bound converges to zero geometrically.Source: §2 · p. 4
- Clipping an e-variable whose logarithm may be unbounded can retain positive e-power for sufficiently small γ and yield the geometric type-II bounds.The clipped version remains an e-variable for H0.Source: §2 · p. 5
3 Construction of e-variables in the density-free, simulation-based setting
This section develops density-free, simulation-based e-variables for sequential testing, including their selection, learning, and adaptation. It covers optimality criteria, choices of K and training objective, and inverse-temperature tuning for a known finite horizon, with a past-dependent inverse-temperature sequence retaining the anytime type-I error guarantee.
3 Construction of e-variables in the density-free, simulation-based setting
- Using only simulations, the construction turns a nonnegative function h into a valid e-variable by normalizing against the function values on K null samples and multiplying by K + 1.This symmetrization addresses the fact that replacing an unavailable null expectation with a finite-sample average need not itself yield a valid e-variable.Source: §3 · p. 6
- For fixed K, e-power is maximized when h is proportional to the likelihood ratio dP1/dP0; the resulting e-variable approaches the overall most powerful e-variable as K grows.The proposition also gives a Jensen inequality for its e-power and uniform convergence when the likelihood ratio is bounded.Source: §3 · p. 7
- With a parameterized hθ = exp(gθ), empirical e-power maximization estimates the optimal e-variable from i.i.d. simulations under P0 and P1.Theorem 3.4 establishes uniform strong convergence of the objective and near-maximizer e-power under its stated assumptions; approximate growth optimality includes approximation and finite-K penalties.Source: §3 · pp. 7–8
- If the approximation and optimization errors are sufficiently small, some K and training-sample size N yield positive e-power for H1 with high probability.For deep neural networks, the stated conditions are argued to hold on compact sample spaces; the section notes extension to σ-compact spaces by applying the theorem on compact subsets.Source: §3 · p. 9
- Applying the learned conditional e-variable at each time step recursively defines an e-test martingale, and the test rejects when its wealth crosses 1/α.Clipping the learned e-variable can prevent values near zero from quickly destroying the wealth process and restores explicit type-II error bounds.Source: §3 · pp. 9–10
Remark 3.8
- The choice of K is constrained by a deterministic upper bound on e-power, which should be close to or exceed KL(P1∥P0) to allow approximate attainment of that limit.The resulting K condition can be tracked and adjusted during training.Source: §3 · p. 11
- A second constraint on K comes from the finite-K penalty in the lower bound for the optimized limit e-variable; a criterion can make that penalty smaller than a chosen δ.This quantity can also be checked and adjusted during training.Source: §3 · p. 11
Remark 3.9
- The alternative NWJ variational objective for KL-divergence has a unique density-ratio log maximizer and is known to yield lower bias but higher variance; its trained estimator can be used in the e-variable with a chosen K [NWJ10, BBR+18, POvdO+19].The empirical objective is optimized using samples from P1 and P0.Source: §3 · p. 12
Remark 3.10
- For a known finite horizon T, an inverse-temperature parameter β modifies the long-run growth-oriented e-variable to minimize type-II error at time T [For26a].The optimization targets rejection at the future time T given the observations and evidence accumulated by the current time.Source: §3 · p. 12
- Repeating the procedure at each time step yields an adaptive inverse-temperature sequence that can be combined with clipping regularization, while anytime type-I error guarantees still hold when each temperature depends only on its own past.Source: §3 · p. 13
4 An alternative testing scenario
For the alternative H0: Q = P0 versus H1: Q ≠ P0, the method adapts training to past observations from Q while retaining anytime type-I error guarantees. The trained e-variable then targets dQ/dP0, which is more optimal if H1 is true.
4 An alternative testing scenario
- The test considers H0: Q = P0 against the broader alternative H1: Q ≠ P0.Source: §4 · p. 13
- Training uses observations through time t−1 and simulations from P0, explicitly excluding the current and future observations.The method may first pre-train using an alternative P1, if possible.Source: §4 · p. 13
- The resulting procedure remains a valid conditional e-variable, so its anytime type-I error guarantees still hold.Source: §4 · p. 13
- In this scenario, the trained e-variable learns to approximate dQ/dP0 rather than dP1/dP0.Source: §4 · p. 13
5 Discussion
The paper presents an e-test martingale for density-free simulation-based sequential testing and identifies broader hypotheses, dependent streams, and more practical growth-optimality conditions as future extensions.
5 Discussion
- The constructed e-test martingale gives anytime-valid type-I error guarantees, approximate growth optimality, geometrically decaying type-II error bounds, and asymptotic power one.Source: §5 · p. 13
- Future work could extend the setting from simple to composite hypotheses and from i.i.d. observations to filtration-adapted dependent streams.Source: §5 · p. 14
- A more realistic approximate growth-optimality theorem could use more practical norms, non-compact spaces, and function classes.Source: §5 · p. 14
A Theorems
The appendix states Ville’s inequality for non-negative supermartingales and the Azuma–Hoeffding inequality under stated conditions on an adapted sequence.
A Theorems
- Ville’s inequality is stated for a non-negative supermartingale adapted to a filtration under P0.Source: A Theorems · p. 18
- The Azuma–Hoeffding theorem assumes an adapted sequence of random variables and constants satisfying stated conditions at each time.Source: A Theorems · p. 18
- The theorem then gives bounds for each real ℓ and time t, including a special case with uniformly bounded ranges and lower-bounded constants.Source: A Theorems · p. 18
B Proofs
The proofs use product-distribution comparisons and uniform convergence arguments, then connect optimization claims to divergence properties under absolute continuity.
B Proofs
- The proof compares distributions formed by placing P1 in one coordinate and P0 in the remaining coordinates.Source: B Proofs · p. 18
- The minimization argument uses that KL is a proper divergence and expresses the relevant quantity through densities under absolute continuity.Source: B Proofs · p. 19
- The proof of the equivalence uses ratios, integration over P0, and a resulting almost-everywhere condition.Source: B Proofs · p. 19
- Continuity and integrable-envelope assumptions support a uniform law of large numbers over the parameter space.The proof notes that compactness and continuity make the objective uniformly continuous.Source: B Proofs · pp. 20, 22
- The proof also invokes an approximation assumption and, when χ2(P1∥P0) is finite, a Jensen inequality.Source: B Proofs · p. 24