Source-linked AI summary
AdaPT: An interactive procedure for multiple testing with side information
Lihua Lei, William Fithian
TL;DR
Large-scale multiple testing can use contextual predictors to focus discoveries, but adaptive selection must retain error control. AdaPT addresses this with an iterative, partially masked thresholding procedure that allows arbitrary analyst-chosen models while controlling FDR in finite samples. Across the paper’s experiments, it is reported to produce consistent power improvements, while its finite-sample guarantee relies on assumptions that do not cover dependent p-values.
Problem
Multiple testing with contextual predictors seeks to focus power on more promising hypotheses while retaining false discovery rate control.
Method
AdaPT iteratively proposes rejection thresholds, reveals partially masked p-values, estimates FDP, and permits arbitrary model updates as information accrues.
Results
AdaPT can give consistently significant power improvements over current state-of-the-art methods while controlling FDR at level α in finite samples under its stated assumptions.
Takeaways & Limitations
Analysts can adapt threshold estimates interactively, including switching models and consulting intuition, without compromising the advertised FDR guarantee under the procedure’s assumptions.
Takeaways & Limitations
Without modification, AdaPT does not control FDR in finite samples for dependent p-values, including settings with local random effects shared by nearby hypotheses.
Abstract
from arXiv · showhide
We consider the problem of multiple hypothesis testing with generic side information: for each hypothesis $H_i$ we observe both a p-value $p_i$ and some predictor $x_i$ encoding contextual information about the hypothesis. For large-scale problems, adaptively focusing power on the more promising hypotheses (those more likely to yield discoveries) can lead to much more powerful multiple testing procedures. We propose a general iterative framework for this problem, called the Adaptive p-value Thresholding (AdaPT) procedure, which adaptively estimates a Bayes-optimal p-value rejection threshold and controls the false discovery rate (FDR) in finite samples. At each iteration of the procedure, the analyst proposes a rejection threshold and observes partially censored p-values, estimates the false discovery proportion (FDP) below the threshold, and either stops to reject or proposes another threshold, until the estimated FDP is below $α$. Our procedure is adaptive in an unusually strong sense, permitting the analyst to use any statistical or machine learning method she chooses to estimate the optimal threshold, and to switch between different models at each iteration as information accrues. We demonstrate the favorable performance of AdaPT by comparing it to state-of-the-art methods in five real applications and two simulation studies.
1. Introduction
AdaPT addresses large-scale multiple testing with contextual side information by letting analysts adapt rejection thresholds interactively while preserving finite-sample FDR control. Its masking protocol supports flexible threshold learning and can improve power over existing approaches.
- Motivation: Modern applications require methods that support exploratory analysis while maintaining statistical guarantees for reported conclusions.Selective inference provides error control, but many existing methods require the selection algorithm to be specified in advance.
- Motivation: AdaPT uses generic side information to focus power on hypotheses that are more promising, extending beyond exchangeable-null settings where BH is nearly optimal.Side information can include prior knowledge, group structure, ordering, sample size, or overall signal.
- AdaPT framework: At each iteration, the analyst proposes a rejection threshold, observes partially masked p-values, estimates FDP, and either stops or chooses a more stringent threshold.The estimator compares rejections below the threshold with p-values in the mirrored upper region.
- AdaPT framework: AdaPT permits arbitrary threshold-selection methods, including changing models as information accrues, provided the interaction protocol is followed.The analyst need not specify or fully understand the update rule, and overfitting does not compromise finite-sample FDR control, though it can reduce power.
- AdaPT framework: Partial masking typically preserves the ability to learn the optimal threshold surface because masked-data local-FDR estimates converge toward full-data estimates before stopping.The vast majority of p-values are revealed near stopping, while many remaining masked values are evidently small or large.
2. The AdaPT procedure
The AdaPT procedure operates through a filtration that progressively reveals p-value information while constraining threshold updates. Under independent uniform or mirror-conservative null p-values, an optional-stopping argument yields finite-sample FDR control.
- Notation and assumptions: The procedure assumes null p-values are mutually independent, independent of non-null p-values, and either uniform or mirror-conservative.Mirror-conservatism compares probability mass in lower and reflected upper intervals, and is distinct from ordinary conservatism.
- FDR control: The algorithm stops at the first step where the estimated FDP is at most α and rejects hypotheses with p_i ≤ s_t(x_i).The stopping time is almost surely finite under the assumption that each iteration reveals at least one censored p-value.
- FDR control: The proof uses conditional independence, mirror-conservative reflected indicators, and an optional-stopping lemma to control the expected ratio of null lower-tail counts to upper-tail counts.This extends the optional-stopping strategy used in earlier FDR procedures to interactively chosen thresholds.
- FDR control: Under the stated assumptions, AdaPT controls FDR at level α both conditional on the initial masking information and marginally.The result holds in finite samples and does not require the analyst’s threshold-update rule to be fixed in advance.
3. A Guideline To Choose Thresholding Rules
Under a conditional two-groups model, Bayes-optimal rejection thresholds are level surfaces of the local false discovery rate, while conservative estimation is needed because null and alternative components are not separately identifiable.
- The two-groups model: The conditional two-groups model represents p-values using predictor-dependent null and alternative densities, with uniform nulls and alternatives favoring smaller p-values.The model assumes independent (x_i, H_i, p_i), f0(p | x) ≡ 1, and non-increasing f1(p | x).
- The two-groups model: The local false discovery rate is the posterior probability that a hypothesis is null given its predictor and p-value.It is defined as fdr(p | x) = P(H_i is null | x_i = x, p_i = p).
- The two-groups model: Because null status is unobserved, the mixture density is identifiable but π1 and f1 are not separately identifiable.Different choices of π1 and f1 can produce exactly the same mixture density.
- The two-groups model: A conservative identification assumption sets 1 − π1(x) to the minimum conditional mixture density, yielding a conservative local-FDR estimate from any estimated mixture density.The approximation is especially good when fdr(1 | x) ≈ 1.
- Optimal thresholds: The paper formulates threshold selection as a Neyman–Pearson problem using global FDR and power, rather than the Bayesian-type loss used by Sun et al.Global FDR is the null probability among rejected hypotheses, while power is the rejection probability conditional on a non-null hypothesis.
- Optimal thresholds: Under the stated regularity conditions, every optimal thresholding rule is a level surface of the local FDR.The result follows from the KKT condition and is formalized by Theorem 2.
4. Implementation
AdaPT estimates conditional mixture components from partially observed p-values and uses the resulting local-FDR surfaces to update thresholds. Its implementation supports flexible model families, feature selection, and gradual threshold changes while retaining FDR control despite modeling choices.
- Modeling local FDR: The implementation estimates conditional density and local FDR flexibly when domain-specific modeling expertise is unavailable.The authors nevertheless recommend incorporating domain knowledge when possible.
- Modeling local FDR: A parametric model assumes uniform null p-values and models non-null p-values with an exponential-family density, yielding a conditional beta-mixture model in one example.The beta-mixture formulation is one instance of the broader exponential-family framework.
- EM estimation: The EM algorithm maximizes a partially observed likelihood using the current threshold and masked p-values, alternating expectation and maximization steps.The M-step separates into optimization problems corresponding to logistic regression with fractional responses and weights.
- Flexible fitting: The generalized linear-model framework can be extended with penalized GLMs, generalized additive models, boosting, random forests, or neural networks.The paper also allows direct modeling of π1(x) and the conditional non-null mean with nonparametric methods.
- Feature selection: Feature sets may be selected using BIC or cross-validation, either repeatedly during AdaPT or once initially when repeated selection is computationally costly.The candidate feature sets can use spline bases or subsets of covariates.
- Threshold updates: Threshold updates approximate local-FDR level surfaces and are constrained to remain no larger than the current threshold.The implementation can choose a local-FDR level that reveals exactly one partially masked p-value, while larger levels typically produce higher estimated FDP.
5. Experiments
Experiments show that AdaPT uses informative orderings and covariates to improve discovery power while retaining strong data adaptivity and low information loss from partial masking. Across simulations and real datasets, it generally outperforms competing procedures, although gains vary by application.
- Gene/drug response data: AdaPT significantly outperforms all other methods across target FDR levels under weak and strong orderings.With random ordering, rejections drop dramatically and power is nearly the same as the non-adaptive Barber–Candès method, providing evidence against overfitting.
- Gene/drug response data: Most discoveries arise among the first 5000 genes under both moderately and highly informative orderings.The highly informative ordering sorts p-values better, while AdaPT learns a nearly monotone threshold rule for it.
- Gene/drug response data: Information loss from partial masking is almost negligible once the target FDR falls below 0.2.At the stopping step, the correlation between local-FDR estimates from unmasked and partially masked p-values is almost 1.
- Simulation studies: Estimated local FDR identifies non-nulls accurately even when most p-values are partially masked and becomes stable in practical target-FDR regimes.The fitted estimates become especially informative after reaching practical α values.
- Other applications: Across additional real datasets, AdaPT retains competitive performance and can learn monotone local-FDR structure, while outperforming other methods in the proteomics example.On the yeast-protein dataset, its performance is similar to Barber–Candès and Storey’s BH but better than the remaining methods.
6. Discussion
AdaPT combines partially masked p-values, adaptive thresholding, and optional-stopping arguments to support interactive multiple testing with side information. The discussion also describes extensions, connections to knockoffs, and the distinction between global FDR control and local FDR estimates.
- AdaPT framework: AdaPT estimates increasingly stringent, covariate-dependent rejection thresholds from partially masked p-values while monitoring an FDP estimator.Thresholds are updated as more p-values are unmasked and information accrues.
- AdaPT framework: Analysts may use arbitrary local-FDR estimation methods, change models during iterations, and overfit without compromising finite-sample FDR control.Overfitting can nevertheless reduce power.
- Generalization: AdaPT can remove threshold curves entirely by updating candidate rejection sets directly, while retaining the same FDP estimator and FDR-control proof.Algorithm 1 is a special case of this more general formulation.
- Limitations and extensions: For dependent p-values, the procedure does not generally control FDR in finite samples because shared local effects can be mistaken for signal.The authors identify relaxing independence as an open direction.
- Connection to knockoffs: AdaPT relates to Knockoff+ through exchangeability, but it permits iterative interaction and introduces no extra randomness.AdaPT uses the mirror p-value 1−p_i rather than an independent knockoff p-value.
- Local FDR: AdaPT returns global FDR-controlled rejections and can also provide local FDR estimates for individual rejected hypotheses.Those local estimates require stronger model assumptions than the global FDR guarantee.
A.1. Derivation of E-step
This section derives the E-step quantities used with revealed and masked p-values. It distinguishes singleton observations from two-element masked sets and uses conditional probabilities for the latent hypothesis state.
- E-step inputs: Revealed p-values are represented as singleton observations, whereas masked p-values are represented as two-element sets.The distinction defines the inputs used in the E-step derivation.
- Conditional probability: The E-step computes the conditional probability that a hypothesis is non-null given its partially observed p-value.The displayed expression combines the non-null prior with the corresponding p-value density and the null component.
- Conditional probability: Bayes’ formula gives the conditional distribution of the latent hypothesis indicator given the masked p-value.Subsequent results follow from these conditional distributions.
A.1.1. Beta-mixture model
The beta-mixture model is inserted into the paper’s general E-step results to obtain the corresponding model-specific expressions.
- Model substitution: The beta-mixture model supplies a concrete distributional specification for the p-values in the general framework.The section then plugs this model into the general results.
A.1.2. Gaussian-mixture model
The Gaussian-mixture section specializes the framework to p-values derived from one-sided z-tests and describes initialization of the mixture parameters from partially masked data.
- Gaussian transformation: For one-sided z-tests, the section uses the transformation g(p)=Φ^-1(1−p) to connect p-values with a Gaussian-mixture model.This transformation is identified as the natural choice for the stated test setting.
- Initialization: Initialization requires estimates of π_1i and μ_i because no information is available at step 0.The procedure begins with imputed or regression-based guesses.
- Initialization: A logistic regression on the maskedness indicators provides an initial estimate of π_1i, which is truncated to the interval [0,1].The construction uses covariates φ_π(x_i).
- Initialization: The initialization favors conservative estimates of π_1i because anti-conservative estimates might cause over-fitting.This preference is stated as a reason for the truncation and estimation choice.
- Imputation: For masked p-values, the procedure imputes values using the observed two-element set before fitting an unweighted generalized linear model.The imputation distinguishes null and non-null hypotheses through their relative likelihoods.
A.3. Other issues
The weighted GLM M-step can be unstable when null p-values are super-uniform or only asymptotically uniform, so the procedure uses an unweighted step instead.
- Super-uniform or asymptotically uniform null p-values can make E-step-derived weights produce abnormal estimates for µ(x).The weighted GLM relies on uniformly distributed null p-values on [0, 1].
- Replacing the weighted M-step with an unweighted step gives consistently good performance across the reported experiments.
B. Technical Proofs
The technical proofs establish conditional independence and exchangeability properties, then use filtrations, convexity, and supermartingale arguments to support the theoretical results.
- The objective and constraint functions used in the proof are convex in s, supporting the optimization argument.
- Slater’s condition is established by constructing a point s̄ and showing g(s)<0 throughout a neighborhood for sufficiently small ϵ.
- The constructed variables ˜b_i are conditionally independent given G−1 and have the same joint distribution as (b_1, . . . , b_n).
- The proof introduces intermediate fractional steps so C_t decreases by at most one element at each step.
- Exchangeability of the Bernoulli variables over configurations with fixed totals is used to analyze the three cases in the filtration argument.
- The process Z^A_t is a supermartingale with respect to the filtration G^A_t, enabling a stopping-time argument.
B.1. Mirror-conservatism
The mirror-conservatism results show that randomized and discrete p-values from permutation tests, as well as one-sided monotone-likelihood-ratio tests, satisfy the required symmetry property.
- Permutation tests can produce discrete p-values by ranking the observed statistic over a transformation group or a sampled subset of that group.
- Fuzzy p-values randomize within jumps of the null distribution and are termed fuzzy p-values.
- A family of densities with monotone likelihood ratio yields mirror-conservative p-values under the construction in (41).
- The proof establishes the required probability bounds by relating inverse images under the p-value transformation to intervals determined by G_0.
- The argument handles both continuity points and atoms of G_0 when proving the randomized p-value property.
- The resulting p-values are mirror-conservative for the one-sided monotone-likelihood-ratio testing problem.