Source-linked AI summary
Controlling the False Discovery Rate via Model-Agnostic Importance Measures
Mengtao Wen, Haojie Ren, Changliang Zou
TL;DR
Selecting relevant covariates for complex models matters for interpretability, but existing approaches can rely on model assumptions or covariate-distribution knowledge. MAS uses agent-X importance with resplitting and aggregation; the authors report asymptotic FDR control across models and learning algorithms, while noting an unresolved theoretical issue for covariate-adaptive importance.
Problem
Covariate selection for complex models faces model-dependent assumptions or reliance on the joint covariate distribution, while LOCO-based testing can incur computational burden, dependence, and power loss.
Source: §1 · p. 1Method
MAS combines cross-fitted agent-X importance with resplit-per-covariate splitting and data aggregation to perform model-agnostic FDR-controlled selection without estimating the distribution of X.
Source: §1.1 · p. 3Results
MAS asymptotically controls FDR across different models and machine-learning or statistical-learning algorithms.
Source: §6 · p. 18Takeaways & Limitations
MAS requires no prior knowledge of the model or covariate distribution, providing a model-agnostic framework for FDR control.
Source: §6 · p. 17The covariate-adaptive importance measure is designed under specific algorithms and still lacks rigorous theoretical results.
Source: §6 · p. 18
Abstract
from arXiv · showhide
Selecting covariates while controlling the false discovery rate (FDR) is critical for interpretable scientific discovery, especially with complex machine learning models. Traditional variable selection methods often rely on specific model assumptions, while Model-X knockoffs provide a general way that leverages knowledge of the covariate distribution to produce a variable selection algorithm. This study puts forward a complementary method to Model-X knockoffs, named Model-Agnostic Selection with agent-X importance (MAS), which utilizes the sample-splitting strategy and a novel variable importance measure to bypass distributional assumptions on covariates. The proposed model-agnostic agent-X importance is defined to evaluate the difference of oracle predictiveness on both the original distribution and the distribution with agent variable in population level, and can be efficiently estimated by constructing the agent variable as some constant and cross-fitting strategy. The MAS method integrates a resplit-per-covariate strategy to decouple estimator dependencies and a data aggregation technique to enhance power. Theoretical guarantees for asymptotic FDR control are established under mild conditions. Extensive simulations and real-data analyses demonstrate that MAS provides competitive performances with existing works in FDR control, computational efficiency and power.
1 Introduction
Identifying relevant covariates can improve interpretability and reproducibility, but existing selection approaches face model or covariate-distribution requirements, computational cost, dependence, and power loss. MAS combines agent-X importance, resplitting, and data aggregation for model-agnostic FDR control.
1 Introduction
- The paper targets selection of conditionally relevant covariates while controlling FDR, addressing the interpretability and reproducibility needs of complex machine-learning applications.Existing approaches include model-dependent procedures, while Model-X knockoffs rely on the joint distribution of covariates.Source: §1 · pp. 1–2
1.1 Our main ideas and contributions
- MAS offers a model-agnostic alternative to Model-X knockoffs without estimating the distribution of X, using agent-X importance and cross-fitting for FDR control.The framework also incorporates resplit-per-covariate splitting and data aggregation.Source: §1.1 · p. 3
- Agent-X importance compares predictiveness on the original distribution with predictiveness after replacing one covariate by an agent variable, avoiding reduced-model fits.The agent can be a fixed constant; the measure can preserve a signal that vanishes for LOCO as covariate correlation increases.Source: §1.1 · p. 3
- Resplitting samples independently for each covariate approximately decouples null importance estimators, supporting valid FDR control.The strategy is designed to address dependence among estimates computed from the same data.Source: §1.1 · p. 3
- Data aggregation combines evidence through ranking statistics that improve signal-to-noise ratio while retaining marginal symmetry for p-value-free thresholding.The approach is motivated by power loss from sample splitting.Source: §1.1 · p. 3
2 Model-Agnostic Selection with Agent-X Importance
MAS estimates agent-X importance with cross-fitting, independently resplits samples by covariate to reduce dependence, and aggregates evidence to improve power. Its algorithm reduces fitting demands relative to LOCO, while preserving the p-value-free thresholding approach.
2.1 Agent-X importance measure and its estimate
- Agent-X importance is nonnegative under the stated conditions and is zero when the best predictive function is unrelated to the covariate; cross-fitting and within-fold splitting support its estimation.For MSPE with the agent set to zero, the measure is expressed as an expected squared difference between predictions with the covariate observed and set to zero.Source: §2.1 · p. 4
2.2 Resplit-per-covariate strategy
- Resplitting each fold independently for each covariate relieves dependence among null importance estimates without changing their asymptotic results, preserving applicability of the p-value-free threshold rule.Without this strategy, null estimates can share dominant terms and become nearly identical.Source: §2.2 · p. 5
2.3 Data aggregation
- Data aggregation uses a positive auxiliary statistic to combine importance estimates, with the potential to raise signal-to-noise ratio when its limiting values are larger for important variables.The aggregated statistics retain asymptotic symmetry under the stated convergence condition, allowing p-value-free thresholding.Source: §2.3 · p. 6
2.4 MAS algorithm
- MAS fits the full predictive model K times for p agent-X measures, compared with at least p + 1 fits for LOCO, reducing computational burden especially when p is large.Its model-fitting complexity is of the same order in n and p as Model-X knockoffs.Source: §2.4 · p. 6
3 Theoretical Results
Under stated conditions, MAS asymptotically controls FDR; this section justifies agent-X nonnegativity, the resplit strategy’s reduction of null-estimator dependence, and the assumptions underlying data aggregation.
3.1 FDR control
- When p = O(n^r), MAS has asymptotic FDR at most α under Assumptions 1–5.The result applies for any α ∈ (0, 1), and the procedure can use other auxiliary-statistic constructions satisfying Assumption 5.Source: §3.1 · p. 8
3.2 Non-negativeness of agent-X importance measure
- Under Proposition 2’s conditions and a valid agent construction, ψj ≥ 0, with equality when Xj ⊥⊥ Y | X−j.The construction requires the distribution with the agent variable to belong to P0,j.Source: §3.2 · p. 8
- Larger positive c increases agent-X importance under alternatives, but amplified fitting error can impair finite-sample normality and FDR control.For standardized covariates, the authors recommend ˜Xj = z0.95 as a practical compromise.Source: §3.2 · p. 9
- As |ρ| approaches one, LOCO importance approaches zero while agent-X importance remains strictly positive for nonzero coefficients in the stated Gaussian linear model.This comparison is for MSPE with agent variable ˜Xj = c.Source: §3.2 · p. 9
3.3 Analysis of resplit-per-covariate strategy
- Resplit-per-covariate makes null importance estimators’ strong dependence asymptotically vanish, supporting more stable and reliable FDR control.The argument approximates the major term by a sum of independent, identically distributed variables.Source: §3.3 · p. 10
3.4 Justification of data aggregation
- Proposition 3 bounds βj below by V(f*, PZ) and gives uniform estimation error o_p((log p)^−1) under its stated residual-rate conditions.The upper bound depends on V; truncating V at a large constant is suggested as a heuristic.Source: §3.4 · p. 10
4 Structure-Powered Enhancement
MAS can incorporate known group or ordering structure through its auxiliary statistics, and can also use covariate-adaptive importance to improve power over Algorithm 1.
4.1 Incorporating prior information
- Known groups can share βj values within groups, while nested variable sets can construct decreasing βj values for ordered hypotheses.The paper also describes combining multiple structures by taking means over their βj constructions.Source: §4.1 · p. 11
4.2 Covariate-adaptive importance
- Covariate-adaptive importance performs better than Algorithm 1 with higher power, though rigorous justification of its threshold rule remains future work.It evaluates local models at steps where the covariate is selected and averages across folds.Source: §4.2 · p. 12
5 Numerical Studies
The simulations compare MAS with alternative agent-variable constructions, ablations, and competing methods; the real-data analysis evaluates selection across seven HIV-1 drug-resistance outcomes. Results show trade-offs across settings, including stronger MAS power from data aggregation and structure-powered variants.
5.1 Simulation settings
- Simulations assess FDR, averaged power, and runtime over 400 replications at target FDR α = 0.2, using MSPE and five-fold cross-fitting by default.The study includes three representative models and a real-data example.Source: §5.1 · pp. 12–13
5.2 Choices of agent-X variable
- With lasso in (S1), MAS using the constant agent z0.95 balances power and FDR control, while larger constants improve power but make FDR control more challenging.MAS-permu performs similarly to MAS-1; z0.95 is recommended for standardized covariates.Source: §5.2 · p. 13, Table 1 · p. 14
5.3 Ablation experiments
- All methods control FDR in the ablation settings, while MAS exceeds MAS-AgentX in power, with an almost 16% largest gain at p = 250.MAS-AgentX exceeds MAS-LOCO and BH-LOCO in empirical power and computation speed; MAS and MAS-AgentX have computation times comparable to Knockoff.Source: §5.3 · pp. 13–14, Table 2 · p. 14
5.4 Comparisons with competing methods
- In (S1), DS and BHq have higher power than MAS and Knockoff, while MAS stays near the target FDR; in (S2), MAS and Knockoff control FDR and have higher power than DS and BHq.In (S2), MAS has lower power than Knockoff.Source: Figure 1 · p. 15, §5.4 · p. 15
5.5 Structure-powered enhancements of MAS
- In (S3), MAS-PI and MAS-CA significantly improve power over MAS; MAS-Ensemble has higher power than MAS-XGBoost and power comparable to Knockoff while maintaining reasonable FDR control.Knockoff FDR is inflated for heavy-tailed t errors at large ρ.Source: §5.4 · p. 16, Table 4 · p. 17, §5.5 · p. 17
5.6 Real data analysis
- Across seven drugs, MAS and Knockoff outperform DS and BHq in FDR control; MAS has power comparable to Knockoff and more stable across drugs.DS has inflated FDR for APV, IDV, NFV, and RTV.Source: §5.6 · p. 17
6 Concluding Remarks
MAS is presented as a model-agnostic framework that asymptotically controls FDR across learning models and algorithms. The appendix states conditions for asymptotic linearity and identifies open directions for strengthening the method.
6 Concluding Remarks
- MAS asymptotically controls FDR across different models and machine- or statistical-learning algorithms.The conclusion describes the framework as requiring no prior knowledge of models or covariate distributions.Source: §6 · pp. 17–18
- Covariate-adaptive importance is algorithm-specific and has numerical evidence of high power, but rigorous theory remains for future study.Further work is also proposed on how agent-X signal retention behaves under nonlinear regression functions and other predictiveness measures.Source: §6 · p. 18
A.1 Asymptotic linearity
- Conditions (C1)–(C2) concern convergence of the fitted model, while (C3)–(C4) address estimation effects and local differentiability of predictiveness.Under the stated conditions, Proposition 5 implies the asymptotic-linearity requirement in Assumption 1(b).Source: §A.1 · pp. 22–23
A.2 Non-negativeness of agent-X importance measures
For the stated class of predictiveness measures, agent-X importance is zero for conditionally unimportant covariates and nonnegative otherwise. The resplit strategy is extended to uneven sample splits under a bounded split proportion.
A.2 Non-negativeness of agent-X importance measures
- For (M1)–(M3), linearity of V in the distribution expresses ψj as a directional predictiveness difference against the original distribution.The comparison distribution replaces Xj with a random variable W while retaining Y and X−j.Source: §A.2 · p. 23
- Under the stated assumption, ψj = 0 when Y is conditionally independent of Xj given X−j, and ψj ≥ 0 otherwise.The directional predictiveness difference ˙V(f∗, PZ; P0 − PZ) is zero when PZ ∈ P0,j and nonnegative when PZ ∉ P0,j.Source: §A.2 · p. 23
B.1 Generalization of resplit-per-covariate strategy
- The resplit-per-covariate analysis allows an uneven split with n1 observations, provided n1/n stays bounded away from 0 and 1.The corresponding approximation uses independent Bernoulli variables with probability n1/n.Source: §B.1 · p. 24
B.2 Proof of Lemma 1 and 2
The proof replaces dependent resplitting indicators with independent Bernoulli variables and bounds the resulting approximation error. This yields a common argument for equal and uneven splits.
B.2 Proof of Lemma 1 and 2
- Within each fold, resplitting indicators can be dependent, so the proof applies binomialization separately by fold and merges independent terms across folds.The indicators for different observations are dependent only when they belong to the same fold.Source: §B.2 · p. 24
- The construction produces independent Bernoulli variables across observations and covariates, each with success probability n1/n.For an equal split, Lemma 1 is the special case n1 = n/2.Source: §B.2 · p. 25
- For each j, the split count N_{n,j} is distributed as Binomial(n, n1/n), with |J_{n,j}| = |N_{n,j} − n1|; the text then invokes Hoeffding’s inequality and truncates |J_{n,j}|.Source: §B.2 · p. 26
C.1 Proof of Theorem 1
The proof establishes asymptotic FDR control by approximating the test statistics, using null symmetry, and applying the threshold rule. Its conclusion is an asymptotic bound on expected FDR.
C.1 Proof of Theorem 1
- The scaled quantity √n(ψ̂j−ψj)β̂j/2 equals G̃jβj + Qn,jβj + Rn,jβj plus √n(ψ̂j−ψj)(β̂j−βj)/2, with Q and R bounded uniformly over j.maxj |Qn,j| = Op([(log p)^3/n]^(1/4)) = op((log p)^−1/2), and maxj |Rn,j| = op((log p)^−1/2).Source: §C.1 · p. 30
- The threshold step shows that the selected threshold lies below t* with probability tending to one, linking the estimated FDP bound to the final FDR result.The proof identifies the condition at t* with the estimated FDR being no greater than α.Source: §C.1 · pp. 32, 34
- lim sup_{n,p→∞} FDRA ≤ α, yielding asymptotic control of the expected false discovery rate.The proof first obtains FDPA ≤ α + op(1) under the threshold rule.Source: §C.1 · p. 34
C.2 Proof of other propositions
The supplied material states a condition for bounded predictiveness quantities, includes a probability lemma, and records implementation choices.
C.2 Proof of other propositions
- For (M1), the predictiveness quantity’s lower bound is usually positive when noise exists, and its upper bound is finite under mild conditions such as bounded f*.The passage gives f* ∈ ℓ∞(R^p) as an upper-bound condition.Source: §C.2 · p. 35
C.3 Basic lemmas
- The lemma states that conditioning Bernoulli indicators on their total count yields marginally independent Bernoulli(p) indicators.The stated conclusion is that ε_i ∼ Bernoulli(p).Source: §C.3 · p. 37
D.1 Implementation details
- Implementation uses the knockoff R package, BHq code for simulations, setting-dependent single or multiple data splits, and a modified threshold for p-value-free methods.The passage specifies single splits in S1–S2 and multiple splits in real-data analysis.Source: §D.1 · p. 38
D.2 Ablation experiments
In the S1 comparisons, MAS-framework methods show more reasonable FDP and power than BH-LOCO, whose outcomes can be extreme.
D.2 Ablation experiments
- BH-LOCO’s strong p-value dependence yields either no discoveries or overwhelming discoveries.The reported extremes correspond to zero or near-one FDP and power per replication.Source: §D.2 · p. 39
- MAS-framework methods show more reasonable FDP and power than BH-LOCO in the plotted S1 comparisons.The violin plots cover p=50, 150, and 250 under the Table 2 setting.Source: Figure 3 · p. 39, §D.2 · p. 39
- Figure 3 plots FDP and power violins for p=50, 150, and 250 under the Table 2 S1 setting.Source: Figure 3 · p. 39
D.3 Classification task
In a logistic classification setting, all methods control FDR at the target level, while strong covariate dependence makes BHq extremely conservative.
D.3 Classification task
- The S4 task uses a logistic response model, 20 randomly selected signals, Lasso, and comparisons with BHq, DS, and Model-X knockoffs.Source: §D.3 · p. 39
- Strong covariate dependence makes BHq extremely conservative; otherwise, conclusions resemble those for S1.Source: §D.3 · p. 40
- All methods control FDR at the target level in logistic setting S4, with n=600 and p ranging from 100 to 500.Figure 4 compares MAS, Knockoff, DS, and BHq at α=0.2; time is reported in seconds.Source: §D.3 · p. 39, Figure 4 · p. 40