PaperlayerView exact revision on arXiv

Source-linked AI summary

False Discovery Rate Control via Data Splitting

Chenguang Dai, Buyu Lin, Xin Xing, Jun S. Liu

arXiv:2002.08542v2stat.ME

TL;DR

High-dimensional feature selection can make valid p-values or accurate estimates of the joint feature distribution challenging. This paper uses data splitting to construct mirror statistics and aggregates repeated splits with MDS. It reports asymptotic FDR control for linear and Gaussian graphical models, with MDS stabilizing selection and improving power.

  • Problem

    In high-dimensional regressions, constructing valid p-values or accurately estimating the joint feature distribution can be challenging, limiting BHq and model-X knockoffs.

    Source: §1.2 · p. 2

  • Method

    DS estimates independent feature-importance measurements from separate data splits to form mirror statistics, while MDS aggregates independent DS replications.

    Source: §1.2 · p. 3, §2.2 · p. 7

  • Results

    DS and MDS asymptotically control FDR in linear and Gaussian graphical models, and MDS stabilizes selection and improves power.

    Source: §2.2 · p. 7, §5 · p. 24

  • Takeaways & Limitations

    The framework is conceptually simple and computationally efficient, without requiring p-values or knowledge of the joint feature distribution.

    Source: §1.2 · p. 3

    The linear Lasso+OLS procedure may not readily extend to group Lasso, which can select more than n features and complicate its OLS step.

    Source: §5 · p. 24

Abstract

from arXiv · show

Selecting relevant features associated with a given response variable is an important issue in many scientific fields. Quantifying quality and uncertainty of a selection result via false discovery rate (FDR) control has been of recent interest. This paper introduces a way of using data-splitting strategies to asymptotically control the FDR while maintaining a high power. For each feature, the method constructs a test statistic by estimating two independent regression coefficients via data splitting. FDR control is achieved by taking advantage of the statistic's property that, for any null feature, its sampling distribution is symmetric about zero. Furthermore, we propose Multiple Data Splitting (MDS) to stabilize the selection result and boost the power. Interestingly and surprisingly, with the FDR still under control, MDS not only helps overcome the power loss caused by sample splitting, but also results in a lower variance of the false discovery proportion (FDP) compared with all other methods in consideration. We prove that the proposed data-splitting methods can asymptotically control the FDR at any designated level for linear and Gaussian graphical models in both low and high dimensions. Through intensive simulation studies and a real-data application, we show that the proposed methods are robust to the unknown distribution of features, easy to implement and computationally efficient, and are often the most powerful ones amongst competitors especially when the signals are weak and the correlations or partial correlations are high among features.

1 Introduction

The introduction motivates feature selection with FDR control and presents data splitting as a framework that avoids requiring p-values or the joint feature distribution. It proposes multiple data splitting to stabilize selection and improve power, and establishes theoretical and empirical support for DS and MDS in linear and Gaussian graphical models.

1.1 Background for FDR control in regression models

  • Feature selection seeks relevant predictors among potentially many features, supporting downstream computability, reproducibility, and interpretability.The paper also notes that its regression methods can be adapted to general multiple-testing problems.Source: §1.1 · p. 1
  • High-dimensional regression can make valid p-values or accurate feature-distribution estimates challenging, limiting BHq and model-X knockoffs.BHq relies on p-values, while model-X knockoffs require the joint feature distribution.Source: §1.2 · p. 2, §1.1 · p. 2

1.2 Motivations and main contributions

  • The data-splitting framework requires neither p-values nor the joint feature distribution, unlike BHq and model-X knockoffs.The framework is described as conceptually simple and computationally efficient.Source: §1.2 · p. 3
  • MDS aggregates independent DS replications using adjusted inclusion rates, stabilizing selection and improving power over a single split.The paper reports that MDS reduces FDR and boosts power in most cases.Source: §1.2 · pp. 3–4
  • Across tested cases, DS and MDS control FDR at the designated level, while MDS achieves the best or nearly best power across simulations and real-data applications.The paper establishes the methods under standard conditions for linear and Gaussian graphical models.Source: §1.2 · p. 3
  • DS constructs each feature’s mirror statistic from two independent feature-importance measurements; null-feature symmetry about zero supports estimating false positives from the negative tail.Features with mirror statistics above a cutoff are selected.Source: §1.2 · p. 3, Figure 1 · p. 3
  • For linear models, DS uses Lasso on one data split to screen features, then OLS on the other; its FDR guarantee requires sure screening.For Gaussian graphical models, the methods apply to nodewise regressions at level q/2, with selections combined by the OR rule.Source: §1.2 · p. 4

2 Data Splitting for FDR Control

The section develops FDR control using null-symmetric mirror statistics from independent data-split estimates, and studies repeated splitting through MDS, including conditions for control and its feature rankings in the Normal means model. There, MDS ranks features more consistently with their p-values than DS and nearly recovers the power lost to sample splitting.

2.1 Single data splitting

  • A single split estimates two independent coefficient sets and combines them into a mirror statistic; for null features, that statistic is symmetric about zero.The coefficient estimates may use different procedures, and the mirror function is nonnegative, symmetric in its arguments, and increasing in each; relevant features tend to produce positive, larger statistics.Source: §2.1 · pp. 4–5
  • Under null symmetry, weak dependence, and a feasibility condition, the single-split procedure controls FDR at any designated level q asymptotically.The weak-dependence assumption restricts correlations among null features, while the feasibility condition requires a threshold whose FDP is at most q with probability approaching one.Source: §2.1 · pp. 6–7

2.2 Multiple data splitting

  • MDS repeats data splitting, ranks features by their inclusion rates, and chooses a cutoff using the average FDP across split-specific selections.The inclusion rate is an importance measure relative to the single-split procedure, not an estimate of selection probability; simulations reported in the section confirm FDR control and broadly improved power over single splitting.Source: §2.2 · pp. 7–8
  • MDS asymptotically controls FDR in sparse and non-sparse regimes when the single-split procedure controls FDP and its mirror statistics satisfy the respective ranking-consistency conditions.In the non-sparse regime, ranking consistency yields o(p0) false positives; in the sparse regime, stronger consistency yields o(p1) false positives.Source: §2.2 · p. 8

2.3 A theoretical study of MDS for the Normal means model

  • In the Normal means model, MDS inclusion rates asymptotically order feature pairs by their absolute sample-mean signals when their standardized difference lies within fixed positive bounds.Conditional on that bounded-gap event, the probability of reversing the inclusion-rate ordering is o(1).Source: §2.3 · p. 9
  • When two p-values differ by O(1/√n), MDS ranks their features consistently with the p-values with probability approaching 1, whereas DS misranks them with non-vanishing probability.Source: §2.3 · p. 9
  • MDS inclusion rates yield less noisy feature rankings than DS mirror statistics and are approximately a monotone decreasing function of p-values.Source: §2.3 · p. 9, Figure 2 · p. 10
  • At n ≥ 5000, MDS’s swap probability approaches 0, while DS’s remains approximately constant, slightly below 0.5.Source: §2.3 · p. 9, Figure 2 · p. 10
  • For this model, MDS almost recovers DS’s power loss from sample splitting because full-data p-values summarize all information related to testing.Source: §2.3 · p. 10
  • MDS power gains generally diminish after about 100 or 200 DS replications; increasing replications from 400 to 10,000 only slightly reduces ranking noise.Source: §2.3 · p. 10

3 Specializations for Different Statistical Models

The section specializes data splitting (DS) and multiple data splitting (MDS) for linear and Gaussian graphical models, establishing asymptotic FDR control under model-specific conditions. It also identifies settings where guarantees or power analyses remain limited.

3.1 Linear models

  • For linear models, the proposed Lasso + OLS procedure screens features on one data half and estimates coefficients by OLS on the other half before constructing mirror statistics.Sure screening makes the symmetry assumption hold; other dimension-reduction methods can replace Lasso if sure screening holds with probability approaching one.Source: §3.1 · pp. 10–11, §3 · p. 10

Assumption 3

  • For linear models, Proposition 5 establishes asymptotic FDR control for DS and MDS using Lasso + OLS under Assumption 3.The stated regime includes log p = o(n^ξ), for some ξ ∈ (0, 1).Source: Assumption 3 · p. 11, §3.1 · p. 11
  • Detailed power analyses for DS and MDS beyond Proposition 5 remain an open direction identified by the authors.The paper notes that these analyses await future investigation.Source: Assumption 3 · p. 11

3.2 Gaussian graphical models

  • Gaussian graphical-model estimation uses nodewise regressions and combines their Lasso selections with an OR rule, selecting an edge if either endpoint’s regression selects it.For MDS, the authors use repeated splitting within each nodewise regression followed by OR aggregation, which empirically tends to have higher power than repeating the entire procedure.Source: §3.2 · pp. 12–13

Assumption 4

  • Under Assumption 4 and a minimum nodewise-degree condition, DS and MDS asymptotically control FDR at the designated level in Gaussian graphical models.The result is stated for log p = o(n^ξ), with ξ ∈ (0, 1).Source: Assumption 4 · p. 13
  • DS and MDS can handle graphs that are not too sparse, but may suffer when the true graph is ultra-sparse.The authors contrast this with GFC, which tends to work well in the ultra-sparse setting.Source: Assumption 4 · p. 13

4 Numerical Illustrations

Across linear and Gaussian graphical models, the simulations compare FDR control and power across feature distributions, correlations, dimensions, and signal strengths. MDS is often among the strongest methods, with advantages in settings including highly correlated features and the HIV drug-resistance application.

4.1 Linear model

  • Among three mirror-statistic choices, all achieve FDR control, while f3 yields the highest power and is used in the subsequent simulations.The empirical results suggest f3 may also be a good choice beyond orthogonal designs.Source: §4.1 · p. 14
  • MDS power increases with the number of DS replications and becomes relatively stable after m ≥ 50, suggesting that a small number of replications can approach full power.The examined settings use n = 500, p = 500, p1 = 50, and correlations ρ = 0.0 or ρ = 0.8.Source: §4.1 · pp. 14–15, Figure 4 · p. 15
  • Across normal-design settings, all four methods control FDR; MDS and the knockoff filter lead in power, with MDS stronger for higher correlations or weaker signals and the knockoff filter stronger in the opposite regimes.MDS is reported as more robust to highly correlated designs; under constant pairwise correlation, the knockoff filter is significantly less powerful when ρ ≥ 0.4.Source: §4.1 · p. 15, Figure 5 · p. 16
  • MDS maintains FDR control and reasonably high power under both Gaussian-mixture and multivariate t designs, while the knockoff filter is overly conservative for the mixture and loses FDR control for t features.The authors attribute the knockoff results to misspecification in its construction.Source: §4.1 · pp. 15–16
  • For the GWAS design, all methods achieve FDR control, and MDS has the highest power across the tested dimensions and signal strengths.The design uses 800 samples and 70 relevant features; the compared settings vary dimension or signal strength.Source: §4.1 · p. 16, Figure 7 · p. 18
  • Across the linear-model simulations, DS and the knockoff filter have comparable FDP variance except when knockoffs are overly conservative, while MDS has lower FDP variance than knockoffs in most settings.The authors connect the lower MDS variance to its de-randomized nature.Source: §4.1 · pp. 16, 19

4.2 Gaussian graphical model

  • Across banded graphs, all methods control FDR, and DS and MDS have higher power and lower FDR than the two competitors; for blockwise diagonal graphs, MDS has higher power and lower FDR than DS across scenarios.GFC generally performs similarly to DS in blockwise graphs, except that its power drops significantly when p is large.Source: §4.2 · p. 19, Figure 8 · p. 20, Figure 9 · p. 21

4.3 Real data application: HIV drug resistance

  • For PI drugs, MDS is best for three of seven drugs and is comparable to DeepPINK for three others; for NRTI drugs, MDS is best for four of six, with drug-specific ties or comparisons elsewhere.For DDI, TDF, and X3TC, the knockoff filter selects no mutations.Source: §4.3 · p. 21, Figure 10 · p. 22, Figure 11 · p. 23

5 Concluding Remarks

The paper concludes that DS and MDS asymptotically control FDR in linear and Gaussian graphical models, with MDS offering stability and improved power. It identifies extensions to group-structured features, nonlinear models, covariance testing, and dependent or hierarchical data as future work.

5 Concluding Remarks

  • DS and MDS asymptotically control FDR in linear and Gaussian graphical models, while MDS stabilizes selection and improves power.Both methods require little knowledge of the joint feature distribution and are described as simple to implement using existing high-dimensional regression software.Source: §5 · p. 24
  • Future work includes group-structured features, nonlinear models, sparse covariance testing, and data with dependent observations or hierarchical structure.For group Lasso, the companion OLS step may be difficult to apply because it can select more than n features; the framework is applicable to covariance testing when zero-entry estimators are symmetric about 0.Source: §5 · p. 24

6.1 Proofs

The proofs analyze symmetric, increasing nonnegative mirror-statistic functions and establish properties of the proposed selection rules and MDS cutoff. They also establish asymptotic power and FDR guarantees for data splitting in linear and graphical-model settings, and show that mirror-statistic and p-value rankings may differ when signal magnitudes are very close.

6.1.1 Proof of Proposition 1

  • For Proposition 1, the likelihood-ratio rule using f(u, v) = u + v controls type-I error and achieves asymptotic FDR control while attaining the optimal power.The proof concludes that this rule is also optimal among the considered choices of f.Source: §6.1.1 · pp. 28–29

6.1.2 Proof of Proposition 2

  • For Proposition 2, the proof uses mirror-statistic symmetry and tail convergence to establish asymptotic FDR control at the selected cutoff.The supporting lemma assumes p0 → ∞ as p → ∞; the proof concludes that FDP†p(τq) ≤ q.Source: §6.1.2 · pp. 29–30

6.1.3 Proof of Proposition 3

  • For Proposition 3, the proof bounds the MDS cutoff with high probability and derives k = o(p0) in the non-sparse regime and k = o(p1) in the sparse regime.The cutoff bounds and false-positive conclusions are established separately for the two regimes.Source: §6.1.3 · pp. 30, 32, 33

6.1.4 Proof of Proposition 4

  • For Proposition 4, the proof establishes that the mirror-statistic variance is of order 1/n and analyzes split-specific selection events using pivotal quantities.It partitions sample splits according to which of two features is selected and characterizes those events through pivots.Source: §6.1.4 · p. 33
  • When signal magnitudes differ by only o(1/√n), the rankings by mirror statistics need not align with p-value rankings.The proof compares shifted intervals under the standard Normal distribution; in this close-signal case, the O(1) term becomes o(1).Source: §6.1.4 · p. 34

6.1.5 Proof of Proposition 5

  • DS asymptotically achieves power 1 with FDR control, while MDS consistently ranks signals above nulls and has FDR control.The MDS proof also bounds the number of missed signals through its FDP argument.Source: §6.1.5 · p. 36

6.1.6 Proof of Proposition 6

  • Under Assumption 4, sure screening and the symmetric assumption hold simultaneously across all nodewise regressions with probability approaching 1.The proof combines restricted-eigenvalue and Lasso estimation bounds across the regressions.Source: §6.1.6 · p. 37

6.1.7 Proof of Proposition 7

  • For graphical models, DS has FDR control, and MDS has FDR control and power guarantees under Assumption 4.The proof obtains concentration of the null-statistic counts using a Lipschitz approximation and Gaussian concentration.Source: §6.1.7 · p. 39

6.2 The Normal means model

In the Normal means model, MDS is more conservative than DS but retains competitive power, and its power stabilizes with enough data-splitting replications. MDS also improves feature rankings over DS.

6.2 The Normal means model

  • MDS is more conservative than DS, yet its power is comparable to BHq and higher than DS across signal strengths.DS controls FDR slightly below q = 0.1; BHq controls it at qp1/p = 0.08.Source: §6.2 · p. 39
  • MDS power becomes quite stable when the number of data-splitting replications exceeds 100.Source: §6.2 · p. 39
  • MDS improves feature rankings over DS, and its ROC curve largely overlaps the p-value curve except at the right tail.At q = 0.5, that tail represents features with inclusion rates too low for MDS to rank properly.Source: §6.2 · p. 39

6.3 Simulation details

The simulations vary signal strength, correlation, and replication count across Normal means and linear-model designs. They report empirical FDR and power, with additional FDP-variability comparisons in several designs.

6.3 Simulation details

  • The blockwise diagonal Toeplitz covariance setup uses p′ = p/10, with ρ denoting the correlation factor.Source: §6.3 · p. 39
  • The Normal means experiments compare empirical FDR and power while varying signal strength or the number of MDS replications.They use n = 500, p = 800, p1 = 160, and q = 0.1; a separate ROC experiment uses q = 0.5 and m = 1000.Source: Figure 12 · p. 40, Figure 13 · p. 40
  • A separate experiment compares empirical FDR and power for three mirror statistics built from f1, f2, and f3.Its algorithmic settings follow Figure 3.Source: Figure 14 · p. 41
  • Low-dimensional linear-model experiments use independent Gaussian features with Toeplitz covariance, varying correlation or signal strength and tracking FDP standard deviations.They use 50 relevant features and designated FDR level q = 0.1.Source: Figure 15 · p. 41, Figure 16 · p. 42
  • Other linear-model experiments report empirical FDR, power, and FDP standard deviations under settings specified by Figures 5 and 6.Source: Figure 17 · p. 42, Figure 18 · p. 43, Figure 21 · p. 45
  • For a constant-correlation design, simulations vary pairwise correlation or signal strength and report empirical FDR, power, and FDP standard deviations.The design uses 50 relevant features and q = 0.1.Source: Figure 19 · p. 44, Figure 20 · p. 45
  • A further comparison reports FDP standard deviations for linear models with a GWAS design matrix.Its algorithmic settings follow Figure 7.Source: Figure 22 · p. 46
Loading 2002.08542v2…