DeepPINK: reproducible feature selection in deep neural networks

1 Introduction

• Small random perturbations can dramatically change feature importance from existing DNN explanation methods, raising concerns about reproducible explanations.Those methods may also be difficult to generalize beyond specific DNN architectures.Source: §1 · p. 2
• Meaningful p-values for measuring DNN feature importance remain unknown, making feature selection with controlled error rates challenging.The false discovery rate measures the expected proportion of falsely selected features among all selected features.Source: §1 · p. 2
• DeepPINK pairs each input feature with its knockoff in an additional pairwise-connected layer, integrating knockoff filtering with a DNN.The knockoff framework compares each original feature's importance with its knockoff counterpart.Source: §1 · p. 2
• DeepPINK achieves FDR control with higher power than many state-of-the-art methods and is applied to two real data sets.The authors describe extensions to CNNs and RNNs as ongoing research.Source: §1 · pp. 2–3

2.1 Model setting

• The model setting assumes i.i.d. observations and a subset of features conditionally independent of the response given the relevant features.The goal is accurate selection of relevant features with controlled error while enabling effective prediction.Source: §2.1 · p. 3

2.2 False discovery rate control and the knockoff filter

• Model-X knockoffs control FDR at a target level in finite samples, with arbitrary dependence between the response and features.Knockoff features match the original feature distribution under swaps and are independent of the response given the original features.Source: §2.2 · p. 3
• For Gaussian features, the paper describes a valid knockoff construction subject to a positive-definiteness condition on the conditional covariance matrix.The construction uses a diagonal matrix with positive components.Source: §2.2 · p. 3
• Knockoff statistics contrast each feature's importance with its knockoff's using an antisymmetric function.Important features are expected to have large positive statistics, while unimportant ones should have magnitudes symmetric around zero.Source: §2.2 · p. 4
• The knockoff filter selects features whose statistics exceed a threshold; the knockoff+ threshold controls exact FDR under mild conditions.The standard knockoff threshold controls a slightly modified version of FDR.Source: §2.2 · p. 4

• DeepPINK restricts its knockoff-DNN implementation to Gaussian designs, where knockoff features can be constructed using Equation 1.The paper notes that knockoffs can be generated from a general covariate distribution, but this implementation uses Gaussian design.Source: §3 · p. 4
• Naively feeding original and knockoff features into an MLP controlled FDR but produced extremely low, sometimes zero, power, motivating DeepPINK.Source: §3 · p. 4
• Each of DeepPINK’s p filters connects one feature with its knockoff, and equally initialized weights compete during training.A larger-magnitude original-feature weight than its knockoff counterpart provides evidence of importance; similar values indicate less importance.Source: §3 · p. 4
• A linear activation encourages competition among features, while the filter outputs feed a fully connected MLP that predicts the response.The illustrated MLP has two hidden layers, each with p neurons, and uses ReLU activation and L1 regularization.Source: §3 · p. 4, Figure 1 · p. 5
• DeepPINK combines each feature’s relative weight against its knockoff with its importance among all features to define importance measures.The measures are used to form knockoff statistics and apply the feature-selection filter.Source: §3 · p. 5

4.1 Model setups and simulation settings

• The linear-model simulation used y = Xβ + ε, with X drawn from a Gaussian design and target FDR q = 0.2.The Single-Index simulation used an unknown link function; the reported noise was Gaussian with σ = 1.Source: §4.1 · pp. 5–6
• Simulations covered linear and Single-Index models with n = 1000 and feature dimensions p from 50 to 3000.The linear model used 30 nonzero signals; the Single-Index model used 10 and the link g(x) = x^3/2.Source: §4.1 · pp. 5–6

4.2 Simulation results

• The study compared DeepPINK with MLP, DeepLIFT, random forest, and linear-kernel SVR using method-specific feature-importance measures.Linear-kernel SVR is misspecified for Single-Index data, and nonlinear-kernel SVR was not included because the authors could not find an available feature-importance measure.Source: §4.2 · p. 6
• DeepPINK controlled empirical FDR well below the target and had the highest power in almost all simulated settings.The reported results cover linear and Single-Index models; the authors attribute its advantage over MLP and DeepLIFT to direct competition between original and knockoff features.Source: §4.2 · p. 6, Table 1 · p. 6

5 Real data analysis

• For the real-data studies, the target FDR level was q = 0.2.Source: §5 · p. 7

5.1 Application to HIV-1 data

• The HIV-1 analysis selected mutations associated with resistance across 7 protease inhibitors, 6 nucleoside reverse-transcriptase inhibitors, and 3 nonnucleoside reverse-transcriptase inhibitors.The response was log-transformed resistance; patients with missing resistance data were removed and mutations appearing fewer than three times were excluded.Source: §5.1 · p. 7
• DeepPINK matched or exceeded Knockoff’s power in 9 of 13 cases and BHq’s power in 10 of 13, with comparable or better false discovery proportion, respectively.For APV, it recovered 18 mutations, none outside the treatment-selected mutation list.Source: §5.1 · p. 7

5.2 Application to gut microbiome data

• At target FDR q = 0.2, DeepPINK identified eight nutrient and bacteria-genus features associated with BMI in a cross-sectional study of 98 healthy volunteers.Three identified genera overlapped the four genera found using Lasso, and Table 2 reports literature evidence for the selected features.Source: §5.2 · p. 7

• DeepPINK identifies a subset of relevant input features under FDR control using a pairwise-connected layer that encourages competition between original features and their knockoffs.Source: §6 · pp. 7–8
• DeepPINK achieves FDR control with much higher power than the naive combination of knockoffs and a vanilla multilayer perceptron.Source: §6 · p. 8
• Extending DeepPINK to CNNs and RNNs is identified as a future direction.Source: §6 · p. 8
• DeepPINK improves the original knockoff inference framework and lets practitioners avoid handcrafting feature-importance measures.The paper notes that feature-importance measures depend heavily on the algorithm used to fit the model.Source: §6 · p. 8