PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Knockoffs

Yaniv Romano, Matteo Sesia, Emmanuel J. Candès

arXiv:1811.06687v1stat.MEmath.STstat.APstat.ML

TL;DR

Model-X knockoffs require accurate knowledge of the feature distribution, limiting their use when that distribution is unknown or difficult to model. This paper trains deep generative knockoff machines by optimizing a validity criterion related to exchangeability, extending controlled variable selection to broader settings. Numerical experiments and a drug-resistance application support the method’s flexibility, while the authors note important scope and overfitting limitations.

  • Problem

    Model-X knockoff validity depends on knowing the feature distribution and constructing exchangeable knockoffs, which remains challenging for general distributions.

  • Method

    A deep neural network maps observed features and random noise to knockoffs, trained to optimize a loss measuring how well the generated copies satisfy knockoff validity.

  • Results

    Extensive numerical experiments show performance matching available exact knockoff constructions across several distributions, with the method also applied to mutations associated with HIV drug resistance.

  • Takeaways & Limitations

    Deep knockoff machines provide flexible tools for model-free controlled variable selection across a broad range of problems.

  • Takeaways & Limitations

    The HIV analysis uses limited data, trains on the same samples used for selection, and is presented as illustrative rather than as evidence of new scientific findings.

Abstract

from arXiv · show

This paper introduces a machine for sampling approximate model-X knockoffs for arbitrary and unspecified data distributions using deep generative models. The main idea is to iteratively refine a knockoff sampling mechanism until a criterion measuring the validity of the produced knockoffs is optimized; this criterion is inspired by the popular maximum mean discrepancy in machine learning and can be thought of as measuring the distance to pairwise exchangeability between original and knockoff features. By building upon the existing model-X framework, we thus obtain a flexible and model-free statistical tool to perform controlled variable selection. Extensive numerical experiments and quantitative tests confirm the generality, effectiveness, and power of our deep knockoff machines. Finally, we apply this new method to a real study of mutations linked to changes in drug resistance in the human immunodeficiency virus.

1 Introduction

Model-X knockoffs enable controlled variable selection without modeling the response, but their validity depends on accurately describing the feature distribution. The paper develops deep-learning tools to make knockoff construction practically model-free and broadly applicable.

  • Model-X knockoffs identify important variables among many explanatory variables while rigorously controlling the false discovery rate in complex models.
  • Their validity requires a precise description of the joint distribution of the explanatory variables, limiting use when reliable prior knowledge is unavailable.
  • The paper uses deep learning and large unsupervised datasets to sample approximate model-X knockoffs without specifying the feature distribution.
  • The proposed tools target model-free controlled variable selection across a broad range of applications.
  • The approach builds on deep generative modeling while addressing training and tuning challenges associated with alternative interacting-network methods.

2 Model-X knockoffs

Model-X knockoffs create negative controls by requiring pairwise exchangeability between original and knockoff features, enabling false-discovery-rate control without assumptions on the response model. Exact construction is difficult for general feature distributions, motivating weaker second-order approximations.

  • A valid knockoff copy has a joint distribution invariant to swapping each original feature with its corresponding knockoff.
  • Knockoffs act as negative controls, allowing variable-selection procedures to estimate and limit false positives.
  • The knockoff filter can control the false discovery rate without restricting the conditional distribution of the response given the features.
  • Exact validity depends on knowing the feature distribution and generating knockoffs satisfying the exchangeability condition, which is algorithmically challenging in general.
  • Second-order knockoffs relax exact exchangeability by matching the first two distributional moments, offering a practical approximation for some settings.
  • For Gaussian approximations, the vector s is chosen so the covariance construction remains positive semidefinite and knockoffs are as weakly correlated with originals as possible.

3 Deep generative models

Deep generative models learn to synthesize observations resembling an unknown data distribution while remaining nontrivial. The paper repurposes maximum mean discrepancy and differentiable moment matching as objectives for training knockoff generators.

  • Generative models approximate an unknown distribution to synthesize plausible observations that are sufficiently different from the training data.
  • Variational methods optimize a tractable lower bound on training-data log-likelihood, whereas generative adversarial networks minimize inconsistencies between generated and original samples.
  • The discriminator in generative modeling addresses a multivariate two-sample problem: determining whether samples come from the same distribution.
  • Maximum mean discrepancy compares distributions through kernel expectations and equals zero exactly when they match under a characteristic kernel.
  • Because maximum mean discrepancy is differentiable and computable, it can replace a GAN discriminator as a training objective for deep generative models.
  • The same objective can be repurposed to train a knockoff machine using gradient descent on the observed feature data.

4 Deep knockoff machines

Deep knockoff machines use neural networks to generate approximate knockoffs for unknown distributions, training their parameters by minimizing exchangeability and regularization objectives. The method supports second-order and higher-order knockoffs, with stochastic-gradient optimization providing a weak convergence guarantee.

  • Machine construction: A knockoff machine maps X and independent Gaussian noise V through fθ(X,V) to generate approximate knockoffs for an unknown distribution.The network parameters are estimated from observations of X by stochastic optimization.
  • Second-order machines: Second-order machines match means and covariance structure, while decorrelation regularization encourages knockoffs to differ from the original variables.The loss penalizes expectation and second-moment discrepancies and can also penalize empirical correlations between X and X̃.
  • Higher-order machines: Higher-order machines minimize a maximum mean discrepancy objective whose expectation is minimized when original and knockoff variables are exchangeable.The objective uses randomly selected swaps in practice rather than evaluating all p swaps at every iteration.
  • Higher-order machines: Theorem 1 states that equality in the expected JMMD criterion holds if and only if the machine produces valid knockoffs for PX.The expectation averages over the data, machine noise, and random swaps.
  • Optimization: Training samples Gaussian noise, splits observations into mini-batches, randomly selects swapping indices, evaluates the objective, and updates network parameters using its gradient.The objective combines JMMD, second-order, and decorrelation penalties with weights γ, λ, and δ.
  • Optimization: The squared gradient norm decreases on average as O(T^-1/2), but this weak convergence result does not ensure that the iterates reach a fixed point.The analysis assumes uniformly bounded variance for gradient estimates and does not establish convergence to a local minimum.

5 Robustness and diagnostics

The paper evaluates knockoff quality by testing exchangeability through complementary covariance, MMD, nearest-neighbor, and energy diagnostics, then supplements these checks with controlled false-discovery-rate experiments. These tools systematically compare approximate generators, although passing goodness-of-fit tests and controlling false discoveries are distinct concerns.

  • Exchangeability tests: The diagnostics compare the joint distributions of (X, ˜X) and its swapped version to assess compatibility with the exchangeability property.The full and partial exchangeability hypotheses are tested using independent samples from the original and swapped distributions.
  • Covariance diagnostics: Covariance diagnostics test equality of the covariance matrices, while larger discrepancies indicate a poorer knockoff fit.The statistic has an asymptotic non-parametric test under mild conditions, but is used here to grade alternative algorithms.
  • MMD diagnostics: MMD diagnostics capture higher-order discrepancies that covariance comparisons cannot detect, with large unbiased estimates providing evidence against exchangeability.Exact knockoffs yield MMD values equal to zero on average.
  • KNN diagnostics: The 1-nearest-neighbor statistic equals 1/2 in expectation for identical distributions, so constructions closer to 1/2 are preferred.Larger values indicate that nearest neighbors more often come from the same distribution and therefore provide evidence against the null.
  • Energy diagnostics: Energy distance is nonnegative and equals zero only for identically distributed samples; larger estimated values indicate a poorer fit.Under finite second moments, the statistic consistently tests equality in distribution as sample size grows.
  • Experimental comparison: Deep knockoff machines nearly match specialized algorithms when the feature distribution is known and surpass them in other cases.The paper presents this comparison as an experimental result following the diagnostic framework.
  • False-discovery-rate validation: Controlled experiments generate responses from known conditional likelihoods and measure false discoveries to assess whether approximate knockoffs maintain the nominal false discovery rate.These experiments address cases where false-discovery control may hold even when the joint distribution is far from exchangeable.

6 Numerical experiments

The experiments evaluate deep knockoff machines across Gaussian, discrete hidden Markov, heavy-tailed, and sparse Gaussian settings using controlled variable-selection and goodness-of-fit tests. Across these settings, the machines generally match or outperform comparison methods, while performance depends on distributional structure and tuning.

  • Experimental setup: Experiments train machines on synthetic datasets with n = 10^4 realizations, p = 100, stochastic-gradient optimization, and hyperparameters near (γ, λ, δ) = (1, 1, 1).Training uses mini-batches of size n/4, learning rate µ = 0.001, and T = 10^5 gradient steps.
  • Multivariate Gaussian: In the multivariate Gaussian experiment, deep-machine knockoffs are almost equivalent to second-order knockoffs, while oracle knockoffs are perfectly exchangeable and may be slightly more powerful.The comparison uses m = 150 samples and α = 0.1; Figure 5 averages false discovery rate and power over 1000 independent experiments.
  • Multivariate Gaussian: Misspecifying the Gaussian autoregressive oracle produces a significantly poorer fit unless its correlation parameter is close to the true value ρ = 0.5.The comparison indicates that the deep machine and second-order method capture the true distribution accurately despite little prior information.
  • Hidden Markov model: For discrete hidden Markov variables, power and false discovery rate are very similar across methods, while the oracle is slightly more conservative and the machine is almost equivalent to the second-order approximation.The authors note that architecture changes or more careful tuning might improve the discrete-data machine.
  • Multivariate Student’s t-distribution: For the multivariate Student’s t-distribution, deep knockoffs control the false discovery rate while second-order knockoffs fail, and the machine significantly outperforms them on goodness-of-fit diagnostics.This experiment uses m = 200 samples and α = 0, with reduced weighting of the empirical covariance matrix.
  • Sparse Gaussian variables: For sparse Gaussian variables, the machine’s knockoffs are not exact but are more accurate than second-order knockoffs and successfully control the false discovery rate.The authors attribute further possible improvement to parameter tuning, network design, and a larger training set.

7 Application

The application evaluates deep knockoffs on HIV mutation data using simulated responses and applies them to drug-resistance variable selection. Across experiments, the deep machine generally controls false discovery while achieving more discoveries than comparison methods, though the real-data analysis has important validation and overfitting limitations.

  • Limitations: The authors treat the HIV analysis as illustrative because its independent-and-identically-distributed assumptions cannot be rigorously validated without domain expertise and additional data.They also note that fitting and selecting on the same limited dataset may cause overfitting, whose theoretical behavior remains unresolved.
  • Data and setup: The HIV dataset contains n = 1431 viruses and p = 150 binary mutation features, with lopinavir resistance as the response.Half the features were previously associated with resistance changes.
  • Numerical experiments: The real-feature experiments use random subsets of m = 200 or 300 observations and average false discovery rate and power over 1000 replications.The response is simulated because the population distribution of mutations is unknown.
  • Numerical experiments: The deep machine controls the false discovery rate, whereas second-order knockoffs are slightly too liberal in the simulated-response experiments.The deep machine often makes more true discoveries while making fewer mistakes than the second-order approximation.
  • Fixed-feature comparison: With fixed original features, second-order and fixed-X knockoffs appear to control false discovery rate and achieve similar power, while the deep machine outperforms both.These comparisons average results over 1000 replications.
  • Drug-resistance discoveries: On the HIV drug-resistance dataset, deep knockoffs produce more discoveries than second-order and randomized fixed-X alternatives, with selections described as stable across 100 independent knockoff realizations.The knockoff filter uses nominal false discovery rate q = 0.1 and α = 0.1.

8 Discussion

The discussion presents deep knockoff machines as a broadly applicable, computationally practical extension of model-X knockoffs. It also distinguishes empirical behavior from established theory and identifies structural modeling, diagnostics, and theory as future directions.

  • Summary: Deep machines extend knockoffs to a vast range of problems and can be reproduced on a single graphics processing unit within a few hours.The authors report matching exact constructions on several distributions and outperforming previous approximations in more complex cases.
  • Summary: Independent-test diagnostics confirm that the machines learn to generate valid knockoffs without prior distributional knowledge.The discussion presents this as empirical support for the approach.
  • Theory and practice: Finite-sample false discovery control is guaranteed for the true feature distribution, whereas real analyses often estimate that distribution from the same data used for selection.The authors report that overfitting estimated distributions empirically tends to make knockoffs more conservative, but lack a rigorous explanation.
  • Future work: Future work includes exploiting known feature structure, expanding diagnostics, and extending theoretical results.The paper specifically mentions sequential dependencies, hidden Markov models, and systematic diagnostic comparisons.

A Proofs

The proofs establish that the swap-based discrepancy criterion characterizes pairwise exchangeability and analyze convergence of its stochastic optimization. They use unbiased gradient estimates and nonnegativity of maximum mean discrepancy.

  • Exchangeability proof: The exchangeability argument uses the fact that the maximum mean discrepancy is nonnegative and vanishes only when the compared distributions are equal.Applying this to every random swap yields equality in distribution between original and swapped knockoff pairs.
  • Exchangeability proof: The proof averages over random swaps, data partitions, and machine noise to connect empirical discrepancy estimates with the population discrepancy.The empirical estimator is treated as unbiased and the population discrepancy as nonnegative.
  • Optimization proof: The convergence proof starts from a first-order expansion of the gradient update and sums the resulting inequalities over T optimization steps.It then conditions on the initial iterate and uses unbiased stochastic gradients.
  • Optimization proof: The final bound follows by multiplying the accumulated inequality by 2/[LT(2µ − Lµ^2)].The displayed result retains terms involving the initial gap, smoothness, step size, and gradient-noise variance.

B Table of discoveries for the HIV dataset

The table lists drug-resistance mutations discovered by a deep knockoff machine and compares them with discoveries from second-order and fixed-X knockoffs, alongside annotations of existing knowledge.

  • Comparison table: Table 1 compares drug-resistance mutations selected by deep, second-order, and fixed-X knockoff methods.Annotations indicate existing knowledge about each mutation’s importance.
Loading 1811.06687v1…