PaperlayerView exact revision on arXiv

Source-linked AI summary

A general framework for inference on algorithm-agnostic variable importance

Brian D. Williamson, Peter B. Gilbert, Noah R. Simon, Marco Carone

arXiv:2004.03683v2stat.MEmath.STstat.MLstat.TH

TL;DR

Algorithm-specific importance may not capture features’ population-level predictive potential. This paper defines algorithm-agnostic importance through oracle predictiveness contrasts and develops nonparametric estimation and inference. Its cross-fitted estimator has good operating characteristics in numerical experiments, with results matching theoretical expectations.

  • Problem

    Algorithm-specific variable importance describes a fixed algorithm’s use of features, but does not necessarily measure their algorithm-agnostic predictive potential.

    Source: §1 · p. 1

  • Method

    The framework defines importance as a contrast in oracle predictiveness and uses nonparametric estimation and inference, including machine-learning-based estimation and a zero-importance test.

    Source: §1 · pp. 2–3, §3.2 · p. 8

  • Results

    The cross-fitted VIM estimator had good operating characteristics in numerical experiments, matching theoretical expectations.

    Source: §7 · p. 27

  • Takeaways & Limitations

    The framework supports inference on population-level variable importance across predictiveness measures, including complex measures and groups of variables.

    Source: §1 · p. 3, §7 · p. 27

    When features may have zero importance, inference should generally use sample-splitting to obtain valid confidence intervals and test the zero-importance hypothesis.

    Source: §7 · p. 29

Abstract

from arXiv · show

In many applications, it is of interest to assess the relative contribution of features (or subsets of features) toward the goal of predicting a response -- in other words, to gauge the variable importance of features. Most recent work on variable importance assessment has focused on describing the importance of features within the confines of a given prediction algorithm. However, such assessment does not necessarily characterize the prediction potential of features, and may provide a misleading reflection of the intrinsic value of these features. To address this limitation, we propose a general framework for nonparametric inference on interpretable algorithm-agnostic variable importance. We define variable importance as a population-level contrast between the oracle predictiveness of all available features versus all features except those under consideration. We propose a nonparametric efficient estimation procedure that allows the construction of valid confidence intervals, even when machine learning techniques are used. We also outline a valid strategy for testing the null importance hypothesis. Through simulations, we show that our proposal has good operating characteristics, and we illustrate its use with data from a study of an antibody against HIV-1 infection.

1 Introduction

Algorithm-specific importance can miss a feature’s population-level predictive potential, motivating an algorithm-agnostic measure. The paper defines importance as a contrast in oracle predictiveness and develops inference tools for flexible prediction settings.

1 Introduction

  • Algorithm-specific importance characterizes how a fixed algorithm uses features, not their algorithm-agnostic relevance for prediction.A predictive feature may be deemed unimportant if it plays only a minimal role in that algorithm.Source: §1 · pp. 1–2
  • Simple population models can yield misleading intrinsic-importance estimates, while broad frameworks for algorithm-independent inference using machine learning have received limited attention.The introduction contrasts simple-model measures with algorithm-specific measures and recent algorithm-independent proposals.Source: §1 · p. 2
  • The paper defines variable importance as the population-level contrast in oracle predictiveness with all features versus with the features under consideration excluded.Predictiveness can be chosen to suit the task, and the definition does not depend on a particular prediction algorithm.Source: §1 · pp. 2–3
  • The framework extends earlier R2-based work to general predictiveness measures, complex settings, and groups of variables, with machine learning incorporated into efficient estimation and valid inference.It also proposes a strategy for testing the zero-importance hypothesis.Source: §1 · p. 3
  • The paper develops an asymptotically efficient estimator and valid zero-importance test, covering nonparametric extensions of measures including AUC and classification accuracy.The article also examines complex predictiveness measures, simulations, and HIV-1 viral protein features related to antibody neutralization resistance.Source: §1 · pp. 3–4

2 Variable importance

Variable importance is defined by comparing the best population-level predictiveness using all covariates with the best predictiveness when a subgroup is excluded. The paper gives data notation and examples of predictiveness measures that fit this definition.

2.1 Data structure and notation

  • The data are independent observations of covariates and an outcome; the subgroup s indexes features whose exclusion defines the restricted function class F_s.F_s contains functions that ignore the covariates indexed by s.Source: §2.1 · p. 4

2.2 Oracle predictiveness and variable importance

  • The importance of X_s is the oracle predictiveness lost when its features are excluded, and this contrast is nonnegative.The remaining oracle predictiveness after exclusion is optimized over functions that ignore X_s; whether the loss is meaningful depends on context.Source: §2.2 · p. 5

2.3 Examples of predictiveness measures

  • R2, deviance, classification accuracy, and AUC are examples of predictiveness measures in the framework, with oracle functions linked to conditional means or their transformations.For R2 and binary-outcome deviance, the conditional mean is the optimizer when it belongs to a sufficiently rich function class; the Bayes classifier optimizes accuracy, while the conditional mean optimizes AUC when it belongs to F.Source: §2.3 · pp. 6–7

3 Estimation and inference

This section develops plug-in estimation of population variable importance and conditions for efficient inference, including how cross-fitting addresses learner-complexity restrictions. It also describes testing the zero-importance null using confidence intervals, with sample-splitting and the power trade-off between one- and two-sided intervals.

3.1 Plug-in estimation

  • Plug-in VIM estimates contrast empirical predictiveness from full-feature and reduced-feature models; flexible learning can limit misspecification bias, but naive plug-in estimators may fail to achieve root-n consistency.Williamson et al. (2020) noted that the R2 VIM plug-in estimator needs no debiasing; the authors show this also holds for a large class of VIMs.Source: §3.1 · pp. 7–8

3.2 Large-sample properties

  • Under stated regularity and rate conditions, plug-in predictiveness estimates are asymptotically linear and, with (A3)–(A4), nonparametrically efficient; this result extends to VIM contrasts.The resulting VIM estimator has an influence function given by the difference between the full-feature and reduced-feature predictiveness influence functions.Source: §3.2 · p. 10

3.3 Implementation based on cross-fitting

  • Cross-fitting yields asymptotically linear, efficient estimates under weaker conditions, eliminating the Donsker-class restriction on the complexity of machine-learning estimators.The cross-fitted result requires adapted rate and convergence conditions; the stated minimum rate is oP(n^-1/4).Source: §3.3 · p. 12

3.4 Inference under the zero-importance null hypothesis

  • At zero importance, the VIM influence function degenerates; estimating full and reduced predictiveness on distinct data subsets restores nondegenerate asymptotics and enables a valid Wald test, except when all covariates have null predictiveness.Sample-splitting supports strict-null and threshold β-null testing, whereas cross-fitting removes Donsker restrictions; the two procedures can be used together.Source: §3.4 · pp. 13–14
  • Rejecting the zero-importance null is equivalent to checking whether zero lies in the one-sided confidence interval.The interval contains the true importance with probability at least 1 − α asymptotically, even when the true importance is zero.Source: §3.4 · p. 15
  • A two-sided confidence interval gives lower and upper bounds, but reduces testing power because zero lies at the parameter-space boundary.Source: §3.4 · p. 15

4 Extensions to more complex settings

For complex predictiveness measures, the ordinary plug-in approach may fail, motivating debiasing for valid inference. Under a regularity condition, the efficient influence function can be derived while treating the optimal prediction function as fixed; the section illustrates the framework in causal and missing-data settings.

4 Extensions to more complex settings

  • Complex predictiveness measures may be undefined on empirical distributions or fail the conditions needed for simple plug-in inference, making debiasing necessary.This can arise when identification expresses predictiveness through more than population averaging, such as through an outcome regression.Source: §4 · pp. 16–17
  • In the treatment-rule example, observed-data predictiveness identifies mean counterfactual outcome under standard causal identification conditions, although its outcome regression is ill-defined on empirical distributions unless covariates are discrete.The framework still applies to this setting, where the prediction function is a treatment rule rather than an outcome prediction.Source: §4 · pp. 16–17
  • For complex measures, a one-step estimator using the nonparametric EIF is efficient under regularity conditions, while cross-fitting can improve performance with flexible learners.The simple plug-in estimator can have excessive bias after flexible learning, which motivates debiasing.Source: §4 · pp. 17–18
  • Under condition (A5), the parameter using the optimal prediction function has the same efficient influence function as the parameter with that function fixed at its true value.This can simplify EIF derivation by avoiding the need to account for changes in the optimizer.Source: §4 · p. 18
  • For classification accuracy with missing outcomes, identification requires missingness at random given X and a positive probability of observing outcomes across covariate values.The section gives the unrestricted and feature-restricted optimal classifiers in terms of the conditional probability of a correct classification.Source: §4 · pp. 19–20

5 Numerical experiments

Simulations assessed the estimator and inference procedures across nonzero- and zero-importance settings, using several learning algorithms and cross-fitting variants. Cross-fitting was particularly important with flexible learners: it supported calibrated inference where non-cross-fitted procedures could fail, while additional simulations found greater small-sample difficulty as dimensionality increased.

5.1 Simulation setup

  • The simulations used 1,000 probit-generated datasets per scenario, with n from 100 to 4,000, and evaluated accuracy- and AUC-based VIMs using several learners.Scenario 1 had two nonzero features; Scenario 2 had two nonzero and two zero-importance features.Source: §5.1 · pp. 21–22

5.2 Primary empirical results

  • For nonzero-importance X2, bias decreased faster than n^-1/2 for all cross-fitted estimators, while non-cross-fitted random forests and Super Learner failed to show this pattern and had coverage that did not approach nominal.With correctly specified parametric regression, cross-fitted and non-cross-fitted procedures performed similarly well.Source: §5.2 · pp. 22, 25, Figure 2 · p. 23
  • Cross-fitted procedures controlled type I error at the nominal level for zero-importance X3, whereas non-cross-fitted tests had inflated type I error.The Figure 3 results also report near-nominal interval coverage across sample sizes.Source: §5.2 · p. 23, Figure 3 · p. 24

5.3 Additional empirical results

  • In additional simulations, many independent or correlated null features increased small-sample bias and reduced coverage for non-null VIMs, but these effects dissipated as sample size increased.Correlation did not exacerbate the challenge beyond making population VIM interpretation more difficult.Source: §5.3 · p. 25

6 Studying an antibody against HIV-1 infection

The HIV-1 analysis compared feature-group importance for predicting VRC01 neutralization sensitivity using several variable-importance measures. The same five groups ranked highest across measures, although importance magnitudes and p-values depended strongly on the measure; these groups may be prioritized for forthcoming trial analyses.

6 Studying an antibody against HIV-1 infection

  • The CD4 binding sites, VRC01 binding footprint, ESA sites, co-varying sites, and glycosylation sites were the five highest-ranked groups across the examined VIMs.The analysis used a Super Learner and sample-split cross-fitted inference, comparing classification accuracy- and AUC-based VIMs with the prior outcome definition.Source: §6 · pp. 26–27
  • Importance magnitudes and p-values depended greatly on the VIM, even though the top-ranked feature groups differed little across measures.For the AUC-based strict-null tests, the five groups had p-values from 6.98 × 10^-9 to 8.07 × 10^-6.Source: §6 · p. 27
  • The five groups may be prioritized for forthcoming AMP trial analyses, while the authors recommend examining all feature sets in secondary or supporting analyses.Using a minimum threshold among top-ranked features may help narrow the sequence features to pre-specify.Source: §6 · p. 27

7 Discussion

The discussion presents a model-agnostic framework for population-level variable importance and reports theoretical and numerical support for its estimators. It also highlights interpretive and inferential boundaries, particularly for correlated features and zero-importance testing.

7 Discussion

  • The framework defines population-level importance as a contrast in oracle predictiveness, with plug-in estimators efficient under regularity conditions and cross-fitted estimators showing good operating characteristics in numerical experiments.The discussion also notes that more complex predictiveness measures and sampling settings, including missing data, may require additional work such as deriving an influence function.Source: §7 · p. 27
  • Figure 4 reports importance by accuracy and AUC for feature groups, with stars marking values significantly different from zero at the 0.0038 level.The threshold is 0.05 / 13.Source: Figure 4 · p. 28
  • Interpretation may focus on ranked importance values or on features whose values exceed a scientifically meaningful threshold.Source: §7 · p. 28
  • When features are strongly correlated, individual importance is challenging to define; grouping correlated variables or using a suitable causal model are suggested approaches.Clustering variables may be useful when background knowledge is lacking, but preserving inferential validity with such procedures needs further study.Source: §7 · p. 28
  • For features with potentially zero importance, sample-splitting supports valid confidence intervals and zero-importance tests, but does not fully exploit available data; a more powerful test remains an open need.The discussion notes that inference following Theorems 1 and 2 applies when non-zero importance is known a priori.Source: §7 · p. 29

Software and supplementary material

The authors provide R and Python implementations, supplementary technical details, and resources for reproducing the results and accessing the study data.

Software and supplementary material

Appendix A Special case: standardized V-measures

This appendix describes standardized V-measures as a class of predictiveness measures and explains why their plug-in estimators and influence functions have useful explicit forms.

Appendix A Special case: standardized V-measures

  • A standardized V-measure combines a prediction-dependent V1 term with a V2 normalization term, and its plug-in estimator of V1 is a degree-m V-statistic.The denominator V2 does not depend on the prediction function and typically serves as a normalization constant.Source: Appendix A · p. 30
  • The appendix lists four standardized V-measure examples, specifying their functions G, constants, normalizers, and degrees m.Source: Appendix A · p. 30
  • For standardized V-measures, the influence function of the plug-in predictiveness estimator can be written explicitly, with example-specific forms given for the measures in Section 2.3.Source: Appendix A · p. 30

8 Proof of theorems

The appendix proofs establish the stated asymptotic and differentiability results by decomposing estimator errors and analyzing perturbations of the data-generating distribution.

8.1 Proof of Theorem 1

  • Theorem 1’s proof decomposes the estimator error and uses the functional delta method and remainder bounds to establish asymptotic efficiency under the stated conditions.Source: §8.1 · pp. 35, 37

8.2 Proof of Theorem 2

  • Theorem 2’s proof uses fold-specific estimators and controls their remainder terms under the stated conditions, while its efficiency argument follows the proof of Theorem 1.Source: §8.2 · pp. 37–38

8.3 Proof of Theorem 3

  • Theorem 3’s proof considers regular parametric perturbations and establishes pathwise differentiability with a nonparametric efficient influence function.Source: §8.3 · pp. 38–39

9 Explicit description of estimation procedure for Examples 1–4

The section gives plug-in and cross-fitted estimators for the paper’s predictiveness-based variable-importance examples. It covers R², deviance, classification accuracy, and AUC.

Estimation procedures for Examples 1–4

  • For R² and deviance importance, the estimators contrast predictiveness using all features with predictiveness using the reduced feature set, with fold-specific versions combined across folds.The cross-fitted construction estimates the prediction functions outside each fold and evaluates them on that fold.Source: §9 · pp. 39–40

Example 3: classification accuracy

  • Classification-accuracy importance uses the difference between the two prediction rules’ empirical accuracy, with rules formed by thresholding estimated conditional means at 0.5.The section also gives a fold-specific version for cross-fitting.Source: §9 · p. 40

Example 4: area under the ROC curve

  • AUC importance is estimated as the difference in AUC predictiveness for the full and reduced feature sets, with a fold-specific form for cross-fitting.The supplied passages identify the estimator but do not provide its complete displayed formula.Source: §9 · pp. 40–41

10 Additional technical details

These technical details establish the optimal prediction functions for accuracy and AUC, verify conditions for the estimators, and derive influence functions for two further examples.

10.1 Bayes classifier maximizes classification accuracy

  • The Bayes classifier, which predicts 1 when the conditional mean exceeds 1/2, maximizes population classification accuracy.The proof compares any candidate classifier’s accuracy with the Bayes classifier’s and shows the difference is nonpositive.Source: §10.1 · p. 41

10.2 Conditional mean maximizes the area under the ROC curve

  • The conditional mean maximizes AUC because it is an optimal score at every fixed specificity level.The passage grounds this result in the Neyman–Pearson lemma and the likelihood ratio.Source: §10.2 · p. 42

10.3 Verification of conditions (A1) and (A2) for Examples 1–4

  • The verification derives the required conditions for examples including accuracy and AUC, with the AUC argument relying on a margin condition.For the deviance example, the stated conditions require predictions bounded away from 0 and 1; the R² argument uses bounded support for Y.Source: §10.3 · pp. 42–45

10.4 Derivation of the EIFs for Examples 5 and 6

  • For outcome-missingness classification accuracy, the section derives the observed-data influence function by relating it to the ideal-data form and defining a conditional nuisance function.The binary-intervention example’s nonparametric influence function is referenced to Luedtke and van der Laan (2016).Source: §10.4 · p. 45

11 Additional numerical experiments

The additional experiments examine cross-fitting, bootstrap intervals, and performance as dimension and feature correlation increase. They report that small samples are more affected by noise covariates, while correlated-feature groups can help in smaller samples.

11.1 Replicating all numerical experiments

  • The experiments use 1,000 simulated datasets over increasing sample sizes and specify individual algorithms and Super Learner candidates, including boosted trees, random forests, and lasso.The higher-dimensional experiments use p ∈ {50, 100, 200} and assess both individual features and groups.Source: §11.2 · p. 46, §11.1 · p. 46, Table 3 · p. 47

11.2 Properties of our proposal under the alternative hypothesis

  • With flexible nuisance estimators, cross-fitting is important for practical inference; with simple correctly specified parametric estimators, its impact appears minimal.The supplemental alternative-hypothesis experiments report near-zero coverage for non-cross-fitted random-forest intervals in one setting.Source: §11.2 · p. 48, Figure 5 · p. 48, Figure 7 · p. 50

11.3 Properties of our proposal under the null hypothesis

  • For the null-feature experiments, cross-fitted random forests showed residual bias but near-nominal coverage and type I error with cross-fitted standard errors; non-cross-fitted standard errors could be too small.The comparison reports inflated type I error for the cross-fitted random-forest estimator when paired with non-cross-fitted standard errors.Source: §11.3 · p. 52, Figure 10 · p. 54, Figure 11 · p. 55

11.4 Using the bootstrap for interval estimation

  • The bootstrap fixes nuisance-function estimates across resamples and gives coverage and interval widths similar to influence-function intervals, with slightly narrower intervals in smaller samples.The procedure bootstraps the empirical distribution without refitting nuisance estimators or resampling cross-fitting.Source: §11.4 · pp. 52, 56, 57

11.5 Higher dimensions and correlated features

  • In higher dimensions, n = 500 shows increasing bias and decreasing coverage as p grows, whereas n = 3000 recovers similar performance and nominal type I error control.Under correlated features, groups can outperform individual features in smaller samples, with the performance difference diminishing as sample size grows.Source: §11.5 · pp. 61, 63, Figure 17 · p. 61, Figure 18 · p. 62

12 Additional details for the study of an antibody against HIV-1

This appendix reports a harmonized variable-importance analysis and details the Super Learner library and performance across two HIV-1 neutralization outcomes.

12.1 Harmonized analysis with Magaret et al. (2019)

  • In the harmonized analysis, only the VRC01 binding footprint had importance significantly different from zero after correction, and only for AUC (p = 6.1 × 10−4).The correction threshold was 0.0038 for testing 13 groups.Source: §12.1 · p. 65, Figure 21 · p. 66

12.2 Library of candidate learning algorithms

  • The candidate library included lasso, random forests, and boosted trees, and the final estimator combined learners to minimize ten-fold cross-validated negative log likelihood.The analysis also adjusted for geographic region as a potential confounding variable.Source: §12.2 · pp. 65–66

12.3 Super Learner performance

  • Boosted trees and random forests were commonly selected for both outcomes, while elastic net was often not selected for IC50 < 1.The reported Super Learner weights vary across outcomes and cross-validation folds.Source: §12.3 · p. 66, Table 6 · p. 67
  • Cross-validated AUC was 0.90 (95% CI 0.87, 0.94) for IC50 censored and 0.83 (0.80, 0.86) for IC50 < 1.Random forests performed best among the individual algorithms for both outcomes.Source: Figure 22 · p. 68, §12.3 · p. 68
  • The Super Learner, discrete Super Learner, and random forests had similar ROC curves, with high true-positive rates at chosen false-positive rates.The authors say these results suggest the predictor is well-calibrated for discriminating between outcome classes.Source: §12.3 · p. 68, Figure 23 · p. 69

Related papers

Loading 2004.03683v2…