Source-linked AI summary
Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance
Giles Hooker, Lucas Mentch, Siyu Zhou
TL;DR
The paper examines why permute-and-predict methods can mislead when features are correlated, reviewing their behavior and advocating alternatives that avoid extrapolation. Its examples and simulations identify flexible-model extrapolation as a significant error source and report upward bias for correlated covariates, while refitting-based alternatives yield more intuitive, stable, and consistent importance orderings.
Problem
Permutation-based importance treats correlated features as separately manipulatable, potentially producing misleading interpretations of black-box predictions.
Method
The paper reviews prior work, analyzes permutation diagnostics across random forests and neural networks, and advocates conditional-data generation or refitting alternatives that avoid extrapolation.
Results
The examples show misleading permutation results, upward-biased estimates for correlated covariates, and feature-selection orderings that can retain weakly predictive correlated variables while discarding stronger predictors.
Takeaways & Limitations
Refitting-based alternatives can produce more intuitive, stable, and consistent importance orderings, but permutation diagnostics should not be used uncritically with correlated features.
Takeaways & Limitations
Correcting variable-importance orderings may not be sufficient for using variable importances within hypothesis tests.
Abstract
from arXiv · showhide
This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions. Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because they are both model-agnostic and depend only on the pre-trained model output, making them computationally efficient and widely available in software. However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features. The purpose of our work here is to (i) review this growing body of literature, (ii) provide further demonstrations of these drawbacks along with a detailed explanation as to why they occur, and (iii) advocate for alternative measures that involve additional modeling. In particular, we describe how breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data. We explore these effects across various model setups and find support for previous claims in the literature that PaP metrics can vastly over-emphasize correlated features in both variable importance measures and partial dependence plots. As an alternative, we discuss and recommend more direct approaches that involve measuring the change in model performance after muting the effects of the features under investigation.
1 Introduction
The paper reviews popular permute-and-predict diagnostics and argues that feature dependence can make them misleading by forcing models to extrapolate into unrealistic regions. It advocates conditional perturbation or re-learning approaches that avoid this problem but require additional modeling effort.
- Permute-and-predict methods are popular because they are computationally cheap, model-agnostic, tuning-free, stable, and widely implemented.They require O(N) operations and use only the fitted prediction function.
- Permutation importance changes one feature independently, ignoring dependence with correlated features and potentially creating unrealistic feature combinations.The paper distinguishes this inherent choice of importance target from estimation bias caused by applying the measure to a fitted model.
- Breaking feature dependence evaluates fitted models in sparse regions where little or no training data exist, causing extrapolation-related bias.The paper identifies this as especially important when flexible learners are used with correlated features.
- For linear models with standardized covariates, permutation importance equals the squared coefficient, regardless of feature correlation.This corresponds to the interpretation of changing one feature while holding all others fixed.
- In experiments, flexible-model estimates were biased upward for correlated covariates, potentially retaining redundant correlated features while discarding stronger predictors.The bias increased with correlation and could distort backward-elimination variable selection.
- The paper advocates conditional perturbation and re-learning as alternatives that avoid extrapolation, while noting that both require an additional model or modeling effort.Its preferred strategy combines conditioning on remaining features with rebuilding the model.
2 A Simple Simulated Example
The simulated example compares variable-importance diagnostics across linear models, random forests, and neural networks under independent and correlated feature distributions. It is designed to isolate how flexible learners and feature correlation jointly produce misleading importance rankings.
- The simulation uses a 10-feature linear regression with additive signal in x1 through x5 and no influence from x6.The response includes small Gaussian process noise, allowing diagnostic behavior to be compared with known coefficients.
- Each feature is marginally uniform on [0, 1], while x1 and x2 are correlated through a Gaussian copula with parameter ρ.The remaining features are generated independently, making correlation strength the central distributional manipulation.
- The design uses x3, x4, and x5 as equal-coefficient reference features, with x6 providing a zero-effect reference and x1 and x2 assigned distinct relative importance.This establishes expected importance ordering under the known generating model.
- Large bias requires both correlated features and flexible learning methods; neither feature correlation nor flexibility alone is sufficient to induce large bias.The paper presents this combination as common in practice.
- With ρ = 0, random forests, neural networks, and linear models agree on the covariate ordering, including ties between x1 and x5.The result matches the ordering predicted by the linear-model theorem.
- The experiments fit random forests and single-layer neural networks, evaluate training-data permutation importance, and average diagnostic results over repeated simulations.The setup also records random-forest out-of-bag importance, partial dependence, and ICE plots.
3 Simulation Results
Simulations show that strong feature correlation can distort permutation importance and partial-dependence diagnostics, especially for random forests and neural networks. These distortions vary with sample size and appear as increased neural-network variability and misleading importance rankings.
- Variable importance: At ρ = 0, random forests, neural networks, and linear models agree on covariate ordering, including ties among x1-x5.The simulations use 2,000 observations across 50 replications.
- Variable importance: With ρ = 0.9, permutation methods rank x1 and x2 above (x3, x4, x5) and often above x9, despite x9 having a larger coefficient.Linear models retain the same importance ordering across correlation structures, consistent with the known response coefficients.
- Sample size and correlation: For small N, nonzero correlation makes x1 and x2 appear more important than (x3, x4, x5), particularly under out-of-bag measures.For small ρ, this effect disappears at larger N; for large ρ, the rank declines with N but never below 5, the theoretical value.
- Sample size and correlation: Figure 2 examines average importance ranks as correlation and sample size vary, with the theoretical rank set to 4.The remaining panels report average importance versus N for fandom forests, neural networks, and random forests using OOB measures.
- Partial dependence and ICE: When ρ = 0.9, neural-network partial dependence becomes more variable, and its ICE curves diverge considerably from the underlying relationship.Random-forest partial dependence shows less attenuation from edge effects under correlation, but remains compared with features having attenuated edge effects.
4 Extrapolation and Explanations
The paper explains misleading permute-and-predict results as extrapolation caused by breaking dependence among correlated features. Permuted queries enter sparse regions where tree and neural-network predictions can depart from the underlying relationship, inflating importance and distorting interpretation plots.
- Conceptual explanation: The paper distinguishes the interpretive choice of permutation summary from the separate extrapolation problem caused by evaluating correlated features at unrealistic combinations.It identifies extrapolation as the explanation for the second effect and notes that repeated observations of these patterns have not reduced diagnostic use.
- Random-forest behavior: In the x1 + ϵ example, random-forest contours are approximately vertical inside the data hull but become diagonal outside it.The experiment uses 200 points with correlation 0.9 and repeats the procedure 100 times.
- Extrapolation mechanism: Permutation importance evaluates points across the unit square even when correlated training data concentrate along the diagonal.Consequently, much of the evaluation occurs far from the training data, where random forests poorly mimic the data-generating function.
- Random-forest behavior: Tree predictions in extrapolative regions become local averages based on splits extending horizontally or vertically from the observed data distribution.This leaf-based behavior determines the potential nearest neighbours used in the permuted prediction.
- Extrapolation mechanism: For correlated features, permuted potential nearest neighbours can be far from the original query in both coordinates, producing a large perceived importance for an irrelevant feature.With more uniformly distributed observations, potential nearest neighbours remain localized and provide a more reasonable comparison.
- Partial dependence and ICE: ICE plots track the underlying relationship in data-supported regions but can differ substantially outside them; neural networks show greater variability, while random forests show importance inflation.The lower-dimensional setting makes the extrapolation effect more evident by reducing bias and variance.
5 Variable Importance Alternatives
The paper presents alternatives to permute-and-predict importance that preserve feature dependence or refit models, while cautioning that improved importance ordering does not guarantee valid hypothesis tests.
- Conditional importance generates feature replacements from the distribution conditional on the remaining features, preserving dependence among covariates.
- LOCO measures the accuracy drop after removing a feature and learning a new model without it.
- Refitting-based alternatives produce similar, more intuitive importance orderings and reduce the importance assigned to correlated features.For conditional measures, this follows because the conditional distribution has smaller variance than the marginal distribution; for LOCO, correlated features can account for overlapping signal.
- For linear models, the discussed measures have approximately the same expectation, but the paper cautions that these results are exact only for linear models.Jointly removing or permuting feature pairs and relearning is recommended as a more reliable diagnostic for correlated-feature effects.
- Local explanations avoid extrapolation by examining model behavior near observed data, but they generally lack a global summary of feature contribution.A threshold transition far from observed data can produce low local saliency despite a large prediction range.
- Relearning after permutation resolves some problems but does not fully eliminate permutation-based importance issues, especially for testing.Conditional replacements and MX knockoffs yielded rejection rates near the nominal 0.05 level, whereas Permute-and-Relearn often exceeded it as noise features increased.
6 Partial Dependence Alternatives
Alternatives to partial dependence and ICE plots aim to avoid extrapolation, but they require choices or specialized modeling to represent effects over dependent feature distributions.
- ICE alternatives: ICE plots can restrict displayed feature values to ranges consistent with dense regions of the data, providing local information about feature dependence.The appropriate range for each feature still must be determined.
- ICE alternatives: Averaging only dense portions of ICE lines can misattribute changes caused by other features to the feature being plotted.
- Model-based alternatives: Functional-ANOVA alternatives estimate component functions against the feature distribution, but high-dimensional remainder functions require specialized learning methods.The relevant formulation jointly estimates f1(x1) and f−1(x−1).
- Model-based alternatives: Generalized additive models provide an extrapolation-free representation of the underlying function using one-dimensional component functions.
7 Conclusions
The paper concludes that naïve permute-and-predict diagnostics can mislead because flexible models extrapolate beyond the training distribution. It recommends alternatives that preserve dependencies or relearn models, while noting computational and representational trade-offs.
- Conclusions: Permute-and-predict methods can give misleading results, with extrapolation by flexible models identified as a significant source of diagnostic error.The precise bias depends on the learner and the dependence structure of features and responses.
- Conclusions: Alternative importance measures require additional computational effort, including relearning models after permuting features.
- Conclusions: Condition-and-relearn preserves approximate dependencies among remaining features, while conditional permutation or simulation generally requires modeling xij|xi,−j.Shapley values and generalized Sobol indices can also inherit extrapolation bias if calculated without care.
- Conclusions: Local explanation methods avoid extrapolation but do not provide a global representation across the full feature-value range.
- Conclusions: Additive-model displays better represent overall function behavior than methods combining local explanations, but specialized methods remain necessary for some diagnostics.
A Proofs of Results
The proofs derive identities for permutation-based and alternative importance quantities in linear-model settings through algebraic decomposition, orthogonality, centering, and direct calculation.
- Importance quantities: Theorem 2 begins by identifying the quantity estimated directly by VIC_j, while the constant-replacement case for VID_j uses the stated conditional independence assumption.
- Proof strategy: An invertible linear transformation of the feature matrix leaves the regression equivalent, supporting the proof's invariance argument.
- Proof strategy: The proof decomposes the fitted response into contributions from the target feature and the remaining design matrix, using an intercept-inclusive feature matrix.
- Proof strategy: Orthogonality of the residual and auxiliary terms to X−j makes the relevant second term vanish, yielding the stated result by substitution.
- Permutation identities: The permutation expectation result follows from a cross-product condition, while centered variants are obtained by subtracting the fitted conditional component before repeating the calculation.
- Proof strategy: The remaining identities are established by direct calculation, with VIC_j obtained from the same calculations after observing the relevant expectation relation.