PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach

Christoph Molnar, Gunnar König, Bernd Bischl, Giuseppe Casalicchio

arXiv:2006.04628v2stat.MLcs.LG

TL;DR

Dependent features can make feature-importance and effect interpretations misleading through extrapolation, while conditional sampling requires transparent conditioning. The paper uses decision-tree-defined subgroups for conditional permutations and reports that its methods often match or outperform alternatives across dependence settings. The subgroup descriptions also support interpretation of feature effects and importance alongside dependence structure.

  • Problem

    Marginal PDP and PFI ignore feature dependence, which can lead to low-density extrapolation and misleading interpretations; existing conditional samplers also lack readily interpretable conditioning.

    Source: §1 · p. 2

  • Method

    The paper computes PDP and PFI within decision-tree-defined subgroups using conditional subgroup permutations, producing interpretable conditional estimates and local results.

    Source: §1 · p. 3, §11 · p. 26

  • Results

    Across various dependence settings, cs-PFIs and cs-PDPs are on par with or outperform other methods in many settings.

    Source: §11 · p. 26

  • Takeaways & Limitations

    Subgroup descriptions combined with cs-PFIs and cs-PDPs can reveal insights into the model and the data's dependence structure.

    Source: §1 · p. 4

    The framework assumes dependence can be modeled with discrete groups, which may only approximate some relationships such as simple linear Gaussian dependencies; evaluating alternative tree and decision-rule approaches remains future work.

    Source: §4.1.1 · p. 10, §11 · p. 27

Abstract

from arXiv · show

The interpretation of feature importance in machine learning models is challenging when features are dependent. Permutation feature importance (PFI) ignores such dependencies, which can cause misleading interpretations due to extrapolation. A possible remedy is more advanced conditional PFI approaches that enable the assessment of feature importance conditional on all other features. Due to this shift in perspective and in order to enable correct interpretations, it is therefore important that the conditioning is transparent and humanly comprehensible. In this paper, we propose a new sampling mechanism for the conditional distribution based on permutations in conditional subgroups. As these subgroups are constructed using decision trees (transformation trees), the conditioning becomes inherently interpretable. This not only provides a simple and effective estimator of conditional PFI, but also local PFI estimates within the subgroups. In addition, we apply the conditional subgroups approach to partial dependence plots (PDP), a popular method for describing feature effects that can also suffer from extrapolation when features are dependent and interactions are present in the model. We show that PFI and PDP based on conditional subgroups often outperform methods such as conditional PFI based on knockoffs, or accumulated local effect plots. Furthermore, our approach allows for a more fine-grained interpretation of feature effects and importance within the conditional subgroups.

1 Introduction

When features are dependent, marginal PDP and PFI can extrapolate into low-density regions, while conditional versions avoid that issue but require interpretation relative to other features. The paper proposes decision-tree-based conditional subgroups to make that conditioning interpretable and support local effect and importance analysis.

1 Introduction

  • Dependent features can make marginal PDP and PFI perturbations extrapolate into low-density regions, misleading interpretation for non-additive models.The conditional variants sample a feature given the others, preserving the joint distribution, but their interpretation is conditional on those features.Source: §1 · p. 2
  • Conditional PFI measures a feature's additional, unique contribution given the conditioned features, so a lower value can reflect information shared with a dependent feature rather than irrelevance.The paper cautions that conditional PFI should not be read as the feature's overall relevance to the model.Source: §1 · p. 3
  • Conditional PDP can mix a feature's effect with those of dependent features, showing an effect for a feature unused by the model or even one opposite its coefficient.The paper relates this pitfall to omitted variable bias.Source: §1 · p. 3
  • Existing conditional samplers such as knockoffs do not make their conditioning readily interpretable, motivating transparent explanations of feature dependencies.The proposed tree structure indicates how other features influence the effect and importance estimates.Source: §1 · p. 3
  • Decision-tree-based conditional subgroup PDPs and PFIs use subgroup permutations to handle extrapolation while enabling local interpretation of feature effects and importance.Trees are trained so the feature of interest becomes less dependent on other features within the resulting subgroups.Source: §1 · p. 3
  • In the illustrated simulation, the conditional PDP for x2 is positive despite its negative coefficient because it mixes effects with correlated x1.The example uses correlation 0.978 between x1 and x2; conditional PFI for x1 and x2 decreases when each is permuted conditional on the other.Source: §1 · p. 3, Figure 2 · p. 4
  • Subgroup descriptions combined with cs-PFIs and cs-PDPs can support insights into both the model and the data's dependence structure.The paper also describes a trade-off between subgroup human-intelligibility and extrapolation through grouping granularity.Source: §1 · p. 4

2 Notation and Background

This section establishes notation for predictions, feature values, and complementary features, then defines PDP and PFI variants. Marginal PFI uses marginal feature perturbations, whereas conditional PFI samples from the feature's conditional distribution.

2 Notation and Background

  • PFI is the increase in loss after permuting feature Xj, with repeated replacement samples available to stabilize the estimate.The passage distinguishes the original random-forest formulation from the later model-agnostic version.Source: §2 · pp. 4–5
  • A replacement feature with the same marginal distribution yields marginal PFI, while sampling Xj conditional on X−j yields conditional PFI.The notation uses xj for an observed feature vector and Xj for the corresponding random variable; X−j denotes the remaining features.Source: §2 · p. 4
  • The marginal PDP describes the average effect of feature j on the model prediction, while the conditional PDP takes an expectation given Xj.The marginal PDP is estimated using Monte Carlo integration.Source: §2 · p. 5

3 Related Work

Related work addresses dependent-feature interpretation through conditional PDP and PFI variants, local-effect plots, stratification, and alternative conditional samplers. The paper identifies interpretable dependence structure and evaluation of extrapolation avoidance as open gaps.

3.1 Related Work on Conditional PDP

  • ALE reduces extrapolation by accumulating finite differences within feature intervals, but its interpretations are local to those intervals and it has no straightforward categorical-feature approach.The proposed approach can handle categorical features.Source: §3.1 · p. 5
  • Stratified PDP describes the data independently of the model, while ICE curves visualize interactions underlying PDP but still suffer from extrapolation.In ICE, “conditional” refers to individual observations, not conditioning on selected features.Source: §3.1 · p. 5

3.2 Related Work on Conditional PFI

  • CVIRF uses individual random-forest tree splits for conditional permutation, whereas cs-PFI trains separate model-agnostic trees for each feature using the other features as predictors.The cs-PFI subgroup construction is decoupled from the prediction model being interpreted.Source: §3.2 · p. 6
  • Other conditional PFI approaches include matching or imputation, knockoffs, and model retraining, which may be costly and answer a different question from fixed-model importance.Knockoffs preserve the joint distribution but do not transparently show how they condition on features; the paper focuses on fixed models without retraining.Source: §3.2 · pp. 6–7

3 Related Work

  • The paper identifies two gaps: existing approaches do not make feature-dependence structures explicit, and sampling strategies have received little evaluation for addressing extrapolation.It presents an OpenML-CC18 data-fidelity experiment and ground-truth conditional PFI experiments as responses to those gaps.Source: §3.2 · p. 7

4 Conditional Subgroups

The paper partitions observations into interpretable subgroups where the feature of interest is less dependent on other features, then computes subgroup-wise PFI and PDPs. This reduces extrapolation and supports local interpretations, with limits for continuous dependencies and small subgroups.

4 Conditional Subgroups

  • Decision trees partition other-feature space into interpretable groups with more homogeneous distributions of the feature of interest, enabling group-wise sampling and interpretation.When the feature is independent of the others within groups, group-wise permutation removes extrapolation; even without such a grouping, it can reduce extrapolation.Source: §4 · pp. 8–9

4.1.1 Continuous Dependencies

  • For continuous dependencies, partitioning can reduce but not eliminate extrapolation; CART detects changes in conditional means but can miss changes in higher moments.Linear Gaussian dependence cannot generally be represented by the discrete grouping approach unless partitions shrink to individual observations.Source: §4 · p. 9, §4.1.1 · p. 10, Figure 4 · p. 10

4.1.2 Sparse Subgroups

  • Fewer, larger subgroups preserve enough observations for reliable within-group distributions and more human-intelligible decision rules, trading off against distributional homogeneity.Maximum tree depth controls this granularity trade-off.Source: §4.1.2 · p. 10

4.2 Conditional Subgroup Permutation Feature Importance (cs-PFI)

  • cs-PFI permutes the feature within each subgroup and returns both local subgroup importances and a global importance equal to their observation-weighted average.The subgroup assignment is based on terminal nodes of a tree trained to predict the feature from the remaining features.Source: §4.2 · p. 11
  • When the feature is independent of the others, each cs-PFI has the marginal PFI’s expectation but n/n_k times its variance; further noise splits after independence do not change its expectation.The same expectation result holds even when partitions are chosen randomly.Source: §4.2 · pp. 11–12

4.3 Conditional Subgroup Partial Dependence Plots (cs-PDPs)

  • cs-PDPs retain marginal interpretations within groups and reduce extrapolation; the authors recommend displaying group-wise curves rather than aggregating them into a conditional PDP.In the simulation, cs-PDPs separate X1-related intercept changes from X2’s zero slope, while the conditional PDP suggests X2 has an influence.Source: §4.3 · p. 12, Figure 5 · p. 13
  • When the feature is independent of the others, each cs-PDP has the marginal PDP’s expectation but n/n_k times its variance.The subgroup PDPs approximate the marginal PDP under independence.Source: §4.3 · p. 12

5 Training Conditional Sampling Approaches

The section recommends separating training from sampling to guard against overfitting. It describes how the compared conditional sampling methods use training data to construct their samplers and test data for feature replacement or permutation.

5 Training Conditional Sampling Approaches

  • Separate training from sampling: perform data-dependent estimation steps on Dtrain and compute feature importance using Dtest.The authors present this as a general recommendation intended to avoid overfitting.Source: §5 · pp. 13–14
  • For cs-permutation, train CART or transformation trees on Dtrain, assign Dtest observations to terminal nodes, and permute X_j within those nodes.For CVIRF, train a random forest on Dtrain to predict the target and permute X_j within its terminal nodes.Source: §5 · p. 14
  • The comparison also fits Model-X knockoffs, imputation, and interval-based samplers on Dtrain, while marginal PFI and PDP simply shuffle X_j in Dtest.Knockoffs replace X_j with knockoffs; imputation uses random-forest predictions plus residuals, and interval sampling moves observations to interval borders.Source: §5 · p. 14

6 Conditional PFI Ground Truth Simulation

The simulations compare conditional PFI estimates with ground truth across dependence structures and sample settings. cs-PFI performs well in most scenarios, while performance depends on the dependence structure and, for random forests, sample size.

6 Conditional PFI Ground Truth Simulation

  • Simulations varied sample size (n ∈ {300, 3000}) and feature count (p ∈ {9, 90}), comparing the true model with a 100-tree random forest; each experiment was repeated 1000 times.The true conditional distribution of X1 given the remaining features was known, enabling computation of ground-truth conditional PFI.Source: §6 · pp. 14–15

6.1 Conditional PFI Ground Truth Results

  • Transformation-tree cs-PFI generally matched or beat CART except with multiple linear dependencies; cs-PFI matched alternatives for single linear dependence and outperformed them for complex nonlinear dependence.In the nonlinear scenario, knockoffs and random-forest imputation also showed overestimation and higher variance than cs-PFI.Source: §6.1 · p. 15, Table 2 · p. 16
  • For random forests, CVIRF performed well in some small-sample settings but consistently underestimated conditional PFI at high sample sizes, including under independence.The authors recommend cs-PFI for lower-dimensional dependence structures and imputation for multiple linear dependencies.Source: §6.1 · pp. 15–16

7 Trading Interpretability for Accuracy

The section examines how subgroup count affects cs-PFI accuracy. Deeper transformation trees better approximated true conditional PFI, with no observed overfitting.

7 Trading Interpretability for Accuracy

  • In simulation scenario (I), decision trees with maximal depths from 1 to 10 were tested over 1,000 repetitions; deeper transformation trees and more subgroups better approximated true conditional PFI, with no overfitting observed.Source: §7 · pp. 16–17
  • The experiment varied maximum tree depth from 1 to 10 and examined how the resulting subgroup count affected conditional PFI estimates.The section’s Figure 7 compares approaches with the true conditional PFI as a reference.Source: §7 · p. 16, Figure 7 · p. 17

8 Data Fidelity Evaluation

The section evaluates how well conditional sampling preserves the joint feature distribution under intervention. cs-permutation achieves the highest data fidelity overall, and moderate tree depth offers a strong trade-off between fidelity and interpretability.

8 Data Fidelity Evaluation

  • Data fidelity measures how closely an intervened sample preserves the joint distribution, and the study operationalizes it as −log(MMD), with larger values indicating better fidelity.MMD compares the reference and intervened samples; the evaluation used a radial basis function kernel and a median-distance bandwidth heuristic.Source: §8 · pp. 18–19
  • The evaluation trained sampling methods on 40% of each data split, perturbed features in a separate 30%, and used an unchanged 30% reference set.It compared strategies across eligible OpenML-CC18 datasets, using marginal permutation and no perturbation as lower and upper fidelity bounds.Source: §8 · p. 19

8.1 Results A) State-of-the-art comparison

  • cs-permutation had higher data fidelity than the other methods overall and consistently high fidelity across the cited datasets; Model-X knockoffs and random-forest imputation ranked next in the model comparison.Table 3’s average rankings put knockoffs above imputation, although some datasets had very low knockoff fidelity.Source: §8.1 · pp. 20–21, Figure 9 · p. 21, Table 3 · p. 21

8.2 Results B) tree configuration

  • Tree depths of 3–4 were nearly as faithful as depth 30 and outperformed the other methods while remaining interpretable; even depth 1 improved fidelity over marginal permutation for most datasets.Depth 2 brought average data fidelity to parity with knockoffs; deeper trees generally increased fidelity.Source: §8.2 · p. 22

9 Model Fidelity

The section compares how closely feature-effect predictions approximate model predictions. cs-PDPs had the best model fidelity, with deeper subgroup trees improving fidelity in the reported analysis.

9 Model Fidelity

  • Model fidelity for feature effects is measured as mean squared error between model predictions and predictions from the feature-effect function.The evaluation compared effect predictions with model predictions on held-out observations.Source: §9 · p. 22
  • The evaluation used random forests, linear models, and k-nearest-neighbor models on regression datasets, reserving 30% of the data for measuring model fidelity.The datasets were limited to 1,000–8,000 instances, at most 50 numerical features, and no categorical features.Source: §9 · p. 22, Table 4 · p. 23
  • cs-PDPs had the best model fidelity, while ALE and standard PDP had similar fidelity; loss decreased as subgroup-tree depth increased.The study evaluated PDPs within subgroups rather than aggregating them into one conditional PDP, and limited trees to depth 2 for interpretability.Source: Table 5 · p. 23, §9 · p. 23

10 Application

The bike-rental application uses transformation-tree subgroups to examine feature dependence and compare conditional subgroup PFI and PDP interpretations with marginal approaches. The subgroup results show that temperature importance and feature effects vary by season and humidity.

10 Application

  • The analysis trains a random forest on 731 observations and uses a 70%/30% train-test split.The predictors are weather and seasonal information.Source: §10 · p. 23

10.1 Analyzing Feature Dependence

  • A random forest predicts each feature from the others to assess dependence; work is fully predictable from weekday and holiday.Season, temperature, humidity, and weather are partially predictable and therefore not independent.Source: §10.1 · p. 24, Table 6 · p. 24

10.2 cs-PDPs and cs-PFI

  • Transformation trees are limited to depth 2, and most estimated conditional PFI change occurs between depths 0 and 2.Depth 2 limits the number of subgroups to four and is presented as a trade-off between subgroup count and approximating conditional PFI.Source: §10.2 · p. 24, Figure 10 · p. 25
  • Temperature has zero PFI on humid summer days but is important in spring, fall, and winter.Local cs-PFI conditions the subgroup interpretation on season and humidity.Source: §10.2 · pp. 24–25, Figure 11 · p. 25
  • PDP and ALE show increasing rentals up to 25 °C and decreasing rentals beyond, while the PDP's weaker high-temperature decline may reflect extrapolation.The local cs-PDPs make subgroup conditions explicit, and their intercept and slope changes distinguish grouping-feature and within-group temperature effects.Source: §10.2 · pp. 25–26, Figure 12 · p. 26
  • Local cs-PDPs show no temperature effect on humid summer days and a slightly negative effect on dry summer days.The dry-summer subgroup is defined by humidity below 70.75.Source: §10.2 · p. 26, Figure 12 · p. 26
  • Conditional on temperature, seasonal rental differences are much greater than the marginal PDP suggests, especially at low temperatures.At high temperatures, rentals are similar across seasons.Source: §10.2 · p. 26

11 Discussion

The discussion presents cs-PFIs and cs-PDPs as interpretable subgroup-based variants, while noting a boundary for many dependencies. The accompanying theory relates subgroup estimates to conditional PFI and characterizes their behavior under feature independence.

11 Discussion

  • cs-PFIs and cs-PDPs use decision-tree subgroups for feature-dependent data, and the authors report performance on par with or better than other methods in many dependence settings.The grouping is human-comprehensible and conveys how dependencies affect feature effects and importance.Source: §11 · p. 26
  • The methods do not perform well when the feature of interest depends on many other features; the authors recommend random-forest imputation in that case.They recommend analyzing the dependence structure beforehand and using cs-PFIs in other cases.Source: §11 · p. 27
  • The framework allows different partitioning choices, but evaluating the range of tree algorithms and decision rules is left to future research..Source: §11 · p. 27

A Decompose conditional PFI into cs-PFIs

  • With a perfect construction of the groups, the weighted sum of marginal subgroup PFIs coincides with conditional PFI.The appendix states the group-level independence condition for this result.Source: A Decompose conditional PFI into cs-PFIs · p. 31

B Expectation and Variance of the PFI in a Subgroup

C Expectation and Variance of the PDP in a Subgroup

D cPFI Ground Truth Scenario II

  • The appendix describes an experiment comparing conditional-PFI approaches with ground-truth conditional PFI using an intermediate random forest.The supplied passages identify the experiment's scope but report no comparative outcome.Source: D cPFI Ground Truth Scenario II · p. 33, Figure 14 · p. 34

E Data Fidelity on OpenML-CC18 data sets

The data-fidelity appendix describes OpenML-CC18 comparisons of sampling methods and tree configurations, and a separate model-fidelity comparison of feature-effect methods.

E Data Fidelity on OpenML-CC18 data sets

E.1 Data Fidelity Results

  • Data-fidelity boxplots compare unconditional and subgroup permutations, knockoffs, imputation, ALE, CVIRF, and no permutation across features and datasets.The experiments average repeated results before plotting; additional plots compare CART and transformation-tree depths against unconditional-permutation and no-permutation bounds.Source: Figure 15 · p. 36, Figure 16 · p. 37, Figure 17 · p. 38

F Model Fidelity Plots

  • Model-fidelity boxplots report MSE between the model and feature-effect methods, summed over test data for each feature..Source: Figure 19 · p. 40

Related papers

Loading 2006.04628v2…