Source-linked AI summary
Visualizing the Feature Importance for Black Box Models
Giuseppe Casalicchio, Christoph Molnar, Bernd Bischl
TL;DR
Black-box models need model-agnostic methods that clarify feature contributions and predictive performance. The paper introduces local PFI with PI and ICI plots, links these tools to PD and ICE, and proposes SFIMP for fair feature attribution. Averaging ICI curves yields PI curves, whose integration recovers global PFI, while SFIMP distributes importance among features and supports model comparison.
Problem
Black-box models can lack direct explanations, while feature effects and feature importance require complementary model-agnostic analysis of predictions and predictive performance.
Method
The paper introduces local PFI, PI and ICI plots, relates them to PD and ICE, and defines SFIMP using Shapley-based marginal contributions.
Results
Averaging ICI curves yields a PI curve, integrating the PI curve yields global PFI, and SFIMP fairly distributes importance among features.
Takeaways & Limitations
PI and ICI plots reveal regions of higher or lower importance and observation-level heterogeneity, while local components support conditional feature-importance analysis.
Takeaways & Limitations
The paper focuses on independent test data and leaves variability from model estimation via resampling for future investigation.
Abstract
from arXiv · showhide
In recent years, a large amount of model-agnostic methods to improve the transparency, trustability and interpretability of machine learning models have been developed. We introduce local feature importance as a local version of a recent model-agnostic global feature importance method. Based on local feature importance, we propose two visual tools: partial importance (PI) and individual conditional importance (ICI) plots which visualize how changes in a feature affect the model performance on average, as well as for individual observations. Our proposed methods are related to partial dependence (PD) and individual conditional expectation (ICE) plots, but visualize the expected (conditional) feature importance instead of the expected (conditional) prediction. Furthermore, we show that averaging ICI curves across observations yields a PI curve, and integrating the PI curve with respect to the distribution of the considered feature results in the global feature importance. Another contribution of our paper is the Shapley feature importance, which fairly distributes the overall performance of a model among the features according to the marginal contributions and which can be used to compare the feature importance across different models.
1 Introduction and Related Work
Black-box models can achieve strong predictive performance while lacking direct explanations, motivating model-agnostic tools for visualizing feature effects and importance. The paper distinguishes feature importance by its relationship to predictive performance and introduces local and global PFI-based visualizations and SFIMP.
- Black-box neural networks and SVMs model complex relationships and often outperform simpler models, but do not directly explain predictions.
- Feature-effect methods such as PD, ICE, and SHAP describe feature relationships or contributions without requiring true target values.
- Feature importance measures how important a feature was for predictive performance, regardless of the feature effect’s shape or direction, and requires true target values.
- Permutation importance measures performance loss after permuting feature values, with global model-agnostic PFI extending this idea.
- The paper introduces local PFI, PI and ICI visualizations, connects them to PD and ICE, and proposes SFIMP to fairly distribute importance across features and compare models.
2 Preliminaries and Background on Feature Effects
The paper establishes notation for fitted prediction models, test data, and generalization error, then reviews PD, ICE, and Shapley-value constructions as foundations for feature-importance analysis.
- The framework assumes an unknown relationship between features and targets, a fitted prediction model, and independently sampled test observations.
- Generalization error is the expected loss of a fitted model on unseen data and can be estimated with a test set or improved through resampling.
- PD Plots: PD plots show the marginal relationship between selected features and expected model prediction by averaging over complementary features.
- ICE Plots: ICE plots retain one curve per observation to expose feature interactions that averaging in PD plots can obscure.
- Shapley Value: A Shapley value fairly assigns each feature a contribution by averaging its marginal contribution across all coalitions or feature orderings.
3 Permutation-based Feature Importance
Permutation-based feature importance quantifies the performance change caused by perturbing selected features, with Monte-Carlo approximations reducing computational cost.
- PFI is the increase in expected loss when selected feature values are independently replicated or permuted relative to the unperturbed model.
- Larger difference-based PFI values indicate greater feature-set importance, while ratio-based PFI can improve cross-model comparability but become unstable near zero error.
- The perturbed-error term represents expected generalization error after replacing selected features, while complementary features remain unperturbed.
- Monte-Carlo integration estimates permutation-based error using m randomly selected permutations instead of all n! permutations.
- The resulting approximation repeatedly permutes selected features and averages the model performances, matching random-forest permutation importance under corresponding substitutions.
4 Visualizing Global and Local Feature Importance
The paper introduces PI and ICI visualizations to show how feature-value changes affect model performance globally and for individual observations. Averaging ICI curves gives PI, while integrating PI recovers global PFI, connecting local and global importance.
- Individual conditional importance (ICI): Individual conditional importance (ICI) plots show performance changes for each observation when feature values are replaced.The local feature importance of an observation is obtained by integrating its ICI curve over the feature distribution.
- Partial importance (PI): Partial importance (PI) is the expected performance change at a specific feature value and equals the pointwise average of ICI curves.PI curves therefore summarize conditional feature importance across observations at fixed grid points.
- Relationship to global importance: Averaging ICI curves pointwise yields a PI curve, and integrating the PI curve produces an equivalent estimate of global permutation feature importance.The paper also describes Monte-Carlo integration as an efficient approximation using sampled rather than exhaustive summands.
- Interpretation: PI curves reveal regions where feature importance differs from its global PFI, whereas ICI curves expose influential observations and heterogeneity hidden by averages.The paper notes that ICI curves can identify observations strongly affecting global PFI and can support omitting observations with negative curve integrals.
- Implementation: The proposed procedure can produce PI and PD plots, and can return ICI and ICE plots using all feature values or a random grid sample.Randomly selected grid points may skip some feature values in the ICI approximation.
5 Shapley Feature Importance
SFIMP fairly distributes the performance difference between using all features and ignoring all features according to marginal contributions. It adapts Shapley attribution to an already fitted model by marginalizing omitted features, supports efficient approximation, and distributes interaction importance among participating features.
- SFIMP definition: SFIMP fairly distributes the performance difference between the all-features and no-features scenarios among individual features.The measure is designed to visualize and interpret each feature’s contribution to model performance.
- Characteristic function: Unlike feature-selection Shapley attribution, SFIMP marginalizes over omitted features rather than refitting models with features removed.This preserves the setting of measuring importance for an already fitted model.
- Shapley attribution: A feature’s marginal contribution is the change in characteristic-function value after adding it to a coalition, averaged across feature orderings.For an ordering π, Bj(π) contains the features preceding feature j.
- Shapley properties: SFIMP values satisfy efficiency, symmetry, the dummy property, and additivity.Efficiency means all SFIMP values sum to the performance difference between using all features and ignoring all features.
- Approximation: The proposed algorithm approximates SFIMP because exact computation becomes expensive as the number of features p increases.The paper describes a computationally efficient procedure for approximating the Shapley values.
- Interpretation: SFIMP distributes interaction importance among interacting features, whereas PFI can attribute the same interaction fully to every involved feature, enabling comparisons across models.The comparison follows from how each measure handles omitted or marginalized features.
6 Simulations and Application
The simulations and application demonstrate how conditional PI/ICI analyses expose interactions and local variation that unconditional PFI can obscure, while SFIMP fairly allocates interaction importance across features.
- PI and ICI simulations: Averaging local feature importances yields global PFI and enables conditional importance calculations without additional time-consuming computations.The already computed local importances can be averaged over subsets defined by other feature values.
- PI and ICI simulations: Conditional PFI changes the feature ranking relative to unconditional PFI, helping reveal interactions among features.X1 and X2 are almost equally important unconditionally, but conditioning on X3 produces a different ranking.
- Shapley feature importance: The simulation uses four models, 100-observation test sets, and 500 repetitions to compare SFIMP with difference-based and ratio-based PFI.The models include an SVM, random forest, linear model, and a model with 2-way interaction effects.
- Shapley feature importance: Difference-based PFI gives X1 and X2 median importance ratios around 2 relative to X3, whereas SFIMP gives around 1.5 by fairly distributing their shared interaction.The difference-based measure counts the X1–X2 interaction in both feature importances, while SFIMP distributes that interaction between them.
- Application on real data: For Boston housing, PI and ICI plots identify feature-value regions and individual observations with high or low contributions to global PFI.The application uses a random forest and focuses on the two most important features, LSTAT and RM.
- Application on real data: For LSTAT, excluding observations with LSTAT ≥10 lowers conditional PFI, whereas removing 18 observations with negative ICI integrals increases it.The PI plot suggests LSTAT is more important when LSTAT < 10; the negative-integral observations have a negative impact on global PFI.
7 Conclusion and Future Work
The paper presents PI and ICI curves as model-agnostic tools for examining feature importance globally, locally, and conditionally. It also identifies resampling-based analysis of model-estimation variability as future work.
- Conclusion: PI and ICI curves disaggregate global PFI into local components, enabling feature-importance analysis across individual observations and groups.Subsetting observations by other feature values supports conditional comparisons that may reveal interactions.
- Conclusion: The tools visualize how features affect model performance and can compare feature importance across groups defined by other feature values.The paper also proposes aggregating local importances across features to measure the importance of individual observations as future work.
- Future work: The presented analysis focuses on independent test data, while variability from estimating the model itself could be studied through resampling.Resampling could support plotting or aggregating the resulting quantities.