Source-linked AI summary
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Christoph Molnar, Gunnar König, Julia Herbinger, Timo Freiesleben, Susanne Dandl, Christian A. Scholbeck, Giuseppe Casalicchio, Moritz Grosse-Wentrup, Bernd Bischl
TL;DR
Model-agnostic interpretation methods can yield wrong conclusions when applied without regard to model performance, data structure, uncertainty, or causal assumptions. The paper reviews and illustrates these pitfalls, discusses available practitioner solutions, and identifies open research issues. It concludes that interpretation requires caution because causal discovery and inference remain unresolved and interpretation methods need clearer theoretical and practical guidance.
Problem
Model-agnostic interpretation techniques provide useful insights but can mislead when applied in unsuitable contexts or to poorly generalizing models.
Method
The paper reviews general pitfalls of global and local model-agnostic interpretation methods, illustrates them with examples, and discusses solutions and open issues.
Results
The paper identifies recurring risks involving context, generalization, dependencies, interactions, uncertainty, high dimensionality, and unjustified causal interpretation.
Takeaways & Limitations
Practitioners should use cautious, goal-specific interpretations and consider model, data, and causal context before drawing conclusions.
Takeaways & Limitations
Causal discovery and inference remain open issues, and further work is needed to specify assumptions supporting insights into data-generating mechanisms.
Abstract
from arXiv · showhide
An increasing number of model-agnostic interpretation techniques for machine learning (ML) models such as partial dependence plots (PDP), permutation feature importance (PFI) and Shapley values provide insightful model interpretations, but can lead to wrong conclusions if applied incorrectly. We highlight many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in high-dimensional settings, or making unjustified causal interpretations, and illustrate them with examples. We focus on pitfalls for global methods that describe the average model behavior, but many pitfalls also apply to local methods that explain individual predictions. Our paper addresses ML practitioners by raising awareness of pitfalls and identifying solutions for correct model interpretation, but also addresses ML researchers by discussing open issues for further research.
1 Department of Statistics, LMU Munich, Munich, Germany
The listed passages provide institutional affiliations, keywords, and funding information for the authors.
- The authors are affiliated with the LMU Munich, University of Vienna, and Leibniz Institute for Prevention Research and Epidemiology.
- The paper is associated with interpretable machine learning and explainable AI.
- The work reports support from Bavarian, German, Austrian, and university research institutions.
1 Introduction
Machine learning’s predictive strengths and flexible nonlinear structure increase the need for careful interpretation. This paper focuses on general pitfalls of model-agnostic methods, especially global explanations, and reviews solutions and open research issues.
- Non-parametric and nonlinear ML models have gained use partly because they outperform traditional models on many prediction tasks.Many such models also handle interactions and nonlinear effects automatically.
- The paper concentrates on model-agnostic methods and primarily on global interpretations describing average model behavior across the data distribution.Many discussed pitfalls also apply to local explanations of individual predictions.
- Interpretation methods include effect methods such as SHAP, LIME, ICE, PDP, and ALE, and importance methods such as PFI, ICI, PI, and SAGE.The paper distinguishes effects from contributions to model performance or prediction-function variance.
- Model-agnostic interpretation methods can be applied broadly, but their conclusions may mislead when models, data, or interpretation contexts are unsuitable.Dependencies, interactions, and lower-dimensional or linear approximations are highlighted as recurring concerns.
- The authors review pitfalls, illustrate them with examples, discuss practitioner solutions, and identify open issues for further research.Example code is made available for reproducing and extending the demonstrations.
2 Assuming One-Fits-All Interpretability
No single interpretation method answers every question: methods expose different aspects of models and data, so practitioners must define their goal and select methods accordingly.
- Assuming one IML method fits every context can produce dangerous misinterpretation because methods summarize different aspects of models and data.
- PFI and SHAP can rank features differently because they target different notions of relevance.PFI on unseen test data addresses relevance to generalization error, whereas SHAP can expose features used by the model.
- In a simulated problem where features and target are independent, test-set PFI treats features as irrelevant to generalization error while global SHAP exposes spurious model reliance.
- Extrapolation can be problematic for realistic-data interpretation but appropriate for studying sensitivity outside the observed data distribution.
- Practitioners should state detailed interpretation goals before analysis and tailor the method to the intended context.
- Open issues include connecting interpretation goals to methods mathematically and translating those links into practical diagnostics and guidelines.
3 Bad Model Generalization
Interpretations can be misleading when the fitted model does not represent the data-generating process well. Reliable interpretation therefore requires out-of-sample model evaluation, especially when overfitting is plausible.
- Underfitting or overfitting can distort inferred feature effects and importance because interpretation methods primarily describe the fitted model.
- Training-data evaluation gives overly optimistic performance estimates under overfitting and should be replaced by out-of-sample validation.
- Holdout, cross-validation, or repeated cross-validation should be used according to dataset size, with nested resampling for model selection and hyperparameter tuning.
- In the illustrated xgboost example, overfitting makes test-set PFI treat all features as irrelevant while mean marginal SHAP exposes mechanistically used features.
- A linear model can miss nonlinear relationships, producing high training and test errors and linear PDPs that deviate from the ground truth.
4 Unnecessary Use of Complex Models
Opaque complex models are unnecessary when interpretable alternatives perform comparably, and simpler models provide greater transparency and a better chance of revealing the true data-generating function.
- Opaque complex models are a pitfall when an interpretable model performs negligibly worse, equally well, or better.
- Inherently interpretable models provide greater transparency and increase the chance of discovering the true data-generating function.
- More than 90% of the predictive power of complex models was achieved by simple models across UCI benchmark data.
- Simple interpretable classifiers perform competitively for credit scoring, so classifier complexity or recency can mislead judgments of prediction performance.
- Start with linear regression and decision trees, using generalized additive models as a transition to smooth nonlinear effects.
- No single perfect definition of interpretability exists because it depends on context, leaving complexity–performance trade-offs as an open issue.
5 Ignoring Feature Dependence
Dependent features make perturbation-based interpretations vulnerable to unrealistic extrapolation, while conditional methods avoid extrapolation by changing the meaning of relevance.
- 5.1 Interpretation with Extrapolation: PFI, PDP, and Shapley values can extrapolate into poorly observed regions when features are dependent, producing misleading interpretations.
- 5.1 Interpretation with Extrapolation: Perturbation grids, permutations, and subsampling can create artificial points outside the multivariate joint distribution, especially with dependent features.
- 5.1 Interpretation with Extrapolation: Evaluating models in sparsely observed regions is risky because uncertainty is high, yet global methods may weight those points like realistic, high-confidence samples.
- 5.2 Confusing Linear Correlation with General Dependence: Pearson correlation near zero does not establish independence because it detects only linear association; nonlinear dependence can still distort interpretations.
- 5.3 Misunderstanding Conditional Interpretation: Conditional methods conform perturbations to conditional distributions, avoiding extrapolation but entangling effects of dependent features and changing relevance.
- 5.3 Misunderstanding Conditional Interpretation: CFI measures performance loss from permuting a feature given the others, whereas conditional SAGE may count information-bearing features not directly used by the model.
6 Misleading Interpretations due to Feature Interactions
Global averaging can hide feature effects when interactions create heterogeneous individual responses, so interaction-aware visualizations are needed alongside aggregate plots.
- 6.1 Misleading Feature Effects due to Aggregation: PDP and ALE can mislead when features interact because averaging masks heterogeneous effects.
- 6.1 Misleading Feature Effects due to Aggregation: The interacting feature X2 appears to have almost no effect in its PDP despite a constructed effect on the target.
- 6.1 Misleading Feature Effects due to Aggregation: ICE curves reveal heterogeneity between individual predictions that PDP and ALE average out.
- 6.1 Misleading Feature Effects due to Aggregation: Derivative or centered ICE curves remove intercept differences and highlight interaction-related variation and regions of strongest interaction.
- 6.1 Misleading Feature Effects due to Aggregation: Separately averaging upward- and downward-trending ICE curves can expose dependence of X2’s influence on interacting feature X3.
- 6.2 Misleading Feature Importance due to Interactions: Most interaction visualizations cannot identify interaction type or higher-order interactions, and automatic detection and ranking remain open problems.
- 6.2 Misleading Feature Importance due to Interactions: PFI combines a feature’s importance with the importance of all its interactions, while LIME and Shapley values provide additive explanations without separating them.
7 Ignoring Model and Approximation Uncertainty
Model training and interpretation estimation both introduce uncertainty, so a single PDP can misrepresent the stability of a feature effect. Repeated estimates can quantify this uncertainty, although general model-agnostic uncertainty measures remain incomplete.
- 7 Ignoring Model and Approximation Uncertainty: Model training on random data and Monte Carlo approximation make interpretation results uncertain.Ignoring either source can produce interpretations of noise and non-robust results.
- 7 Ignoring Model and Approximation Uncertainty: A single PDP can be misleading because it omits variance from both PDP estimation and model fitting.The second and third plots in Figure 8 expose these two sources of variation.
- 7 Ignoring Model and Approximation Uncertainty: Repeating PDP or PFI computations across permutations or bootstrap samples can quantify estimation uncertainty with confidence intervals.For PFI, existing confidence-interval and hypothesis-testing frameworks generally assume a fixed model.
- 7 Ignoring Model and Approximation Uncertainty: A general model-agnostic uncertainty measure for feature-effect methods such as ALE and PDP remains an open issue.Existing confidence bands for probabilistic ML models address model uncertainty, but do not resolve the broader problem.
8 Failure to Scale to High-Dimensional Settings
High-dimensional settings make interpretation outputs difficult to analyze and can make exact or multiple-testing procedures computationally and statistically problematic. Dimensionality reduction, meaningful feature grouping, selective interaction analysis, and multiplicity corrections offer partial solutions.
- 8 Failure to Scale to High-Dimensional Settings: Applying feature-level visualizations or importance scores in high-dimensional datasets produces overwhelming outputs that impede human analysis.The difficulty is especially pronounced for visualization-based interpretation methods.
- 8 Failure to Scale to High-Dimensional Settings: Reducing dimensionality or selecting features before interpretation can make outputs sparser, but usefulness depends on the semantic interpretability of the reduced space.Feature selection is preferable when users need interpretations in the original feature space.
- 8 Failure to Scale to High-Dimensional Settings: Meaningful feature grouping can make interpretation more efficient and more appropriate than analyzing single features.Examples include sensor data, time-lagged features, one-hot-encoded categories, and interaction terms.
- 8 Failure to Scale to High-Dimensional Settings: Grouped PFI perturbs an entire feature group, preserving within-group associations while breaking associations with other groups and the target.This approach is especially useful when features can be grouped meaningfully.
- 8 Failure to Scale to High-Dimensional Settings: Grouping-based interpretation remains limited when groups lack human intelligibility, and how groups influence predictions remains largely unanswered.Recent approaches use dimension reduction such as PCA, but the question remains open for further research.
- 8 Failure to Scale to High-Dimensional Settings: Exact Shapley values and full functional ANOVA do not scale linearly, with ANOVA components growing as O(2^p).Approximate Shapley methods sample feature orderings, whose estimate variance decreases in O(1/m).
- 8 Failure to Scale to High-Dimensional Settings: Ignoring multiple comparisons causes false-positive importance findings, increasingly so as dataset dimension grows.Bonferroni correction limits false positives but increases false negatives; Figure 9 shows a modest increase after correction versus considerable uncorrected growth.
9 Unjustified Causal Interpretation
Model-agnostic interpretation methods generally describe predictive associations rather than causal mechanisms, so feature relevance does not identify causes, effects, or intervention consequences. Causal interpretation requires strong assumptions about the data-generating process, model, and adjustment set, leaving causal discovery and inference an open challenge.
- 9 Unjustified Causal Interpretation: Interpretation methods do not generally answer causal questions about causes, interventions, effects, or counterfactuals.Practitioners may nevertheless be tempted to interpret their outputs causally.
- 9 Unjustified Causal Interpretation: Predictive models exploit associations rather than causal relationships, so feature relevance does not show whether a variable causes, affects, or is unrelated to the target.Models may also use variables that reconstruct unobserved influences on the target.
- 9 Unjustified Causal Interpretation: Even models relying only on direct causes require the causal structure among features to account for intervention-induced changes in other variables.Without causal-structure assumptions, interpretation methods cannot guide such actions.
- 9 Unjustified Causal Interpretation: A linear model treated all five features as relevant despite x3, x4, and x5 not causing Y in the constructed structural causal model.The fitted model achieved R^2 = 0.943 while assigning nonzero coefficients to every feature.
- 9 Unjustified Causal Interpretation: A PDP may represent an average causal effect only when the model performs well and the remaining variables form a valid adjustment set.Partial identification of causal roles also requires knowing whether the model is causal or anti-causal and accepting strong, non-testable assumptions.
- 9 Unjustified Causal Interpretation: Determining which causal insights can be gained under which assumptions remains a key open issue in ML.Further research is needed to make the connection between model interpretation and data-generating mechanisms explicit.
10 Discussion
The discussion consolidates common pitfalls of local and global model-agnostic interpretation and calls for more cautious practice. It also identifies the need for concrete guidance, better diagnostics, and further research toward well-understood interpretation tools.
- 10 Discussion: The paper reviews numerous pitfalls involving model generalization, dependent features, feature interactions, and causal interpretations.The review covers both local and global model-agnostic interpretation techniques.
- 10 Discussion: The authors advocate cautious interpretation, practical guidance on dangers and properties, and continued research on better-understood tools.They particularly highlight comparisons of interpretation techniques and diagnostic tools for misleading interpretations.