Source-linked AI summary
Explanations of model predictions with live and breakDown packages
Mateusz Staniak, Przemyslaw Biecek
TL;DR
Complex models can be difficult to interpret, which can erode trust in their predictions. This paper introduces live and breakDown for explaining black-box predictions and attributing their parts to features; its breakDown strategies yield consistent contributions despite differing variable orderings.
Problem
Complex models can be difficult to interpret, and lack of interpretability can undermine trust in predictions, an obstacle for applications such as personalized medicine.
Source: Introduction · p. 1Method
The paper introduces live to approximate a black-box model's local structure and breakDown to attribute parts of a prediction to variables.
Source: Introduction · p. 1, Motivation · p. 3, The live package · p. 8Results
Step-up and step-down breakDown strategies produce consistent variable contributions despite different variable orderings.
Source: The breakDown package · p. 11Takeaways & Limitations
The packages provide approaches for explaining complex-model predictions and attributing prediction parts to input features.
Source: Introduction · p. 1For non-additive models, ag-break uses additive attributions and therefore loses some information about model structure.
Source: The ag-break: model agnostic approach · p. 4
Abstract
from arXiv · showhide
Complex models are commonly used in predictive modeling. In this paper we present R packages that can be used to explain predictions from complex black box models and attribute parts of these predictions to input features. We introduce two new approaches and corresponding packages for such attribution, namely live and breakDown. We also compare their results with existing implementations of state of the art solutions, namely lime that implements Locally Interpretable Model-agnostic Explanations and ShapleyR that implements Shapley values.
Introduction
Complex black-box models can be difficult to interpret, undermining trust in their predictions. The paper surveys prediction-explanation methods and introduces two new approaches implemented in the R packages live and breakDown.
Introduction
- Complex models may be hard to understand because of their many parameters, and low interpretability can undermine trust in predictions, including in regulated applications.The introduction notes personalized medicine as an example of a regulated area where trust matters.Source: Introduction · p. 1
- Existing model-interpretation approaches include variable-importance rankings, partial-dependence plots, ICE, and ALE, but focus on one or a small set of variables.ICE can help identify interactions, while ALE was proposed for strongly correlated predictors.Source: Introduction · p. 1
- LIME explains a black-box prediction by fitting an interpretable model locally, and later work extended it as MAGIX and anchor-LIME.The cited works are Tulio Ribeiro et al. (2016b), Puri et al. (2017), and Tulio Ribeiro et al. (2016a), respectively.Source: Introduction · p. 1
- The discussed R and Python LIME implementations support several data types and model interfaces, but neither implements sp-LIME for selecting globally representative observations.The R package is described as handling tabular and text data and models supported by caret or mlr.Source: Introduction · p. 1
- The paper introduces live and breakDown as two new R-package approaches for explaining predictions from complex models.It also gives a short overview of prediction-explanation methods.Source: Introduction · p. 1
Related work
The related-work section reviews LIME and SHAP as approaches to explaining individual black-box predictions, emphasizing their distinct explanation frameworks.
Related work
- The section introduces LIME and SHAP among recognized methods for explaining a single prediction from a complex black-box model.LIME uses local fitting, while SHAP uses additive feature attributions.Source: Related work · p. 2, Locally Interpretable Model-agnostic… · p. 2, Shapley values (SHAP) · p. 2
Locally Interpretable Model-agnostic Explanations (LIME)
- LIME explains an individual prediction by perturbing features in a simplified binary input space, weighting sampled observations by similarity, and fitting an interpretable model.The fitted model may be penalized to limit its complexity.Source: Locally Interpretable Model-agnostic… · p. 2
Shapley values (SHAP)
- SHAP frames explanations as additive feature attributions, with Shapley values providing unique weights that satisfy local accuracy and consistency within that class of models.The framework encompasses LIME, and the paper reports that SHAP values were not then computable with a CRAN R package.Source: Shapley values (SHAP) · p. 2
Local Interpretable Visual Explanations (LIVE)
LIVE is presented as an alternative to LIME for regression that emphasizes visualization; it simulates local observations by changing features and fits a white-box model to black-box predictions.
Motivation
- LIVE is introduced as an alternative to LIME for regression, with model visualization emphasized as a way to understand complex models.Its local exploration and treatment of interpretable inputs differ from the original LIME approach.Source: Motivation · p. 2
- LIVE uses original variables as interpretable inputs and treats all simulated observations as similar to the explained instance, using an identity kernel.Variable selection is optional for linear regression when sparsity is required.Source: Motivation · p. 2
- LIVE supports visualizing local explanations with waterfall plots for linear-model predictor contributions and forest plots for local linear-approximation structure.The package can also visualize other interpretable models with plotting functions, including decision trees.Source: Motivation · p. 2
Methodology
- LIVE creates a simulated dataset around the selected observation, then fits a white-box model to black-box predictions at those simulated points.The observations are generated by changing one variable at a time; when predictors outnumber the desired observations, a random subset of predictors is used.Source: Methodology · p. 3
Model agnostic greedy explanations of model predictions (breakDown)
breakDown attributes parts of a single model prediction to input features, with a direct approach for additive models and a greedy extension for non-additive models.
Motivation
- The section frames feature attribution as decomposing a single observation's model score into parts associated with its input variables.The ag-break extension is not limited to additive models, but its additive attributions lose some information about non-additive model structure.Source: Motivation · p. 3, The ag-break: model agnostic approach · p. 4
The lm-break: version for additive models
- For linear and generalized linear models, lm-break attributes use centered feature values so the contributions are invariant to changes in feature scale or origin.The centered feature contributions are expressed in the same units.Source: The lm-break: version for additive… · pp. 3–4
The ag-break: model agnostic approach
- A relaxed prediction holds the selected features at the explained observation and averages responses over the population distribution for the remaining features, whose joint distribution is estimated.The method defines a feature's added contribution as the prediction change after relaxing that feature.Source: The ag-break: model agnostic approach · pp. 4–5
- The ag-break extension iteratively relaxes features, choosing each step to minimize the distance from the original prediction to a relaxed prediction.Each feature's contribution is the prediction change after that feature is relaxed.Source: The ag-break: model agnostic approach · pp. 4–5
- For non-additive models, ag-break approximates Shapley values; the approaches are identical for additive models.The paper says the greedy strategy gives easier-to-interpret explanations for non-additive models, but loses some information about model structure.Source: The ag-break: model agnostic approach · pp. 4–5
- The ag-break proceeds by either removing features from a full set or adding them from an empty set, and its plots show prediction changes across relaxations.In the step-down illustration, rows show score distributions, red dots mark relaxed-prediction estimates, and gray lines track individual observations.Source: The ag-break: model agnostic approach · p. 5, Figure 1 · p. 7, Figure 2 · p. 7
Case study: How good is this red wine?
The case study applies four approaches to explain an SVM prediction of wine quality, using the fifth wine as the example. It illustrates local surrogate explanations and direct feature attributions, including agreement between breakDown’s step-up and step-down strategies.
Case study: How good is this red wine?
- For the fifth wine, actual quality is 5 and the SVM predicts 5.03; the dataset predicts Portuguese Vinho Verde quality from 11 chemical properties.Source: Case study: How good is this red wine? · p. 8
The live package
- live approximates the SVM’s local structure around one observation with a simpler white-box model, allowing feature contributions to a single prediction to be assessed.The example generates artificial observations around the selected wine and fits a linear regression model to them.Source: The live package · p. 8
The lime package
- The lime package produces an explanation of the SVM prediction by showing contributions from particular features.Source: The lime package · p. 10, Figure 5 · p. 10
The breakDown package
- breakDown directly calculates variable attributions for a selected observation without using a surrogate model.Source: The breakDown package · p. 11
- Step-up and step-down ag-break attributions are very similar despite different feature orderings; the fifth wine’s SVM prediction is 5.032, 0.581 below average.Source: The breakDown package · p. 11, Figure 6 · p. 12
Shapley values (SHAP)
- The ShapleyR and breakDown packages’ feature attributions are compared in Table 2.Source: Shapley values (SHAP) · p. 13, Table 2 · p. 13
Discussion
The discussion contrasts four model-agnostic approaches and their attribution strategies, including the two introduced here. It notes trade-offs in interpretation and computation, as well as limits involving parameter choices, feature interactions, and the scope of the comparison.
Discussion
- The paper presents four R packages for explaining black-box predictions, two previously introduced and two new: live and breakDown.Source: Discussion · p. 14
- live and lime explain with local surrogate models, whereas breakDown and ShapleyR derive feature attributions from conditional responses of the black box.Source: Discussion · p. 14
- live perturbs original variables in data space for local exploration and offers visualization tools, but lacks Shapley values’ theoretical guarantees.Source: Discussion · p. 14
- breakDown uses one greedy sequence of conditionings, making it easier to interpret and faster to compute than averaging across Shapley conditionings; complex-model contributions can differ substantially.For linear models, the two methods produce nearly identical results; for more complex models, estimated contributions may even point in opposite directions.Source: Discussion · p. 14
- With the kernel and regularization settings of Lundberg and Lee (2017), lime estimates Shapley values; other choices can yield inconsistent results, and zero penalty prevents sparse explanations.Source: Discussion · p. 14
- Additive feature decompositions may work poorly for models with heavy feature interactions, and the paper’s method comparison is not comprehensive.The authors call for more research to better understand differences between the approaches.Source: Discussion · p. 14