PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable AI for Trees: From Local Explanations to Global Understanding

Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, Su-In Lee

arXiv:1905.04610v1cs.LGcs.AIstat.ML

TL;DR

Tree-based models lack well-developed methods for explaining individual predictions. This paper introduces TreeExplainer for exact local explanations, interaction effects, and global tools built from many local explanations, demonstrating faithful model understanding across medical applications.

  • Problem

    Existing tree-model interpretation emphasizes global feature importance, while local explanations for individual predictions remain limited and inconsistent.

  • Method

    TreeExplainer computes exact game-theoretic local explanations for trees, quantifies feature interactions, and aggregates local explanations to characterize global model structure.

  • Results

    Across three medical machine-learning problems, combining local explanations represented global structure while retaining local faithfulness and supported risk-factor, subgroup, interaction, and deployment-monitoring analyses.

  • Takeaways & Limitations

    The approach provides a unified basis for interpreting tree models locally and globally, including identifying model-behavior patterns and feature-specific deployment problems.

  • Takeaways & Limitations

    The medical applications include chronic kidney disease data without identifiable information, limiting direct access to identifiable patient information.

Abstract

from arXiv · show

Tree-based machine learning models such as random forests, decision trees, and gradient boosted trees are the most popular non-linear predictive models used in practice today, yet comparatively little attention has been paid to explaining their predictions. Here we significantly improve the interpretability of tree-based models through three main contributions: 1) The first polynomial time algorithm to compute optimal explanations based on game theory. 2) A new type of explanation that directly measures local feature interaction effects. 3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to i) identify high magnitude but low frequency non-linear mortality risk factors in the general US population, ii) highlight distinct population sub-groups with shared risk characteristics, iii) identify non-linear interaction effects among risk factors for chronic kidney disease, and iv) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model's performance over time. Given the popularity of tree-based machine learning models, these improvements to their interpretability have implications across a broad set of domains.

1 Introduction

TreeExplainer addresses the limited attention to local explanations for widely used tree-based models by enabling exact, game-theoretically optimal explanations, direct interaction measurements, and global structure analysis from many local explanations.

  • Motivation: Tree-based models are widely used across domains, but local explanations for individual predictions have received much less attention than global feature-impact methods.Existing local approaches include decision paths, heuristic credit allocation, and model-agnostic methods that may be unhelpful, biased, or slow.
  • Contributions: TreeExplainer exactly computes optimal local explanations for tree-based models using Shapley values that otherwise are generally NP-hard to compute exactly.The method builds on game-theoretic properties that characterize Shapley values as optimal within a broad class of feature-importance approaches.
  • Contributions: TreeExplainer directly measures local feature interactions through a theoretically grounded generalization of Shapley values.Single-number feature explanations are intuitive but cannot directly represent interaction effects.
  • Contributions: Combining exact local explanations across datasets yields tools that represent global model structure while retaining local faithfulness to the original model.These tools enable a range of analyses of global model behavior from many individual predictions.
  • Applications: The paper demonstrates TreeExplainer on three medical datasets covering mortality, chronic kidney disease progression, and hospital procedure duration.The datasets span different prediction targets and loss functions, including 14,407 mortality records, 10,745 renal visits, and 147,000 hospital procedures.

2 Results

The study introduces TreeExplainer and SHAP interaction values to make tree-model predictions faster, theoretically grounded, and more informative about local feature interactions. It also combines local explanations into global representations that preserve local faithfulness while revealing feature effects, dependencies, and population-level patterns.

  • Gradient boosted trees outperform deep learning and linear regression across all three medical datasets.
  • TreeExplainer exactly computes Shapley values in polynomial rather than exponential time, producing consistent, locally accurate explanations without estimation variability.It assigns credit impartially among features in multi-way AND operations and avoids inconsistency problems faced by Saabas values.
  • TreeExplainer consistently outperforms alternative methods across 21 local explanation metrics and matches human intuition across 12 user study scenarios.Shapley-based methods agreed with human intuition in all tested scenarios, unlike the heuristic Saabas values.
  • SHAP interaction values extend local explanations from single feature attributions to pairwise credit allocations that distinguish main effects from interaction effects.Traditional local attributions combine main and interaction effects, obscuring their distinction in individual explanations.
  • Combining local explanations yields global representations that preserve local faithfulness while capturing richer and more accurate model-wide patterns.SHAP summary plots display the magnitude, prevalence, and direction of feature effects while avoiding inconsistency and improving detection of true feature dependencies.
  • SHAP analyses reveal nonlinear, heterogeneous risk patterns, including long-tailed individual effects, age-dependent blood-pressure impacts, and an age-60 difference in mortality risk between men and women.In chronic kidney disease, blood-pressure interaction dispersion is partly driven by blood urea nitrogen; the age-sex interaction shows its largest risk difference at age 60.

3 Discussion

The discussion presents three interpretability advances for tree-based models: exact game-theoretic Shapley values, interaction-aware local explanations, and tools for interpreting global structure. It also shows applications to monitoring model deployment and discovering population subgroups through local explanation embeddings.

  • Interpretability contributions: The paper contributes an exact method for computing game-theoretic Shapley values, which provide explanations with several desirable properties.This is presented as the first of three main ways to improve interpretability.
  • Interpretability contributions: A richer local explanation directly captures feature interaction effects.This is the second main interpretability contribution.
  • Model monitoring: Monitoring plots reveal problems that would otherwise remain invisible during retrospective hospital model deployment.The plots include averaged squared error and SHAP values for model loss associated with a procedure-room feature.
  • Global model understanding: Local explanation embeddings support supervised clustering and interpretable dimensionality reduction, revealing mortality-risk subgroups and projected kidney-study visit structure.Hierarchical clustering identifies population subgroups with distinct mortality risk factors, while kidney-study embeddings are projected onto two principal components.

Methods … Methods 3 Model accuracy performance experiments

The study evaluates gradient boosted trees, linear models, and neural networks across mortality, chronic kidney disease, and hospital procedure duration datasets using task-specific losses, metrics, and train/validation/test splits. It also describes the data sources, ethical handling, and a time-based deployment split for monitoring model performance.

  • Methods 1 Institutional review board statement: The CRIC study data were determined not to involve human subjects because identifiable information was unavailable, while hospital data were IRB-approved and excluded protected health information.The hospital procedure data came from three institutional electronic medical record and data warehouse systems under Approval no. 46889.
  • Methods 2 The three medical datasets used for experiments: The study used NHANES I and NHEFS mortality data, CRIC kidney disease data, and hospital procedure records for three medical prediction tasks.The CRIC dataset contained 333 features across 10,745 visits from 3,939 patients, while NHANES I examined 23,808 individuals.
  • Methods 2.3 Hospital procedure duration dataset: A separate hospital split used the first year for training and the final three years for testing to simulate deployment and support ongoing model monitoring.These monitoring models were not hyper-parameter tuned because the goal was to demonstrate monitoring methods rather than perfect performance.
  • Methods 3 Model accuracy performance experiments: Gradient boosted trees achieved the highest accuracy across all three tabular medical datasets, assessed using 100 random train/test splits followed by retraining.The models were compared with linear and neural network approaches.
  • Methods 3.1 Mortality dataset: Mortality prediction used Cox proportional hazards loss and the C-statistic with a 64/16/20 train/validation/test split across 14,407 samples.Linear and neural network inputs were mean imputed and standardized, whereas gradient boosted trees used the original data with missing values.
  • Methods: The experiments implemented gradient boosted trees with XGBoost, linear models with lifelines or scikit-learn, and neural networks with DeepSurv or Keras.Model-specific hyper-parameters were selected using validation-based procedures, including coordinate descent, grid search, and early stopping.
  • Methods 3.2 Chronic kidney disease dataset: Chronic kidney disease prediction used logistic loss and area under the precision-recall curve with a 64/16/20 train/validation/test split across 10,745 samples.Gradient boosted trees used the original data, while linear and neural network inputs were mean imputed and standardized.
  • Methods 3.3 Hospital procedure duration dataset: Hospital procedure duration prediction used squared error loss and R2, with 147,000 procedures divided into an 80/10/5/5 train/validation/test1/test2 split.The test2 set was reserved for final performance numbers and was not used during method development.

Methods 4 Interpretability comparison of linear models and tree-based models in the presence of non-linearities

The comparison used synthetic mortality outcomes with tunable age–BMI non-linearity to evaluate logistic regression and gradient-boosted trees. Results showed that linear models slightly improved accuracy by weighting irrelevant features, whereas boosted trees retained the true age and BMI structure across non-linearity levels.

  • Interpretability comparison: Gradient-boosted trees correctly used only age and BMI across non-linearity levels, while linear models slightly increased accuracy by assigning weight to features not generating the outcome.The linear model’s additional features approximated non-linear relationships but moved its learned relationships farther from the truth.
  • Synthetic data generation: The synthetic label used a logistic function combining a linear age effect with a tunable quadratic BMI effect.The BMI term interpolated between linear and quadratic forms using parameter p.
  • Synthetic data generation: Eleven datasets varied non-linearity across p ∈ [0.0, 0.1, 0.2...1.0] while retaining real mortality covariates and synthetically generated labels.The predictors came from true data covariates, while labels were generated by a known mechanism.
  • Model comparison: Logistic regression and gradient-boosted trees were tuned and evaluated using a 64/16/20 train/validation/test split across all 11 datasets.Logistic regression varied its L1 penalty, while gradient-boosting hyperparameters were optimized over the training setup.
  • Interpretability analysis: SHAP values quantified each model’s total weight on features other than age and BMI by summing absolute irrelevant-feature attributions across samples.Linear SHAP values assumed feature independence, while TreeExplainer’s Tree SHAP was used for boosted trees.

Methods 5 Previous Global Explanation Methods for Trees … Methods 8 Convergence experiments for model agnostic Shapley value approximations

The methods review contrasts global and local explanation approaches for tree models, model-agnostic attribution methods, and convergence experiments evaluating their computational cost and estimation reliability. It highlights TreeExplainer’s global aggregation, the limitations of prior local methods, the uniqueness of Shapley-value explanations, and empirical comparisons of explainer convergence.

  • Methods 5 Previous Global Explanation Methods for Trees: Global feature importance methods range from counting split usage to attributing reductions in model loss, but split counts ignore differing split impacts.The passage identifies split counting as the most basic approach and contrasts it with loss-reduction attribution.
  • Methods 5 Previous Global Explanation Methods for Trees: Averaging TreeExplainer’s local explanations yields a global feature-importance measure that avoids classic Gain inconsistencies and retains high-order interaction effects missed by permutation methods.TreeExplainer-based global attribution also has higher power to detect important interacting features than current state-of-the-art methods, with implications for feature selection.
  • Methods 6 Previous local explanation methods for trees: Only two prior tree-specific local methods are identified: reporting the decision path and Saabas’s unpublished heuristic difference-in-expectations method.Because decision paths are not useful for large tree ensembles, the methods discussion focuses on Saabas’s heuristic approach.
  • Methods 6 Previous local explanation methods for trees: Saabas explanations follow a prediction’s decision path and assign feature attributions whose sum equals the difference between expected model output and the explained prediction.For ensembles formed by summing decision trees, the Saabas values are summed across the constituent trees.
  • Methods 7 Model agnostic local explanation methods: Model-agnostic local methods explain models through input-output behavior without assumptions about internal structure, using global mimics or locally fitted models such as MAPLE.The passage contrasts these approaches with gradient-based methods that can be misleading when relying on a single point.
  • Methods 7 Model agnostic local explanation methods: Backpropagation methods, local linear mimics, and game-theoretic methods form a unified class of additive feature attribution methods with a unique optimal approach implied by cooperative game theory.The cited prior study identifies Shapley values as the unique optimal explanation approach in this class.
  • Methods 8 Convergence experiments for model agnostic Shapley value approximations: Convergence experiments compare Kernel SHAP, IME, and TreeExplainer on runtime, variability, and estimation error using increasing-size XGBoost models and a chronic kidney disease dataset.Ground truth for estimation-error comparisons is obtained with TreeExplainer’s exact Independent Tree SHAP algorithm.
  • Methods 8 Convergence experiments for model agnostic Shapley value approximations: Low variance is defined as 1% of the tenth-largest feature impact among 333 features, with sample requirements measured until standard deviation falls below that threshold.The experiment evaluates both maximum standard deviation across features and mean standard deviation across repeated explanation runs.

Methods 9 Unifying previous heuristics with Shapley values

The section establishes SHAP values as the unique additive attribution method satisfying local accuracy, consistency, and missingness, and connects them to tree-based heuristics. It shows that Saabas values use a single tree-path ordering, whereas SHAP averages over all orderings and coincides with Saabas values in an infinite random-tree ensemble.

  • SHAP computation: SHAP computes each feature’s contribution by averaging changes in a conditional expectation function over all possible feature orderings.The conditional expectation is fx(S) ≈ E[f(x) | xS], with features introduced one at a time.
  • Shapley-value properties: SHAP values are the only feature attributions satisfying local accuracy, consistency, and missingness.Local accuracy requires attributions to sum to the model output; consistency prevents increased contributions from receiving lower attribution; missingness assigns zero to ineffective features.
  • Tree heuristics and SHAP: Saabas values consider only the single ordering defined by a tree’s decision path, while SHAP averages feature contributions over all orderings.Using one ordering can produce inconsistent credit allocation; averaging over orderings guarantees consistent allocation.
  • Tree heuristics and SHAP: In the limit of an infinite ensemble of totally random, fully developed trees with binary features, Saabas values equal SHAP values.Each tree supplies a random feature ordering, and an infinite ensemble represents all orderings equally.

Methods 10 TreeExplainer algorithms

TreeExplainer provides exact polynomial-time Shapley-value algorithms for tree models, replacing exponential subset enumeration with path-based computations. Independent Tree SHAP extends this framework to nonlinear output transformations by using background samples under an independence assumption.

  • Tree SHAP: Tree SHAP exactly computes Shapley values for trees and sums of trees in O(TLD^2) time, with balanced trees reducing this to O(TL log^2 L).T is the number of trees, L the maximum number of leaves, and D the maximum tree depth.
  • Tree SHAP: Tree SHAP uses tree traversal to compute conditional expectations, following the observed decision path for conditioned features and coverage-weighting both branches otherwise.The direct recursive approach has complexity O(TLM^2M) for all M feature SHAP values, motivating the polynomial algorithm.
  • Tree SHAP: The polynomial-time algorithm tracks subset proportions along unique-feature paths, using zero-path, one-path, and Shapley-weight information to avoid enumerating all subsets.Each path records the feature index, fractions of paths excluding or including the feature, and weighted subset cardinalities.
  • Independent Tree SHAP: Independent Tree SHAP explains nonlinear transformations by combining compositional approximation with Tree SHAP and enforcing independence between conditioned and remaining features.It averages attributions across R background samples using hybrid paths formed from one foreground and one background sample.
  • Independent Tree SHAP: Independent Tree SHAP runs in O(TRL) time across T trees and R background samples because each foreground-background traversal performs constant work per node.The algorithm considers O(L) terms per tree by indexing contributions by matched and total unique path features.

Methods 11 Benchmark evaluation metrics

The benchmark evaluates explanation methods across 21 metrics covering runtime, local accuracy, consistency, and feature-importance behavior. Its perturbation metrics test whether explanations correctly identify features that increase, decrease, or most affect model outputs under masking, resampling, and imputation.

  • Metric framework: 21 evaluation metrics capture runtime, local accuracy, consistency, and multiple ways to measure feature importance.The metrics were selected from practical runtime considerations, desirable explanation properties, and prior evaluations.
  • Metric framework: Runtime is reported as the time to explain 1,000 predictions, scaling per-prediction time from 100 random predictions while separately measuring initialization.Only per-prediction time was scaled by 10; initialization time was not scaled.
  • Metric framework: Local accuracy is exact when attribution values sum from a constant base value to each model output, with normalized deviation measuring near-compliance when exactness fails.The benchmark also computes the normalized standard deviation of output differences over 100 samples.
  • Metric framework: Consistency is classified as an exact guarantee, an infinite-sampling guarantee, or no guarantee because computational testing requires an exponential feature-space search.The benchmark reports these theoretical categories directly rather than exhaustively testing consistency.
  • Feature-importance metrics: Keep Positive metrics retain features estimated to increase outputs, while analogous negative and absolute metrics target output-decreasing features or overall importance.The framework implements mask, resample, and impute variants, replacing hidden features with means, training-set values, or correlation-based imputations.

Methods 12 User study experiments

The user study tested whether local feature-attribution methods agree with human intuition on simple decision trees. Shapley-based methods matched human consensus across all twelve cases, whereas the heuristic Saabas method diverged in several nonlinear cases.

  • Study design: 33 U.S. English-speaking Amazon Mechanical Turk participants explained three samples for each of four depth-two decision-tree models using binary symptoms.The models were AND, OR, XOR, and SUM; the sickness-score framing used fever, cough, and headache.
  • Study design: Participants allocated blame for each sickness-score output among fever, cough, and headache, producing twelve consensus credit allocations.Inputs were unconstrained except that fields could not be blank when the sickness score was non-zero.
  • Evaluation: The study used the twelve human consensus allocations as ground truth for twelve metrics based on summed absolute attribution differences.Explanation methods computed conditional expectations using a healthy background population with independent input features.
  • Results: All local explanation methods based on Shapley values agreed with human consensus across all twelve cases.The experiment evaluated consistency between method-generated feature attributions and human allocations on models that people could understand.
  • Results: The heuristic Saabas method differed significantly from human consensus in several nonlinear cases.The results therefore showed agreement between Shapley-based explanations and human intuition, alongside disagreement from Saabas on some nonlinear examples.

Methods 13 SHAP interaction values

SHAP interaction values extend feature explanations from individual attributions to pairwise local interaction effects, while preserving uniqueness guarantees and enabling separate analysis of main and interaction effects. For tree models, their computation can be made substantially cheaper by reusing the existing SHAP algorithm.

  • Interaction-value definition: SHAP interaction values use the Shapley interaction index to allocate credit among pairs of features, extending standard vector-valued feature attribution.The interaction index generalizes classic Shapley values and assigns credit among all pairs of players.
  • Interaction-value definition: Each pairwise interaction is split equally between its two features, so Φi,j(f, x) = Φj,i(f, x), while main effects remove off-diagonal interactions from SHAP values.The total pairwise interaction effect is Φi,j(f, x) + Φj,i(f, x).
  • Efficient computation: For tree models, SHAP interaction values reuse Algorithm 2 twice—once with feature j present and once absent—to drastically reduce computational cost.This computes the interaction as the difference between feature i’s SHAP values under those two conditions.
  • Interpretability benefits: SHAP interaction values retain uniqueness guarantees similar to SHAP values and separately expose main and interaction effects for each prediction.This local separation can reveal important interactions in tree ensembles that global interaction measures may miss.

Methods 14 Model summarization experiments

The model summarization experiments use SHAP explanations to improve global feature understanding for tree-based models by separating effect magnitude from prevalence. Summary plots reveal both overall importance and person-specific, nonlinear risk patterns that classic global importance can miss.

  • Methods 14 Model summarization experiments: SHAP-based global importance avoids classic Gain’s inconsistency problems and provides stronger global feature-selection power than Gain or permutation testing.This is important because tree-based models are often used for feature selection.
  • Methods 14 Model summarization experiments: Beeswarm-style SHAP summary plots show both the magnitude and prevalence of feature effects, rather than conflating them in a single importance value.The plots compile many local explanations to expose how feature contributions vary across individuals.
  • Methods 14 Model summarization experiments: In the NHANES I mortality experiment, TreeExplainer explanations of an XGBoost Cox proportional hazards model were compiled into a summary plot based on average SHAP magnitude.The model used hyperparameters optimized on a validation dataset.
  • Methods 14 Model summarization experiments: Age dominates mortality risk: high age values produce large positive SHAP values and low age values produce large negative SHAP values for log hazard ratio.The summary plot links feature values to the direction and magnitude of their model contributions.
  • Methods 14 Model summarization experiments: Features with low global importance can still have large effects for specific individuals, as illustrated by high blood protein substantially increasing mortality risk for some people.This person-specific pattern is revealed by the summary plot but not by classic global feature importance.

Methods 15 Feature dependence experiments

SHAP dependence plots extend traditional partial dependence by combining local feature importance across samples and exposing nonlinear patterns and interaction-driven heterogeneity. In mortality and CKD models, they reveal systolic-blood-pressure risk inflection points and feature interactions that conventional plots obscure.

  • Methods: SHAP dependence plots combine local importance across samples, providing richer feature-dependence information than traditional partial dependence plots.They plot a feature’s value against its SHAP value for individual samples.
  • Mortality model: The mortality model reproduces the known systolic-blood-pressure risk inflection between 120 mmHg and 140 mmHg.The plot also shows that interaction effects significantly affect a feature’s impact.
  • Mortality model: At a recorded systolic blood pressure of 180 mmHg, individuals’ log-hazard-ratio impacts vary from 0.2 to 0.6 because of differing factors.Vertical dot dispersion represents interaction effects with other model features.
  • Chronic kidney disease model: The CKD model reveals a previously documented nonlinear systolic-blood-pressure risk inflection, with interaction-driven dispersion partly associated with blood urea nitrogen.The finding underscores the importance of correctly modeling blood-pressure risk in CKD populations.

Methods 16 Interaction effect experiments

This section explains how SHAP interaction values decompose feature impacts into main and pairwise interaction effects at the sample level, and how aggregating samples reveals dataset-wide patterns. The experiments show that these plots expose otherwise hidden relationships in tree-ensemble models, including clinically meaningful interactions.

  • Interaction effect experiments: SHAP interaction values decompose a feature’s impact on each sample into a main effect and pairwise interaction effects with other features.Combining interaction values across many samples reveals patterns of interaction effects across a dataset.
  • Interaction effect experiments: In the mortality dataset, systolic blood pressure dependence is decomposed into a main-effect component and its symmetric interaction effect with age.The main effect plus interaction effects equals the original SHAP value for each sample.
  • Interaction effect experiments: Interaction plots reveal relationships hidden by complex tree ensembles, including the previously described interaction between age and sex.These plots expose interaction structure that is not apparent from feature effects alone.
  • Interaction effect experiments: In the chronic kidney disease model, high white blood cell counts are more concerning when accompanied by high blood urea nitrogen.This pattern represents an interaction effect between ‘white blood cells’ and ‘blood urea nitrogen’.

Methods 17 Model monitoring experiments

The model-monitoring experiments show that SHAP values for model loss can identify problematic features more directly and sensitively than monitoring aggregate loss or feature statistics. In a simulated hospital deployment, SHAP monitoring detected an intentional label swap, transient configuration errors, and feature drift that reduced predictive usefulness over time.

  • Model monitoring method: SHAP values for model loss decompose performance loss among input features, assigning blame directly to features and increasing power to detect otherwise hidden problems.This contrasts with monitoring only global fluctuations in model loss.
  • Experimental setup: The simulated deployment trained on one year of hospital data and evaluated the next three years, revealing previously undetected dataset issues even in a simple batch prediction task.The dataset contained four years of data from two large hospitals.
  • Monitoring results: SHAP monitoring identified an intentional swap of operating-room labels that produced error spikes difficult to distinguish from random fluctuations in smoothed aggregate loss.The labels of operating rooms 6 and 13 were intentionally swapped two-thirds of the way through the deployment simulation.
  • Monitoring results: A SHAP monitoring spike traced a transient general-anesthesia data problem to blank anesthesia fields caused by a temporary electronic medical-record configuration issue.The issue affected a subset of procedures from a single hospital.
  • Monitoring results: The atrial-fibrillation feature initially lowered loss but gradually became less useful and eventually increased loss by the end of deployment, indicating feature drift.SHAP dependence plots also showed that time of day was particularly effective at reducing loss just after 7:30am and 8:30am.

Methods 18 Local explanation embedding experiments

Local explanation embeddings turn samples into a supervised explanation space, enabling clustering and visualization that reveal shared model reasoning and global risk structure. In the kidney disease dataset, this space highlights creatinine- and urine-protein-related risk directions that are not apparent from standard unsupervised embeddings.

  • Explanation-space clustering: Local explanation embeddings address unsupervised clustering’s feature-scale and feature-meaning problems by grouping samples with the same model output for the same reason.SHAP values are comparable within a model because they share the model output’s units, even when original features differ.
  • Explanation-space visualization: Dimensionality reduction of SHAP-based explanation embeddings provides a continuous view of the primary directions of model-output variation.The embedding can be visualized directly rather than used only for explicit clustering.
  • Kidney disease embedding: In the kidney disease dataset, the top two principal components highlight creatinine and urine protein as distinct risk factors.Creatinine indicates lower eGFR, while urine protein measured by the protein-to-creatinine ratio marks kidney damage and complements eGFR in quantifying disease.
  • Kidney disease embedding: Coloring the explanation-space embedding by kidney disease progression risk reveals overall kidney-risk structure that standard unsupervised embeddings do not make apparent.The passage describes the resulting pattern as roughly continuous.

Supplementary Figures

The supplementary figures show that TreeExplainer improves consistency and efficiency over prior tree-specific and model-agnostic methods, while SHAP-based analyses expose interactions and deployment-related model changes. They also illustrate how local explanations support feature selection, embeddings, clustering, and model diagnostics.

  • Explanation methods: 105 evaluations require 1.8 CPU days for SHAP explanations of the chronic kidney disease dataset, versus 3.3 CPU years for SHAP interaction values.The runtime comparison concerns standard SHAP values and SHAP interaction values on the same dataset.
  • Explanation methods: TreeExplainer consistently allocates credit across multi-way AND functions and substantially improves runtime over sampling-based model-agnostic approaches.Saabas under-credits features near the root, whereas Tree SHAP distributes credit among all involved features.
  • Explanation methods: SHAP values average feature contributions over all orderings, unlike path-based Saabas values, because feature order matters for non-linear functions.Game-theoretic proofs identify averaging over all orderings as the only consistent approach satisfying the stated properties.
  • Global understanding: Partial dependence for systolic blood pressure misses the individual heterogeneity caused by model interactions that appears in the corresponding SHAP dependence plot.The supplementary comparison highlights why aggregate dependence can conceal interaction-driven differences between individuals.
  • Model monitoring: A SHAP monitoring plot identified atrial fibrillation as degrading model performance, and laboratory data linked this change to a significant drop in procedure duration during 2014.The duration decrease occurred after the simulated model deployment.
  • Global understanding: Local-explanation embeddings align important kidney-disease features with two largely orthogonal principal components, whereas unsupervised data embeddings need not align with the outcome.The comparison shows that explanation-space structure can reflect predictive organization more directly than unsupervised feature-space structure.
Loading 1905.04610v1…