Source-linked AI summary
Explainable AI for Trees: From Local Explanations to Global Understanding
Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, Su-In Lee
TL;DR
Tree-based models lacked effective local explanations for individual predictions despite established global interpretation methods. The paper introduces exact game-theoretic explanations, interaction measures, and global tools that reveal clinically relevant structure while retaining local faithfulness.
Problem
Existing tree-model interpretation emphasized global feature importance, while local explanations for individual predictions remained limited and inconsistent.
Source: §1 · p. 1, §2.3 · p. 5Method
TreeExplainer computes exact game-theoretic local explanations, captures feature interactions, and aggregates local explanations to characterize global model structure.
Source: §1 · p. 3Results
Across three medical applications, the tools represented global structure while retaining local faithfulness and supported risk-factor, subgroup, interaction, and deployment-monitoring analyses.
Source: §1 · p. 3, §2.7.4 · p. 10, §3 · p. 10Takeaways & Limitations
Combining faithful local explanations can support detailed interpretation of tree-model behavior, including subgroup discovery, interaction analysis, and detection of feature-specific deployment problems.
Source: §1 · p. 3, §2.7.4 · p. 10, Methods 13 SHAP interaction values · p. 29The monitoring approach assumes that attributing model loss to features can identify which features affect performance over time.
Source: Methods 17 Model monitoring experiments · p. 33
Abstract
from arXiv · showhide
Tree-based machine learning models such as random forests, decision trees, and gradient boosted trees are the most popular non-linear predictive models used in practice today, yet comparatively little attention has been paid to explaining their predictions. Here we significantly improve the interpretability of tree-based models through three main contributions: 1) The first polynomial time algorithm to compute optimal explanations based on game theory. 2) A new type of explanation that directly measures local feature interaction effects. 3) A new set of tools for understanding global model structure based on combining many local explanations of each prediction. We apply these tools to three medical machine learning problems and show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. These tools enable us to i) identify high magnitude but low frequency non-linear mortality risk factors in the general US population, ii) highlight distinct population sub-groups with shared risk characteristics, iii) identify non-linear interaction effects among risk factors for chronic kidney disease, and iv) monitor a machine learning model deployed in a hospital by identifying which features are degrading the model's performance over time. Given the popularity of tree-based machine learning models, these improvements to their interpretability have implications across a broad set of domains.
1 Introduction
Tree-based models are widely used but lack effective local explanations for individual predictions. The paper introduces TreeExplainer, which enables exact, theoretically grounded local explanations, interaction effects, and global model understanding from many local explanations.
- Tree-based models are broadly used, yet existing local explanation methods are limited by unhelpful decision paths, tree-depth bias, computational cost, and sampling variability.Global interpretation methods are well established, but comparatively little attention has been paid to explaining individual predictions.Source: §1 · pp. 1, 3
- Combining exact local explanations across datasets yields global structure representations that retain local faithfulness and provide more detailed, accurate views of model behavior.These explanations enable a range of tools for understanding global model structure.Source: Figure 1 · p. 2, §1 · p. 3
- TreeExplainer computes exact Shapley-value explanations for tree models in polynomial time, with guarantees of local accuracy and consistency.This avoids the NP-hard exact computation required generally and the potentially years-long CPU time of low-variance sampling for the smallest dataset.Source: §1 · p. 3
- TreeExplainer directly measures local feature interaction effects through a theoretically grounded generalization of Shapley values.Single-number feature attributions are intuitive but cannot directly represent interaction effects.Source: §1 · p. 3
- TreeExplainer is demonstrated on three medical datasets representing three loss-function types: mortality risk over twenty years, chronic kidney disease progression, and hospital procedure duration.The datasets contain 14,407 individuals and 79 features; 3,939 patients across 10,745 visits and 333 features; and 147,000 procedures with 2,185 features, respectively.Source: §1 · p. 3
2 Results
Across medical tabular datasets, gradient boosted trees outperform deep and linear models while offering more faithful explanations under non-linearity. TreeExplainer provides exact, consistent, fast local and interaction explanations that support global summaries, subgroup discovery, and deployment monitoring.
2.1 Tree-based models can be more accurate than neural networks
- Gradient boosted trees outperform deep learning and linear regression across all three medical datasets, while remaining more interpretable than linear models under non-linearity.As non-linearity increases, linear models lose accuracy and assign importance to irrelevant features, whereas gradient boosted trees retain a more faithful connection to the data.Source: §2.1 · p. 5, §2.2 · p. 5
2.3 Current local explanations for tree-based models are inconsistent
- TreeExplainer computes exact SHAP values in polynomial time, eliminating the inconsistency of Saabas values and the sampling variability and scalability limits of model-agnostic methods.It assigns credit impartially regardless of tree depth, uniformly handles multi-way AND operations, and is several orders of magnitude faster than model-agnostic alternatives for moderate models.Source: §2.5 · pp. 5–6, §2.4 · p. 5
2.5 TreeExplainer provides fast local explanations with guaranteed consistency
- TreeExplainer consistently outperforms alternative methods across 21 local explanation metrics and matches human intuition in all 12 tested user-study scenarios.The evaluation covered eight explanation methods, three model types, and three datasets.Source: §2.5 · p. 6
2.6 TreeExplainer extends local explanations to measure interaction effects
- SHAP interaction values extend local explanations from individual feature attributions to a matrix separating main effects from pairwise interaction effects.This preserves uniqueness guarantees and enables interaction patterns that single-number feature attributions can obscure.Source: §2.6 · p. 6
2.7.1 Local model summarization reveals rare high-magnitude effects on mortality risk and increases feature selection power
- Combining local explanations into SHAP summaries reveals feature-effect magnitude, prevalence, direction, and rare high-magnitude mortality risks that global averages can conceal.The mortality summary shows long right tails for medical measurements, with rare effects consistently increasing mortality risk rather than indicating unusually long survival.Source: §2.7.1 · p. 8
2.7.2 Local feature dependence reveals both global patterns and individual variability in mortality risk and chronic kidney disease
- SHAP dependence plots recover blood-pressure risk inflection points while exposing individual variability from age interactions in mortality and blood-urea-nitrogen interactions in chronic kidney disease.For mortality, early-onset high blood pressure has greater modeled impact than late-onset hypertension; in kidney disease, vertical dispersion is partly driven by blood urea nitrogen.Source: §2.7.2 · p. 8
2.7.3 Local interactions reveal sex-specific life expectancy changes during aging and inflammation effects in chronic kidney disease
- SHAP interactions reveal age-dependent sex differences in mortality risk and show that high white blood cell counts intensify the negative risk associated with high blood urea nitrogen.The largest male–female mortality-risk difference occurs at age 60, while the kidney-disease pattern supports an interaction between inflammation and high blood urea nitrogen.Source: §2.7.3 · pp. 8, 10, Figure 4 · p. 9
2.7.4 Local model monitoring reveals previously invisible problems with deployed machine learning models
- Local loss explanations identify deployment bugs, temporary medical-record configuration problems, and feature drift that overall model loss fails to reveal.In the simulated hospital deployment, TreeExplainer detected a room-label swap, a general-anesthesia error spike, and atrial-fibrillation drift that eventually harmed performance.Source: §2.7.4 · p. 10
3 Discussion
The discussion presents exact Shapley-value computation, interaction-aware local explanations, and global interpretation tools built from local explanations. It also shows that these tools support interpretable subgroup discovery and detection of otherwise invisible problems in deployed hospital models.
Contributions
- The paper contributes an exact method for game-theoretic Shapley values, interaction-aware local explanations, and model-interpretation tools that combine faithful local explanations.The authors argue that local explanations can serve as foundational building blocks for downstream machine-learning tasks because they remain faithful to individual predictions.Source: §3 · p. 10
Deployment monitoring
- Monitoring SHAP values exposed swapped room labels, transient anesthesia-data corruption, and atrial-fibrillation feature drift that overall model loss did not reveal.The atrial-fibrillation drift had P-value 5.4 × 10−19.Source: Figure 5 · p. 11
Global interpretation
- Local-explanation embeddings enabled interpretable mortality-risk subgroups and dimensionality reduction for kidney-disease visits, unlike an equivalent unsupervised PCA embedding.Embedding axes correspond to feature attributions in model-output units, allowing projections to be colored by predicted risk or specific feature contributions.Source: Figure 6 · p. 12
Methods … Methods 3 Model accuracy performance experiments
The study evaluates tree-based, linear, and neural models on three medical prediction tasks, finding gradient boosted trees achieve the highest accuracy across all datasets. The datasets span mortality, chronic kidney disease progression, and hospital procedure duration, with approved or non-human-subjects data handling.
Methods 1 Institutional review board statement
- The CRIC study was determined not to involve human subjects because researchers lacked identifiable information, while hospital data use received institutional review board approval and excluded protected health information.The CRIC determination was recorded under IRB ID STUDY00006766; hospital data approval number 46889 covered records from three institutional systems.Source: Methods 1 Institutional review board… · p. 14
Methods 2.1 Mortality dataset
- NHANES I and its follow-up provided 79 features for 14,407 individuals, including 4,785 recorded deaths, for mortality modeling.The data covered U.S. participants from 1971–1974 with mortality follow-up through 1992 and supported Cox proportional hazards models.Source: Methods 2.1 Mortality dataset · p. 14
Methods 2.2 Chronic kidney disease dataset
- The CRIC kidney dataset comprised 333 features across 10,745 visits from 3,939 patients, with four-year end-stage renal disease progression as the binary outcome.The cohort followed patients recruited from seven clinical centers, addressing a clinically important progression outcome affecting chronic kidney disease care.Source: Methods 2.2 Chronic kidney disease… · p. 14
Methods 2.3 Hospital procedure duration dataset
- Hospital records yielded 2,185 preoperative features across 147,000 procedures, with procedure duration defined as the patient’s time in the operating room.Data combined automated AIMS capture, manual anesthesia documentation, and electronic medical records to support scheduling-related prediction.Source: Methods 2.3 Hospital procedure… · p. 14
Methods 3 Model accuracy performance experiments
- Gradient boosted trees achieved the highest accuracy across all three medical datasets, significantly outperforming linear and neural models in several task-specific comparisons.Using 100 random train/test splits, they outperformed lasso and neural networks for mortality, lasso for kidney disease, and both baselines for procedure duration.Source: Methods 3 Model accuracy performance… · p. 15
Methods 3.1 Mortality dataset
- Mortality models used Cox loss and C-statistic evaluation with a 64/16/20 train/validation/test split; trees used unnormalized data with missing values, unlike linear and neural models.The tree model was implemented in XGBoost, while the linear and neural baselines used lifelines and DeepSurv, respectively.Source: Methods 3.1 Mortality dataset · p. 15
Methods 3.2 Chronic kidney disease dataset
- Kidney progression models used logistic loss and area under the precision-recall curve with a 64/16/20 split, and tree models received unnormalized data with missing values.The compared models included XGBoost trees, an L1-regularized linear model, and a tuned 1D convolutional neural network.Source: Methods 3.2 Chronic kidney disease… · pp. 15–16
Methods 3.3 Hospital procedure duration dataset
- Procedure-duration models used squared loss and R2 evaluation on 147,000 procedures split into train, validation, test1, and held-out test2 sets.A separate time-based split trained on one year and tested on three years to simulate deployment monitoring without hyperparameter tuning.Source: Methods 3.3 Hospital procedure… · p. 16
Methods 4 Interpretability comparison of linear models and tree-based models in the presence of non-linearities
The comparison tests whether linear models remain interpretable under controlled non-linearity, using realistic synthetic labels based on age and BMI from NHANES I mortality data. Gradient boosted trees preserve the true feature structure, whereas linear models distribute weight across irrelevant features while gaining only slight accuracy.
Methods 4 Interpretability comparison of linear models and tree-based models in the presence of non-linearities
- Gradient boosted trees use only age and BMI across all non-linearity levels, while all-feature linear models place substantial weight on irrelevant features for a small accuracy gain.The comparison uses simulated outcomes where only age and BMI generate the label.Source: Methods 4 Interpretability comparison… · p. 16
Methods 4.1 Selection of features
- Age and BMI were selected as the true label-generating features because SHAP dependence plots showed linear mortality dependence for age and a U-shaped relationship for BMI.The features were selected from classifiers trained on the full NHANES I mortality dataset.Source: Methods 4.1 Selection of features · p. 16
Methods 4.2 Label generation
- Synthetic labels used a logistic function combining a linear age term with a BMI function that smoothly varied from linear to quadratic as p ranged from 0.0 to 1.0.The quadratic BMI contribution was weighted by p, with labels retained as probabilities rather than binarized outcomes across 11 datasets.Source: Label generation · p. 17
Methods 4.3 Model training
- Gradient boosted trees and L1-regularized logistic regression were tuned on validation data and evaluated using a 64/16/20 train/validation/test split.Tree models varied depth and bagging subsampling, while logistic regression varied λ from 10^-4 to 10^4.Source: Methods 4.3 Model training · p. 17
Methods 4.4 Feature importance
- Irrelevant-feature importance was measured by summing absolute SHAP values for all features except age and BMI across samples and non-linearity levels.Linear SHAP assumed feature independence, whereas TreeExplainer’s Tree SHAP was used for gradient boosted trees.Source: Methods 4.4 Feature importance · p. 17
Methods 5 Previous Global Explanation Methods for Trees … Methods 8 Convergence experiments for model agnostic Shapley value approximations
The paper contrasts global and local explanation methods for tree models, then shows that TreeExplainer computes exact Shapley values efficiently and outperforms model-agnostic approximations in convergence and runtime.
Methods 5 Previous Global Explanation Methods for Trees
- Global feature-importance methods count splits, aggregate Gain, or measure loss changes after permutation, but they are not directly applicable to explanations of individual predictions.Gain can be biased for finite ensembles of greedily built trees, while grouped permutation can account for feature dependencies.Source: Methods 5 Previous Global Explanation… · p. 17
- TreeExplainer’s global feature attributions avoid Gain inconsistencies, capture high-order interactions missed by permutation importance, and better detect important interacting features.These properties support feature selection using tree ensembles.Source: Methods 5 Previous Global Explanation… · pp. 17–18
Methods 6 Previous local explanation methods for trees
- The Saabas method explains a prediction by attributing changes in expected model output to features encountered along its decision path.Node expectations are estimated from training samples reaching each node, making the method efficient.Source: Methods 6 Previous local explanation… · p. 18
- Saabas feature attributions sum to the difference between the model’s expected output and the explained prediction, and ensemble values sum across trees.Source: Methods 6 Previous local explanation… · p. 18
Methods 7 Model agnostic local explanation methods
- Model-agnostic local explainers perturb inputs or fit global or local mimic models, using gradients, linear coefficients, decision rules, or game-theoretic allocations.Examples include LIME, Anchors, IME, and QII.Source: Methods 7 Model agnostic local… · p. 18
- Although Shapley values are generally NP-hard to compute, restricting the problem to tree models enables TreeExplainer to calculate exact values in low-order polynomial time.Shapley values are the unique optimal additive attribution satisfying the stated desirable game-theoretic properties.Source: Methods 7 Model agnostic local… · p. 18
Methods 8 Convergence experiments for model agnostic Shapley value approximations
- TreeExplainer is exact, has no stochastic variability, and remains far faster than Kernel SHAP and IME as feature count increases.Kernel SHAP and IME require repeated model evaluations and additional implementation overhead; their reported runtimes are lower bounds.Source: Methods 8 Convergence experiments for… · p. 19
- Over 3 CPU years were required for low-variance interaction estimates, compared with almost 2 CPU days for basic explanations on the chronic kidney disease dataset.Low variance was defined as 1% of the tenth-largest feature impact among 333 features.Source: Methods 8 Convergence experiments for… · p. 19
Methods 9 Unifying previous heuristics with Shapley values
The section establishes SHAP values as the unique additive feature-attribution method satisfying local accuracy, consistency, and missingness, then unifies tree-specific heuristics with this framework. It shows that Saabas values use one tree-path ordering and can violate consistency, while converging to SHAP values for infinite random fully developed ensembles on binary features.
Methods 9 Unifying previous heuristics with Shapley values
- SHAP computes each feature’s conditional-expectation contribution across all feature orderings, averaging the resulting changes to obtain an attribution.Using fx(S) ≈ E[f(x) | xS] yields SHAP values, a model-output-specific form of Shapley values [41, 63].Source: Methods 9 Unifying previous heuristics… · pp. 19–20
- SHAP values are the only additive attribution method satisfying local accuracy, consistency, and missingness [41, 63].Local accuracy requires attributions to sum to f(x), consistency prevents increased feature contributions from receiving lower attribution, and missing features receive zero attribution.Source: Methods 9 Unifying previous heuristics… · p. 20
- Saabas values follow only a tree’s decision-path ordering, preserving local accuracy but violating consistency and misallocating credit toward deeper splits.In multi-way AND trees, equally important features near the root receive less credit than features near the leaves.Source: Methods 9 Unifying previous heuristics… · p. 20
- Shapley averaging guarantees consistent credit allocation, whereas single-ordering methods such as Saabas and Gain can allocate credit inconsistently.Saabas parallels Gain because both assign changes after splits to the split feature while considering only tree-path orderings.Source: Methods 9 Unifying previous heuristics… · p. 20
- For binary features, Saabas values equal SHAP values in the limit of an infinite ensemble of totally random fully developed trees.Each random decision path supplies a feature permutation, and an infinite ensemble represents all permutations equally for each leaf value.Source: Methods 9 Unifying previous heuristics… · pp. 20–21
Methods 10 TreeExplainer algorithms
TreeExplainer provides exact, low-order polynomial-time Shapley explanations for tree ensembles through Tree SHAP, plus an independence-based method for nonlinear output transformations. Its algorithms use recursive tree traversal, subset-size bookkeeping, and background samples to preserve exactness or obtain efficient estimates.
Methods 10 TreeExplainer algorithms
- TreeExplainer presents a slow direct Tree SHAP method, a polynomial-time Tree SHAP method, and Independent Tree SHAP for nonlinear transformations such as model loss.Although general Shapley-value computation is NP-hard, restricting the model class to trees enables exact low-order polynomial-time algorithms.Source: Methods 10 TreeExplainer algorithms · p. 21, Methods 10.2 Independent Tree SHAP:… · p. 22
Methods 10.1 Tree SHAP
- Tree SHAP exactly computes consistent and locally accurate Shapley explanations in O(TLD^2) time, improving exponentially over prior exact methods with O(TLM^2M) complexity.For balanced trees, the complexity becomes O(TL log^2 L).Source: Methods 10.1 Tree SHAP · p. 21, Methods 10.1.2 Estimating SHAP values… · p. 22
Methods 10.1.1 Estimating SHAP values directly in O(TLM^2M) time
- The direct Tree SHAP procedure recursively follows conditioned split paths and averages both branches otherwise, yielding O(TLM^2M) computation across all M features.Each unconditioned branch is weighted by node coverage, while conditioned splits follow the input decision path.Source: Methods 10.1.1 Estimating SHAP values… · p. 21
Methods 10.1.2 Estimating SHAP values in O(TLD^2) time
- The polynomial Tree SHAP algorithm tracks subset sizes, path fractions, and Shapley weights during recursion, then uses EXTEND and UNWIND to compute feature-specific leaf weights.UNWIND reverses extensions, including repeated splits on a feature, while preserving the weighting required by subset cardinality.Source: Methods 10.1.2 Estimating SHAP values… · p. 22
Methods 10.2 Independent Tree SHAP: Estimating SHAP values under independence in O(TRL) time
- Independent Tree SHAP exactly computes traditional Shapley values under feature independence in O(TRL) time by grouping path contributions into O(L) terms.The algorithm repeats traversal for each tree and each of R background samples.Source: Methods 10.2 Independent Tree SHAP:… · p. 24
Methods 11 Benchmark evaluation metrics
The benchmark evaluates explanation methods with 21 metrics covering runtime, local accuracy, consistency, feature importance, and positive, negative, and absolute effects. It compares mean masking, independent resampling, and correlation-based imputation, while excluding retraining-based metrics because they can mislead in some situations.
Methods 11 Benchmark evaluation metrics
- The benchmark uses 21 metrics spanning runtime, local accuracy, consistency, feature importance, and multiple feature-removal strategies.The metrics were selected to capture practical runtime, desirable explanation properties, and different ways to measure feature importance.Source: Methods 11 Benchmark evaluation metrics · p. 24
Methods 11.1 Runtime
- Runtime measures the time to explain 1,000 predictions, scaling per-prediction time from 100 random predictions while also recording initialization time.Only per-prediction time is scaled; initialization time is measured separately.Source: Methods 11.1 Runtime · p. 24
Methods 11.2 Local accuracy
- Local accuracy is exact when attribution values sum from a constant base value to each model output, with normalized deviation also measured when exactness fails.The deviation is computed over 100 samples.Source: Methods 11.2 Local accuracy · p. 24
Methods 11.3 Consistency guarantees
- Consistency is assessed through theoretical guarantees classified as exact, guaranteed only with infinite sampling, or absent, rather than by exponential computational testing.Enumerating the relevant search space would be exponential in the number of input features.Source: Methods 11.3 Consistency guarantees · p. 26
Methods 11.4–11.12 Keep metrics
- Keep metrics test whether explanations identify features with positive, negative, or absolute importance under mean masking, resampling, or correlation-based imputation.Keep Positive evaluates output increases, Keep Negative evaluates output decreases, and Keep Absolute evaluates accuracy using the most important features.Source: Methods 11.4 Keep positive (mask) · p. 26, Methods 11.5 Keep positive (resample) · p. 26, Methods 11.6 Keep positive (impute) · p. 26
Methods 11.13–11.21 Remove metrics
- Remove metrics reverse the corresponding keep tests: removing positive or negative features should shift model output oppositely, while removing absolute-important features should reduce performance.Each removal family is implemented with masking, resampling, and imputation variants.Source: Methods 11.13 Remove positive (mask) · p. 27, Methods 11.14 Remove positive (resample) · p. 27, Remove negative (mask) · p. 28
Methods 12 User study experiments
The study compares local explanation methods with human feature-credit assignments on simple decision trees. Shapley-based methods match human consensus across all twelve cases, whereas Saabas diverges in several nonlinear cases.
- The experiment recruited 33 U.S. English-speaking Mechanical Turk participants to explain three samples for each of four depth-two decision-tree models using binary features.The models were AND, OR, XOR, and SUM; headache was unused, while fever and cough had linear and, except for SUM, nonlinear effects.Source: Methods 12 User study experiments · p. 28
- Researchers converted twelve consensus credit allocations into ground truth for twelve metrics measuring absolute differences from each explanation method’s feature attributions.A healthy population with independent input features served as the background reference dataset for conditional expectations.Source: Methods 12 User study experiments · p. 29
- Across all twelve cases, Shapley-based local explanation methods matched human consensus feature attributions, while Saabas differed significantly in several nonlinear cases.The study reports that these results complement Shapley values’ theoretical guarantees and quantitative performance.Source: Methods 12 User study experiments · p. 29
Methods 13 SHAP interaction values
SHAP interaction values extend SHAP by representing each prediction’s feature effects as a matrix that separates main effects from pairwise interactions. For tree models, Tree SHAP enables efficient computation while preserving local, uniquely characterized interaction explanations.
Interaction-value definition
- SHAP interaction values allocate prediction credit across feature pairs, yielding a matrix with main effects on the diagonal and interactions off-diagonal.They are derived from the Shapley interaction index, a generalization of the Shapley value that assigns credit among pairs of players.Source: Methods 13 SHAP interaction values · p. 29
- The interaction between features i and j is computed from their joint, individual, and absent-feature model outputs, then split equally so Φ_i,j(f, x) = Φ_j,i(f, x).The diagonal contains main effects, while the off-diagonal contributions are removed from each feature’s SHAP value; Φ_0,0(f, x) = f_x(∅) ensures the matrix sums to the model output.Source: Methods 13 SHAP interaction values · p. 29
Efficient computation
- Tree SHAP computes SHAP interaction values in O(TMLD^2) time by running the algorithm twice per feature, once with the other feature present and once absent.This reduces the cost relative to direct computation for tree models.Source: Methods 13 SHAP interaction values · p. 29
Local interaction explanations
- SHAP interaction values have uniqueness guarantees similar to SHAP values [21] and provide a local way to separate main and interaction effects beyond listing decision paths.This separation can reveal important tree-ensemble interactions that global interaction measures may not expose for individual predictions.Source: Methods 13 SHAP interaction values · p. 29
Methods 14 Model summarization experiments
The experiments show that dataset-averaged SHAP values improve global feature importance measurement, while SHAP summary plots reveal effect magnitude, prevalence, direction, and person-specific variation.
Global feature importance
- Dataset-averaged SHAP values retain SHAP’s theoretical guarantees, avoid Gain’s inconsistency, and provide better global feature-selection power than Gain or permutation testing.This addresses the use of tree-based models for feature selection.Source: Model summarization experiments · p. 30
SHAP summary plots
- SHAP summary plots distinguish effect magnitude from prevalence and additionally show each feature’s effect direction across individual predictions.Beeswarm plots encode impact along the x-axis, density through piled dots, and direction through color.Source: Model summarization experiments · p. 30
NHANES I mortality
- In the NHANES I mortality model, age dominated risk, while men showed about a 0.6 increase in log-hazards relative to women, equivalent to roughly 7 years of age.The sex effect varied across individuals because interactions with other features modulated its importance.Source: Model summarization experiments · p. 30
- Summary plots revealed rare, high-magnitude effects that global importance obscured, including large mortality impacts from high blood protein for some individuals.These asymmetric right-tailed effects reflect many ways to die abnormally early when medical measurements are out of range.Source: Model summarization experiments · p. 30
Methods 15 Feature dependence experiments
SHAP dependence plots combine feature values with local SHAP impacts across samples, revealing nonlinear risk patterns and interaction-driven heterogeneity that partial dependence plots miss. Applied to mortality and CKD models, they expose distinct interaction drivers of systolic blood-pressure risk.
Feature dependence plots
- SHAP dependence plots map each feature’s value to its SHAP impact for every sample, combining local importance across the dataset.This provides richer information than traditional partial dependence plots and global feature-importance measures.Source: Methods 15 Feature dependence… · p. 30
- In the mortality model, systolic blood pressure shows the known 120–140 mmHg risk inflection point, while age explains much of the interaction-driven dispersion.At 180 mmHg, estimated log-hazard impacts range from 0.2 to 0.6 across individuals, with early-onset hypertension appearing more concerning than late-onset hypertension.Source: Methods 15 Feature dependence… · pp. 30–31
- In the CKD model, SHAP dependence plots reveal a nonlinear systolic-blood-pressure risk inflection whose interaction-driven dispersion is partly explained by blood urea nitrogen.This analysis contextualizes blood-pressure modeling alongside evidence that lower blood pressure can improve some kidney and cardiovascular outcomes.Source: Methods 15 Feature dependence… · p. 31
Methods 16 Interaction effect experiments
SHAP interaction values decompose feature effects into main and pairwise interaction components at the individual-sample level, enabling dataset-wide interaction patterns. In mortality and chronic kidney disease models, these analyses revealed age- and clinically meaningful biomarker interactions.
- SHAP interaction values separate each feature’s main effect from pairwise interactions, allowing individual-sample effects to be combined into dataset-wide interaction patterns.This decomposition exposes relationships learned by complex tree ensembles that may otherwise remain hidden.Source: Methods 16 Interaction effect… · p. 31
- Systolic blood pressure’s vertical dispersion in the mortality model is primarily driven by its interaction with age, confirmed by SHAP interaction values.Adding the decomposed main and interaction components reproduces the original SHAP value and corresponding dependence plot.Source: Methods 16 Interaction effect… · p. 31
- High white blood cell counts are more concerning to the chronic kidney disease model when accompanied by high blood urea nitrogen, supporting an inflammation-related interaction.The interaction indicates that inflammation may combine with high blood urea nitrogen to contribute to faster kidney function decline.Source: Methods 16 Interaction effect… · p. 31
Methods 17 Model monitoring experiments
SHAP values of model loss decompose performance changes by feature, enabling monitoring plots that detect deployment errors and feature drift hidden by overall loss. In hospital procedure-duration data, these plots exposed labeling errors, configuration failures, and changing feature usefulness over time.
- Decomposing model loss by feature assigns blame directly and increases power to detect problems hidden within fluctuations of overall performance.Unlike overall-loss monitoring or feature-statistics anomaly detection, SHAP plots quantify each feature’s impact in the model’s loss units.Source: Methods 17 Model monitoring experiments · pp. 31–33
- The experiment used one year of hospital procedure-duration data for training and three subsequent years from two hospitals to simulate deployment.Even this relatively simple batch-prediction setting revealed dataset issues that prior data cleaning had not detected.Source: Methods 17 Model monitoring experiments · p. 32
- Figure 5B identified the operating-room labeling error when overall loss in Figure 5A showed no indication of the problem.Swapping room 6 and 13 labels caused the room 6 SHAP impact to shift from helping accuracy to hurting it, while the affected procedures were too infrequent to change overall loss visibly.Source: Methods 17 Model monitoring experiments · p. 32
- SHAP monitoring revealed a transient anesthesia-data configuration failure and feature drift affecting atrial-fibrillation ablation procedures.The general-anesthesia spike traced to a broken electronic-medical-record link, while atrial-fibrillation usefulness declined as procedural technologies and staffing changed; the trend had P-value 5.4 × 10−19.Source: Methods 17 Model monitoring experiments · p. 32
- Loss dependence plots showed that time of day most reduced error near 7:30am and 8:30am because those periods contained scheduled long-running elective surgeries.This illustrates how feature-level loss explanations can reveal the operational patterns a model uses to improve predictions.Source: Methods 17 Model monitoring experiments · p. 33
Methods 18 Local explanation embedding experiments
Local explanation embeddings enable supervised clustering and visualization by grouping samples according to model outputs and the feature contributions driving them. Applied to mortality and kidney-disease models, these embeddings recover clinically meaningful subgroups and orthogonal risk directions.
- Local explanation embeddings address unsupervised clustering’s incomparable feature units and outcome-independent weighting by grouping samples with the same model output for similar reasons.SHAP values share the model-output units, making feature contributions comparable within a model and outcome-specific.Source: Methods 18 Local explanation embedding… · p. 33
- Hierarchical clustering of mortality-model SHAP values separates broad age groups and identifies smaller high-risk subgroups, including early-onset hypertension, inflammation-marked youth, and underweight older people.These groups recapitulate known high-risk populations that simple unsupervised clustering missed.Source: Methods 18 Local explanation embedding… · p. 33
- The kidney-disease explanation embedding’s first two principal components align with blood creatinine and urine protein, yielding a roughly continuous increase in progression risk.Creatinine reflects eGFR, while urine protein-to-creatinine ratio marks kidney damage; together they define distinct risk directions.Source: Methods 18 Local explanation embedding… · p. 33
- eGFR and urine PCR identify unique individuals at risk of end-stage renal disease, supporting their parallel measurement and revealing structure obscured by standard unsupervised embeddings.Additional chronic kidney disease features also align with the two orthogonal embedding directions.Source: Methods 18 Local explanation embedding… · pp. 33–34
Supplementary Figures
The supplementary figures show that Tree SHAP provides exact, consistent explanations for tree models while greatly reducing computational cost relative to model-agnostic approaches. They also document SHAP-based analyses of feature selection, clinical risk patterns, model monitoring, and attribution evaluation.
Supplementary Figures
- TreeExplainer’s Tree SHAP runs thousands of times faster than model-agnostic approaches and has no sampling variability because it is exact.Model-agnostic methods require sampling, while low-variance estimates can require days or years of CPU time; explaining the chronic kidney disease dataset takes 1.8 CPU days with SHAP values versus 3.3 CPU years with SHAP interaction values.Source: Figure 1 · p. 36, Supplementary Figures · p. 36
Tree SHAP
- Tree SHAP avoids Saabas inconsistencies by averaging feature contributions over all possible orderings, producing consistent attributions for nonlinear tree models.For multi-way AND functions, Tree SHAP distributes credit evenly among involved features, whereas Saabas gives little credit to features near the root; the attributions satisfy φ_i summing to f(x).Source: Figure 1 · p. 36, Tree SHAP · p. 37
- SHAP values improve feature selection over gain and permutation testing in simulated single-tree and ensemble settings, with all paired t-test P-values < 10^-7.The advantage appears for both product-style third-order interactions and minimum functions, because SHAP preserves consistency and accounts for high-order interactions.Source: Figure 7 · p. 41
- Supplementary analyses show that SHAP explanations capture clinically meaningful nonlinear structure, including heterogeneous blood-pressure effects, kidney-disease risk factors, procedure-specific trauma effects, and temporal degradation caused by changing atrial-fibrillation procedure durations.Local-explanation embeddings align with outcome-relevant dimensions more than unsupervised embeddings, while the hospital time-of-day analysis identifies recurring surgery scheduling times.Source: Figure 8 · p. 42, Figure 9 · p. 42, Figure 10 · p. 43
Related papers
This paper cites
- A Unified Approach to Interpreting Model Predictions2017
- XGBoost: A Scalable Tree Boosting System2016
- Axiomatic Attribution for Deep Networks2017
- Striving for Simplicity: The All Convolutional Net2014
- A Benchmark for Interpretability Methods in Deep Neural Networks2018
- DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network2016