Source-linked AI summary
Explaining a Series of Models by Propagating Shapley Values
Hugh Chen, Scott M. Lundberg, Su-In Lee
TL;DR
Local attribution methods either incur high computational costs or cannot explain distributed series of proprietary models, limiting explanations where transparency matters. DeepSHAP connects local attribution propagation through model series to Shapley values and averages single-baseline attributions. Across biological, health, image, and financial applications, it provides useful explanations and enables attribution in distributed model settings, while lacking guarantees for some desirable axioms.
Problem
Existing local attribution methods are costly or generally unable to explain distributed series of proprietary models, where transparency is critical because opaque scores can hide discrimination or unfair practices.
Method
DeepSHAP propagates local feature attributions through complex model series using Shapley-value connections, generalized rescaling, and averages over single-baseline attributions.
Results
DeepSHAP provides useful explanations across biological, health, image, and financial datasets and enables explanations of distributed series of models.
Takeaways & Limitations
DeepSHAP supports explanations of groups of features, model loss, complex pipelines, and distributed proprietary consumer scores.
Takeaways & Limitations
DeepSHAP is not guaranteed to satisfy desirable axioms such as implementation invariance, so axiomatic approaches or interpretable models may be preferable for high-stakes decisions.
Abstract
from arXiv · showhide
Local feature attribution methods are increasingly used to explain complex machine learning models. However, current methods are limited because they are extremely expensive to compute or are not capable of explaining a distributed series of models where each model is owned by a separate institution. The latter is particularly important because it often arises in finance where explanations are mandated. Here, we present DeepSHAP, a tractable method to propagate local feature attributions through complex series of models based on a connection to the Shapley value. We evaluate DeepSHAP across biological, health, and financial datasets to show that it provides equally salient explanations an order of magnitude faster than existing model-agnostic attribution techniques and demonstrate its use in an important distributed series of models setting.
1 Introduction
DeepSHAP addresses the cost and model-access limitations of local feature attribution for complex and distributed model series. It connects attribution rules to Shapley values, supports multiple model types and feature groups, and evaluates these capabilities across several real-world datasets.
- Motivation: Model-agnostic methods can explain model series but have variable sampling estimates and high computational costs, while model-specific methods are faster but generally cannot explain series.These limitations motivate a method that combines tractability with support for complex pipelines.
- Contribution: DeepSHAP is a local feature attribution method designed to explain complex model series faster than model-agnostic methods.Its theoretical basis connects the method to the Shapley value and its desirable axioms.
- Contribution: DeepSHAP connects existing attribution rules to the Shapley value using an interventional conditional expectation set function and a flat causal graph.The framework includes a decomposition of ICE Shapley values into averages of single-baseline attributions.
- Contribution: The generalized rescale rule propagates attributions through complex series composed of linear, deep, and tree models while enforcing efficiency at each layer.This extends attribution beyond individual neural-network layers to broader model compositions.
- Contribution: Group rescale rules propagate attributions to feature groups, improving explanations for models with many features.The paper also identifies baseline choice as a parameter that changes the scientific question and shows that multiple baselines can avoid single-baseline bias under certain assumptions.
- Evaluation: DeepSHAP is evaluated qualitatively and quantitatively on biological, health, image, and financial datasets, including a distributed setting where models belong to separate institutions.The distributed setting is important for consumer scores because opaque proprietary models can hide discrimination or unfair practices.
2 Generalizing DeepSHAP local explanations
DeepSHAP generalizes local explanations from individual deep models to compositions of models by averaging single-baseline attributions and propagating them through the series. Its layer-wise efficient propagation uses model-appropriate attribution rules and supports mixed model types.
- Baseline distributions: Interventional Shapley attributions for an explicand are obtained by averaging single-baseline feature attributions over baseline samples.A single-baseline attribution compares one explicand with one baseline sample; the multi-baseline formulation addresses nonlinear models under the stated assumptions.
- Model series: DeepSHAP generalizes DeepLIFT’s rescale rule beyond neural-network layers to propagate attributions through compositions of models while guaranteeing layer-wise efficiency.The framework represents a series as composed functions with intermediate models.
- Baseline distributions: DeepSHAP’s baseline distribution is an important parameter because it changes the scientific question implicit in local feature attributions.The method uses explicands, baseline samples, and a model composed of a series of models.
- Model-specific rules: Attributions for stacked models use DeepLIFT for deep models, interventional TreeSHAP for tree models, and exact computation for linear models.Intermediate attributions provide feature attributions for each model’s input features, with the raw-space attribution obtained at the first stage.
3 Incorporating a baseline distribution
DeepSHAP uses baseline distributions to reduce attribution bias and make explanations answer contrastive questions about meaningful subpopulations. Baseline selection therefore changes both the interpretation and usefulness of feature attributions.
- Baseline distributions: DeepSHAP evaluates deep models by connecting feature attributions to interventional conditional expectations.The experiments examine how different baseline distributions affect explanations.
- Baseline distributions: A single all-black baseline can bias image attributions, whereas averaging over 1000 random training images produces more sensible attribution patterns.The single-baseline explanation assigns little attribution mass to prevalent dark pixels despite their qualitative importance.
- Subpopulation contrasts: A baseline distribution changes the question answered by an attribution, as shown by contrasting an older male with the general population versus older males.The older-male baseline removes much of the apparent importance of sex and age while increasing the apparent importance of physical activity.
- Subpopulation contrasts: K-means clustering on meaningful, non-modifiable features such as age and gender provides a principled way to select interpretable subpopulation baselines.The paper uses 8-means clustering on age and sex to identify baseline groups.
- Subpopulation contrasts: Clustering-based baselines can reduce the importance of cluster-defining features, potentially limit off-manifold baseline evaluations, and support natural contrastive scientific questions.These advantages are presented as motivations for choosing baselines within reasonable subpopulations.
4 Explaining a series of models
DeepSHAP extends attribution propagation across mixed-model pipelines and supports higher-level explanations for biological, health, image, and financial applications. Across these settings, it provides useful explanations with substantially lower computational cost and can explain distributed proprietary pipelines without requiring model sharing.
- Method: DeepSHAP propagates attributions through series of linear, deep, and tree models while enforcing efficiency at each layer.This generalized rescale rule is the core extension for explaining complex mixed-model pipelines.
- 4.1 Group attributions identify meaningful gene sets: Group attributions transfer gene-level importance to gene sets, enabling higher-level interpretation of Alzheimer’s disease and breast cancer predictions.Residuals summarize contributions from genes absent from analyzed gene sets and output variation not described by those pathways.
- 4.2 Loss attributions provide insights to model behavior: Loss attributions reveal model-performance behavior that output attributions can miss, including covariate-shift effects and samples where blood lead increases test loss.The paper compares loss and output attributions across NHANES training and test distributions.
- 4.3 Explaining deep image feature extractors: DeepSHAP is an order of magnitude faster than model-agnostic approaches and performs comparably to IME while outperforming KernelSHAP and LIME in image ablations.The image pipeline combines a CNN feature extractor with a gradient boosted tree classifier.
- 4.4 Explaining distributed proprietary models: DeepSHAP attributes final risk predictions to original fraud, credit, and bank features without requiring institutions to share proprietary models.The generalized rescale rule uncovers variables hidden behind intermediate fraud and credit scores.
- 4.4 Explaining distributed proprietary models: In the HELOC pipeline, DeepSHAP matches competitive ablation performance while running several orders of magnitude faster than KernelSHAP and IME.It also runs an order of magnitude faster and is more performant than LIME.
5 Discussion
DeepSHAP is positioned as a practical tool for explaining diverse model pipelines, including applications involving high-dimensional features, transformed outputs, neural-network feature extraction, and distributed consumer scores. Its main boundary is that practical tractability does not guarantee the axioms satisfied by exact attribution methods.
- Practical applications: DeepSHAP supports gene set attribution when the number of features far surpasses the number of samples, using grouped attributions to maintain efficiency.The grouping rule aggregates feature attributions to higher-level feature groups.
- Practical applications: Explaining logistic-model probabilities or losses yields interpretable attributions for feature effects, performance debugging, and covariate-shift identification.The discussion contrasts probability explanations with log-odds explanations and identifies loss explanations as useful for debugging.
- Practical applications: DeepSHAP remains computationally tractable for neural-network feature-extraction pipelines and makes intermediary attributions interpretable at each stage.The method propagates attributions while satisfying efficiency at each step.
- Consumer scoring: Distributed proprietary consumer scores obstruct transparency and can conceal discrimination through features correlated with protected factors such as race.DeepSHAP provides attributions for both consumer scores and the original features used by them.
- Relation to prior work: The discussion emphasizes mixed model types, distinguishing this evaluation scope from prior work focused specifically on deep models.Related work is described as evaluating the rescale rule or DeepSHAP primarily for deep-network explanations.
- Limitations: DeepSHAP is not guaranteed to satisfy desirable attribution axioms such as implementation invariance, limiting its suitability for some high-stakes decisions.The paper suggests axiomatic methods or interpretable models when high-stakes decisions matter, while noting their infeasibility in some distributed settings.
6 Methods
The methods adapt Shapley-based feature attribution to model inputs, baseline distributions, composed models, intermediate layers, and feature groups. They connect DeepLIFT-style rules to interventional Shapley values while preserving efficiency under stated conditions.
- Adapting the Shapley value: Shapley values allocate credit by averaging each feature’s marginal contribution across permutations, but ML models require a lift because they generally need values for every feature.The paper focuses on local attributions for an explicand and defines a new set function to apply Shapley values to model inputs.
- Adapting the Shapley value: The interventional conditional-expectation lift breaks dependence between selected and remaining features, avoiding the need to estimate the joint density of X.The paper uses this lift because it is most closely related to DeepSHAP, while noting tradeoffs with the observational approach.
- Baseline distributions: Interventional Shapley values decompose into averages of single-baseline attributions drawn from an empirical baseline distribution, except that the mean baseline is equivalent for linear models.Using many training-data baselines compares the explicand with each baseline and makes the baseline distribution an interpretable parameter.
- Series of models: The generalized rescale rule propagates approximate interventional Shapley values through arbitrary compositions while enforcing efficiency at each intermediate model.The approach uses appropriate efficient attribution rules for deep, tree, and linear models, and produces meaningful intermediate attributions; it is exact when every composed model is linear.
- Connecting DeepLIFT’s rules: The Rescale and RevealCancel rules are respectively 1-partition and 2-partition approximations to interventional Shapley values for a nonlinear function applied to a linear function.More general k-partition variants are possible but incur exponentially larger costs in K.
- Feature groups: The group rescale rule sums attributions for disjoint groups covering all input features, supporting higher-level explanations when models have many features.The paper motivates this setting with high-dimensional gene-expression data and states that group attributions better explain models with many features.
A.1.1 NHANES I
NHANES I is a national longitudinal study of randomly sampled U.S. individuals, with demographic and socioeconomic data. Its follow-up study examines clinical, nutritional, and behavioral factors and tracks cohort outcomes.
- NHANES I is a national longitudinal study conducted on a random sample of individuals from the United States.
- The survey investigates demographic and socioeconomic variables.
- NHEFS measures additional variables and collects death certificates to study clinical, nutritional, and behavioral factors.
- NHEFS follows NHANES I participants aged 25–74 who completed a medical examination.
A.1.2 NHANES 1999-2014
NHANES continually surveys subsamples of the U.S. civilian noninstitutionalized population in two-year cycles. This dataset uses eight cycles from 1999–2014 and retains only consistently collected variables.
- NHANES collects information from subsamples of the U.S. civilian noninstitutionalized population in two-year cycles.
- The dataset covers eight release cycles from 1999–2014.
- Because features do not match across cycles, the analysis uses only variables collected consistently.
A.1.3 ROSMAP Alzheimer’s Gene Expression
ROSMAP combines two longitudinal aging cohorts with complementary participant profiles to study aging and Alzheimer’s disease. The analysis uses ChIP-seq gene-expression data to predict Alzheimer’s disease status.
- ROS is a longitudinal cohort study of aging and Alzheimer’s disease involving clinical analysis and brain donation.
- MAP is a longitudinal epidemiologic cohort study of common chronic conditions of aging.
- MAP complements ROS by enrolling participants with wider life experiences and socioeconomic status.
- ChIP-seq gene-expression data from ROS and MAP are used to predict Alzheimer’s disease status.
A.1.4 METABRIC Breast Cancer Gene Expression
METABRIC analyzes genomic and transcriptomic information from 995 breast cancer tumors. This analysis uses only transcriptomic information to predict tumor status.
- METABRIC analyzes genomic and transcriptomic information from 995 breast cancer tumors.
- The analysis uses transcriptomic information exclusively to predict tumor status.
A.1.5 CIFAR
The experiments span image, health, biological, and financial prediction settings, including pipelines that combine neural networks, tree models, and meta-learners. DeepSHAP uses baseline distributions and layerwise propagation to produce attributions for these model series, while stacked explanations expose contributions at both meta-model and original-input levels.
- Datasets and models: The evaluation covers CIFAR10, MNIST, NHANES mortality prediction, Alzheimer’s and breast-cancer classification, and HELOC repayment prediction.The supplied passages describe these datasets and modeling settings across image, health, biological, and financial applications.
- Experimental setup: DeepSHAP explanations use baseline distributions, including 1000 randomly sampled training images for multi-baseline image experiments.The experiments compare single black-image baselines with distributions of randomly sampled training examples and use shared baseline distributions across attribution methods.
- Feature extraction: A CNN’s extracted MNIST features give gradient-boosted trees much higher accuracy than raw digits for the same number of estimators.The experiment varies the number of tree estimators to assess underparameterized tree performance with raw versus CNN-extracted features.
- Stacked generalization: Stacked generalization with a gradient-boosted-tree meta-model outperforms average voting and logistic-regression weighting for NHANES mortality prediction.The framework also attributes importance to base-learners and propagates meta-level explanations back to the original input space.
- Generalized rescale rule: For compositions of linear models, the generalized rescale rule coincides with interventional Shapley values.The equivalence follows because the composition of linear models is linear.