PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability

Christopher Frye, Colin Rowat, Ilya Feige

arXiv:1910.06358v3stat.MLcs.AIcs.LG

TL;DR

Shapley values provide general, axiomatic explanations but ignore causal structure in data, limiting their faithfulness to known dependencies. The paper introduces Asymmetric Shapley values, which relax one axiom to incorporate partial causal knowledge and demonstrates applications in explanations, fairness, time series, and feature selection.

  • Problem

    Shapley values offer general, rigorous model-agnostic explanations but ignore causal structure, while complete causal models are rarely available for high-dimensional data.

  • Method

    Asymmetric Shapley values generalize Shapley values by relaxing one of four axioms, allowing any amount of known causal structure to enter model explanations.

  • Results

    ASVs support causal model explanations, an unambiguous test for unfair discrimination, sequential time-series explanations, and feature selection without model retraining.

  • Takeaways & Limitations

    ASVs can extract meaningful causal insights from explanations with only a crude causal understanding and can expose unfairness after resolving variables are accounted for.

  • Takeaways & Limitations

    Model explanations should not replace domain expertise or justify regulatory approval of consequential algorithms on their own.

Abstract

from arXiv · show

Explaining AI systems is fundamental both to the development of high performing models and to the trust placed in them by their users. The Shapley framework for explainability has strength in its general applicability combined with its precise, rigorous foundation: it provides a common, model-agnostic language for AI explainability and uniquely satisfies a set of intuitive mathematical axioms. However, Shapley values are too restrictive in one significant regard: they ignore all causal structure in the data. We introduce a less restrictive framework, Asymmetric Shapley values (ASVs), which are rigorously founded on a set of axioms, applicable to any AI system, and flexible enough to incorporate any causal structure known to be respected by the data. We demonstrate that ASVs can (i) improve model explanations by incorporating causal information, (ii) provide an unambiguous test for unfair discrimination in model predictions, (iii) enable sequentially incremental explanations in time-series models, and (iv) support feature-selection studies without the need for model retraining.

1 Introduction

The paper motivates model-agnostic explainability as important for AI safety and development, then introduces Asymmetric Shapley values to incorporate causal knowledge while retaining an axiomatic framework.

  • Explainability supports AI safety and iterative system development by revealing how models work, fail, and can improve.
  • Model-agnostic methods offer broad applicability and a common language for comparing explanations across model types.
  • Shapley values provide principled local explanations that capture feature interactions, but they also have computational, data, causal, and interpretability shortcomings.
  • Causal understanding is difficult because causal graphs are hard to ascertain, discovery methods vary, and complete causal models are rarely known for high-dimensional data.
  • ASVs relax one of Shapley’s four axioms to incorporate partial causal knowledge and support causal explanations, fairness testing, sequential time-series importance, and feature selection.

2 Shapley values for model explainability

Shapley values explain model predictions by treating features as cooperative players whose contributions are averaged across coalitions and permutations. The framework is axiomatic and supports local and global explanations, but unconditional feature marginalisation can generate off-manifold data, while on-manifold computation is difficult in high dimensions.

  • Shapley values assign credit for a model output by averaging each feature’s marginal contribution across all feature orderings.
  • In supervised learning, features act as players whose coalition value is the model’s predicted class probability, yielding local explanations for individual predictions.
  • The standard value function unconditionally marginalises out-of-coalition features using samples from the data distribution.
  • Shapley values are uniquely characterized by efficiency, linearity, nullity, and symmetry, although these axioms can be too restrictive for model explainability.
  • Unconditional marginalisation may combine incompatible features off the data manifold, motivating conditional on-manifold value functions that are nontrivial to compute for high-dimensional data.
  • Global Shapley values average local explanations over the data distribution and can represent the portion of model accuracy attributable to each feature.

3 Asymmetric Shapley values

Asymmetric Shapley values relax symmetry while preserving the other core Shapley axioms, allowing known causal structure—however incomplete—to shape model explanations. By weighting feature orderings, ASVs can distinguish causal ancestors from descendants and span explanations from data-agnostic to causally informed.

  • ASVs relax Shapley symmetry to incorporate causal knowledge while retaining the framework’s axiomatic foundation.The approach preserves Efficiency, Linearity, and Nullity, while allowing Symmetry to be relaxed.
  • 3.1 Argument against symmetry: Redundant features can receive unequal attribution when causal knowledge identifies one as an ancestor of the other.A known deterministic causal ancestor can receive all importance while its descendant receives none.
  • 3.2 Asymmetric Shapley values: ASVs use a distribution over feature orderings to determine each feature’s impact under specified or unspecified related features.If i always precedes j, i is evaluated while j is unknown, whereas j is evaluated after i is specified.
  • 3.2 Asymmetric Shapley values: Distal causal explanations prioritize ancestors, while proximate explanations prioritize descendants through causal or anti-causal orderings.The distal approach attributes ancestor effects before descendants are specified; the proximate approach reverses that ordering.
  • 3.3 A data agnosticism continuum: ASVs reduce to ordinary Shapley values without causal information and can incorporate partial knowledge without requiring a complete causal graph.A fully specified graph may restrict the weighting to one ordering, while limited knowledge imposes weaker ordering constraints.

4 Applications and experimental results

Experiments show that ASVs incorporate partial causal knowledge into explanations, quantify unresolved discrimination, respect temporal ordering, and support feature selection without repeated retraining.

  • Causality-based model explanations: ASVs prioritize known causal ancestors before descendants while preserving equal total attribution under the Efficiency axiom.For Census Income, ancestors included age, sex, native country, and race; descendants included marital status, education, and other features.
  • Causality-based model explanations: Marital status and relationship receive equal on-manifold Shapley values because their tight correlation is incorporated, unlike the off-manifold baseline’s stronger marital-status dependence.The off-manifold value function evaluates unrealistic feature splices, whereas the on-manifold value function respects correlations.
  • Causality-based model explanations: 3% of the model’s 85% accuracy can be attributed to sex, which receives the largest ASV despite a relatively small Shapley value.The class balance was 76/24, limiting attribution of most accuracy to any individual feature.
  • Causal explanations of unfair discrimination: ASVs measure unresolved discrimination by attributing sensitive attributes’ influence after resolving variables are already known, assigning gender vanishing attribution in fair data and exposing unfair data.Synthetic admissions data used department as the resolving variable and gender as the sensitive attribute; the unfair set had 64% versus 36% admission rates for men and women.
  • Data types with intrinsic ordering: ASVs provide sequential feature importance by measuring the additional predictivity of time step t after earlier steps are known, concentrating importance near the beginning of EEG signals.ASVs dropped by roughly 10^2 after 25 of 178 steps, indicating little additional predictive power from steps 26–178 once steps 1–25 are known.
  • Precise, verifiable feature selection: The close relationship between ASVs and empirical accuracies supports feature selection without retraining models on every feature subset.ASVs have a direct interpretation as the accuracy achievable using a subset of the data’s features.

5 Conclusion

The paper introduces Asymmetric Shapley values as a principled, model-agnostic generalization that incorporates causal information and applies across several explainability tasks.

  • ASVs generalize Shapley values to incorporate causal information underlying the model’s data.
  • The framework supports causal model explanations, fairness testing, sequential time-series explanations, and feature selection without model retraining.
  • The authors argue that lowering barriers to causal explainability could contribute to better models and more trustworthy AI systems.

Broader impact

ASVs extend model-agnostic explainability to varied machine-learning models and support investigation of nuanced unfairness, while requiring verified causal knowledge and resisting blind trust in explanations.

  • ASVs can be applied to a wide variety of machine-learning models, creating potential for broad impact.
  • ASVs allow users to investigate nuanced causal notions of unfairness in models.
  • Explainability should not substitute for domain expertise or justify regulatory approval of consequential algorithms by itself.
  • Practitioners should verify causal information with domain experts because incorrect causal relationships negate ASVs’ benefits.

A Examples in two dimensions

In two dimensions, ASVs weight feature permutations according to causal knowledge, whereas ordinary Shapley symmetry weights permutations evenly.

  • For two features, local Shapley values are obtained from the two permutation-specific terms in the value-function construction.
  • Shapley symmetry requires the permutations (12) and (21) to receive equal weights.
  • ASVs let practitioners choose permutation weights w(π) according to the application, subject to w(12) + w(21) = 1.
  • With causal structure Fig. 3(a), the example sets w(12) = 1 and w(21) = 0.
  • v(1) reports x1’s impact over the model’s average output, while v(2) reports x2’s marginal effect after receiving x1.
  • For Fig. 3(b), both permutations receive weight 1/2; Fig. 3(c) uses w(12) = 1 and w(21) = 0.

B.1 Experiment on Census Income data

The Census Income experiment explains a dense neural network using three global Shapley variants, estimated by Monte Carlo under off-manifold or on-manifold value functions.

  • The experiment uses Census Income data, a dense network with two hidden layers of 100 units, and a 75/25 train/test split.
  • The model achieves 84.7% test-set accuracy, and the experiment’s results are computed on the test set.
  • Figure 1(a) compares Off manifold, On manifold, and ASV variants, each aggregating local values.
  • All three variants are estimated with 10^6 Monte Carlo samples, with the off-manifold result showing a standard-error bar.
  • The on-manifold and ASV estimates use a VAE-based method to compute p(x′|xS).

B.2 Experiment on synthetic college admissions data

The experiment uses synthetic fair and unfair college-admissions data sets differing in whether unreported referrals affect admission, then evaluates fitted neural networks on held-out data.

  • The fair and unfair data sets both encode gender as a binary variable, with women represented by X1 = 0 and men by X1 = 1.
  • Women predominantly apply to department X3 = 1, whereas men predominantly apply to X3 = 0.The conditional probabilities are 0.8 for women and 0.2 for men applying to X3 = 1.
  • In the fair data set, admission depends on test score X2 and department choice X3 through sigmoid(x2 + 2 x3 −1).The coefficient on X3 makes department X3 = 1 more competitive.
  • In the unfair data set, admission additionally depends on unreported referrals X4, which are more prevalent among men.Referral prevalence is 1/3 for women and 2/3 for men; X4 is omitted from the reported data but affects admissions.
  • 73.6% and 73.2% test-set accuracy were achieved by the fair and unfair models, respectively, using densely connected networks trained with a 75/25 split.Each model used two hidden layers of 10 units, and results were computed on held-out test sets.
  • Global ASVs were computed with a VAE-based on-manifold method using dense encoder, decoder, and masked-encoder networks without hyperparameter tuning.Each network had two hidden layers of 20 units; bar lengths summarize means over 10^6 Monte Carlo samples with standard-error bars.

B.3 Experiments on Seizure Recognition data

The seizure-recognition experiments use an LSTM classifier and compare explanation values with cumulative and empirically retrained accuracy measures on held-out data.

  • The experiments use the UCI Epileptic Seizure Recognition data and an LSTM model with a 20-dimensional hidden state.The model was trained with Adam, batch size 128, early stopping, and a 75/25 train/test split.
  • 98.3% test-set accuracy was achieved despite the data having an 80/20 class balance.All results in Section 4.3 were computed on the test set.
  • Global Shapley values were estimated from 10^5 Monte Carlo samples using the off-manifold value function, while global ASVs used a VAE-based on-manifold value function.The corresponding recurrent encoder, decoder, and masked encoder used LSTMs with 20-dimensional hidden states.
  • Cumulative values were compared with empirical accuracies from models retrained using only time steps t′ ≤ t.For each t, the retrained model’s restricted-input accuracy difference was computed relative to using no time steps.
Loading 1910.06358v3…