PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Models

Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, Tom Claassen

arXiv:2011.01625v1cs.AIcs.LG

TL;DR

The paper addresses the problem that independence-based Shapley explanations can become undesirable when features are causally related. It develops causal Shapley values using Pearl’s do-calculus, showing how to incorporate causal structure while preserving desirable Shapley properties. The framework also separates direct and indirect effects and supports practical computation from partial causal information.

  • Problem

    Independence-based explanations can produce incorrect or counterintuitive attributions when input features are correlated or causally related.

  • Method

    The paper uses Pearl’s do-calculus to define causal Shapley values and computes them practically with causal chain graphs using partial causal information.

  • Results

    Causal Shapley values provide explanations that incorporate real-world causal relationships, distinguish total effects into direct and indirect contributions, and preserve desirable Shapley properties.

  • Takeaways & Limitations

    Causal Shapley values can give season more credit than temperature for seasonal differences in bike-rental predictions when causal structure is considered.

  • Takeaways & Limitations

    Asymmetric Shapley values can be sensitive to inserting causal links with zero strength, while explanation tools can also be over-trusted because explanations simplify complex systems.

Abstract

from arXiv · show

Shapley values underlie one of the most popular model-agnostic methods within explainable artificial intelligence. These values are designed to attribute the difference between a model's prediction and an average baseline to the different features used as input to the model. Being based on solid game-theoretic principles, Shapley values uniquely satisfy several desirable properties, which is why they are increasingly used to explain the predictions of possibly complex and highly non-linear machine learning models. Shapley values are well calibrated to a user's intuition when features are independent, but may lead to undesirable, counterintuitive explanations when the independence assumption is violated. In this paper, we propose a novel framework for computing Shapley values that generalizes recent work that aims to circumvent the independence assumption. By employing Pearl's do-calculus, we show how these 'causal' Shapley values can be derived for general causal graphs without sacrificing any of their desirable properties. Moreover, causal Shapley values enable us to separate the contribution of direct and indirect effects. We provide a practical implementation for computing causal Shapley values based on causal chain graphs when only partial information is available and illustrate their utility on a real-world example.

1 Introduction

Causal Shapley values address the counterintuitive explanations that can arise when features are dependent by incorporating real-world causal relationships through intervention-based conditioning. The framework preserves Shapley desiderata while supporting individual explanations and practical computation from partial causal information.

  • Motivation: Complex models can outperform simpler approaches but are harder to interpret, creating practical, ethical, and legal concerns in critical applications.Explainable AI seeks to improve the interpretability of such models.
  • Shapley-value explanations: Shapley values attribute the difference between an individual prediction and an average baseline to input features while satisfying desirable mathematical properties.They provide local explanations that can also support global understanding of model structure.
  • Limitations of existing approaches: Assuming feature independence can produce incorrect or counterintuitive explanations when features are correlated or causally related.Marginal Shapley values and many other explanation methods use this simplifying assumption.
  • Contributions: The framework explains total feature effects, separates direct and indirect contributions, and offers computation through causal chain graphs when only partial causal information is available.It also provides a model-agnostic explanation for individual predictions while incorporating causal relationships between features.
  • Desirable properties: Causal Shapley values preserve efficiency, linearity, null-player, and symmetry properties, whereas asymmetric weighting can sacrifice symmetry.The paper notes that symmetry remains available for marginal, conditional, and causal Shapley values alike.
  • Causal Shapley values: The paper defines causal Shapley values by replacing conventional observation-based conditioning with intervention-based conditioning using Pearl’s do-calculus.This retains dependencies among real-world features rather than collapsing interventional expectations to marginal expectations.

3 Decomposing Shapley values into direct and indirect effects

The causal interpretation decomposes each feature’s total Shapley contribution into direct and indirect effects. Direct effects hold dropped-feature distributions fixed, whereas indirect effects capture distributional changes caused by intervention.

  • Total-effect decomposition: For each permutation, the total effect is decomposed into direct and indirect effects using intervention-based expectations.The decomposition applies before averaging contributions across permutations.
  • Direct effect: The direct effect measures the prediction change from replacing stochastic feature X_i with x_i while leaving other out-of-coalition feature distributions unchanged.It isolates the effect of changing the feature value without propagating the additional intervention through those features.
  • Indirect effect: The indirect effect measures the expectation difference arising when other out-of-coalition feature distributions change because of do(X_i = x_i).Thus, it captures causal propagation through feature relationships.
  • Aggregation and comparison: Direct and indirect Shapley components are obtained by averaging their permutation-specific contributions, with conditional analogues formed by using observation-based conditioning.Marginal Shapley values have no indirect effect because they use no conditioning.

4 Shapley values for different causal structures

Across causal structures sharing the same observational distribution, Shapley explanations can differ substantially because causal relationships determine how direct and indirect effects should be attributed. Causal Shapley values use interventions to distinguish these effects and provide explanations that better reflect the underlying structure, while asymmetric and symmetric variants make different trade-offs.

  • Causal structures: Four two-feature causal models share the same observational distribution but encode different explanations for the dependency between X1 and X2.The models are a chain, fork, confounder, and cycle; the linear predictor is constructed to ignore one feature.
  • Explanation patterns: The examples reduce to three explanation patterns: direct effects only, evenly split indirect credit, or assigning all indirect credit to the root cause.Direct and indirect contributions sum to the prediction under the sufficiency property.
  • Limitations of noncausal values: Marginal Shapley values ignore feature dependence, while symmetric conditional values account for dependence but cannot distinguish the four causal structures or intervention consequences.In the chain example, marginal values give identical explanations for equal temperatures in different seasons; conditional values fail to reflect that some interventions leave another feature’s distribution unchanged.
  • Causal explanations: Causal Shapley values correctly treat confounder and cycle cases, whereas asymmetric conditional values incorrectly assign credit to X1 in the confounder case.Both symmetric and asymmetric causal variants distinguish these cases according to the paper’s causal analysis.
  • Causal explanations: In a chain, asymmetric causal values assign indirect credit to the root cause, whereas symmetric causal values average explanations across both intervention orderings.The preferred explanation depends on context: temporal root-cause reasoning favors the asymmetric version, while scenario averaging motivates the symmetric version.
  • Summary: Causal Shapley values incorporate real-world causal relationships, while asymmetric variants may suit clear temporal order and symmetric variants are less sensitive to model misspecification.The paper presents this as its summary comparison with marginal and conditional Shapley values.

6 Illustration on real-world data

The real-world bike-rental example shows that causal Shapley values can assign more credit to season than marginal values, while balancing seasonal and temperature effects for comparable days.

  • The implementation adapts SHAPR to sample from interventional conditional rather than observational conditional distributions.
  • Marginal Shapley values vary less for cosyear and more for temp, whereas causal values show the opposite pattern.
  • Causal Shapley values give season more credit for higher summer and lower winter bike rentals than marginal Shapley values.
  • For October 10 and December 3, similar temperatures accompany predicted bike counts of 6117 and 6241, respectively.
  • Marginal values emphasize temperature, asymmetric values emphasize root-cause season, and symmetric causal values balance both explanations.

7 Discussion

The discussion presents causal Shapley values as a practical extension of feature attribution that reflects causal structure and separates direct from indirect effects. It also notes computational costs, the need for practitioner validation, and broader risks of over-trusting explanations.

  • Causal Shapley values estimate each feature’s total effect on an individual prediction and decompose it into direct and indirect effects.
  • The method requires a partial causal order and an interpretation of dependencies among features on an equal footing.
  • Existing conditional-Shapley code can be generalized without additional computational complexity, but conditional and causal methods may require substantially more sampling cost than marginal values.
  • Additional user studies are needed to assess whether causal Shapley explanations meet practitioners’ needs in real-world settings.
  • Explanation tools simplify complex systems and can create unjustified confidence, including a transparency fallacy when users overestimate their understanding.
  • Causal explanations align with humans’ tendency to reason about causes and may therefore support more intuitive understanding of algorithmic predictions.

Supplement of “Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions

The supplement is identified as an arXiv preprint dated 3 November 2020.

  • The supplemental document is arXiv:2011.01625v1, dated 3 November 2020.

1 Do-calculus for cyclic graphs

The supplement describes do-calculus for cyclic graphs using intervention nodes, graph transformations, and σ-separation to map interventional distributions toward observational ones.

  • Intervened variables receive intervention nodes that point to them and clamp them to specified values.
  • The transformed graph removes incoming edges to intervened variables except edges from their intervention nodes.
  • σ-separation generalizes standard d-separation and is evaluated in graphs modified by interventions.
  • Do-calculus comprises three inference rules for mapping interventional and observational distributions.
  • The rules cover insertion or deletion of observations, exchange of actions and observations, and insertion or deletion of actions.
  • Repeated application can transform an interventional probability into an observational probability.
  • The illustrated causal models share an observational distribution while differing in causal structure and parameter assignments.

2 Shapley values for linear models

For linear models, causal Shapley values use intervention-based expectations to account for feature relationships and decompose contributions into direct and indirect effects.

  • The linear-model derivation uses models with identical observational distributions but different causal structures, showing why causal structure changes attribution.
  • Each feature contribution is the change in the value function before and after the feature is added under a permutation.
  • Causal Shapley contributions replace observational conditioning with intervention-based expectations derived using do-calculus.Conditional values use observational conditioning, while marginal values use the unconditional distribution.
  • The first feature’s contribution separates into direct and indirect effects, whereas the second feature’s contribution contains only a direct effect for the displayed permutation.
  • Asymmetric conditional Shapley values equal the symmetric values for confounders and cycles, but for a causal chain they retain only the causally ordered permutation.
  • For confounders, causal Shapley values coincide with marginal values, while for cycles they coincide with conditional values.

3 Proofs and corollaries on causal chain graphs

For causal chain graphs, the paper derives an interventional value function using do-calculus and gives a sampling algorithm that accommodates partial causal orderings and confounding assumptions.

  • Causal chain graphs represent partial orderings with directed edges between chain components and undirected edges within components.
  • The interventional formula requires specifying whether dependencies within each fully connected chain component arise from confounding or mutual interactions.
  • Do-calculus rules convert intervention-based conditioning into observational conditioning or marginal expectations according to graph structure.
  • Algorithm 1 estimates the value function by sampling from the interventional distribution across chain components in causal order.
  • With one confounded component, causal Shapley values equal marginal Shapley values; with one non-confounded component, they equal conditional Shapley values.
  • Restricting permutations to the causal ordering and assuming mutual interactions within components recovers asymmetric conditional Shapley values.

4 Additional illustrations on the bike rental data

On bike rental data, marginal and asymmetric Shapley values emphasize different feature groups, while asymmetric causal values closely match asymmetric conditional values.

  • Asymmetric causal Shapley plots are virtually indistinguishable from asymmetric conditional Shapley plots in this bike-rental analysis.
  • Marginal Shapley values strongly emphasize temperature and largely ignore seasonal variables.
  • Asymmetric Shapley values instead emphasize seasonal variables, especially cosyear, and place less emphasis on temperature variables.

5 Comparing symmetric and asymmetric Shapley values on the XOR problem

In an XOR experiment with a causal chain, symmetric causal Shapley values vary smoothly with causal strength, whereas asymmetric values are discontinuous at zero strength.

  • The XOR experiment evaluates Shapley values for binary features under different causal-strength settings using learned neural networks and theoretical values.
  • Causal value functions differ by assumed dependency structure: confounding yields marginal values, mutual interaction yields conditional values, and a chain uses intervention-specific expectations.
  • Simulations show the discontinuity most clearly in the difference between feature values; symmetric conditional values remain equal, while symmetric causal values diverge as ϵ increases.
  • For a causal chain X1 →X2, asymmetric values use only the causally ordered permutation, unlike symmetric values, which average both permutations.
  • Asymmetric conditional and causal values assign approximately φ1 ≈ 0 and φ2 ≈ 1/2ϵ − 1/2 for the chain X1 →X2.
  • At ϵ = 0, asymmetric values jump from symmetric φ1 = φ2 ≈ −1/2 to assigning all credit to the second feature.
  • As ϵ → 0, symmetric causal Shapley values converge to marginal values, avoiding the discontinuity caused by inserting a zero-strength causal link.

6 Shapley values for predicting dementia

Using ADNI data, the paper compares marginal, symmetric causal, and asymmetric causal Shapley values for dementia or mild cognitive impairment predictions. The asymmetric values focus more strongly on APOE4 as a root cause by assigning it indirect-effect credit through ABETA.

  • The causal Shapley computation uses a partial causal order placing gender, APOE4, age, and pteducat before ABETA, followed by FDG and PTAU.
  • The ADNI analysis predicts dementia or mild cognitive impairment using marginal, symmetric causal, and asymmetric causal Shapley values.Figure 4 presents these three explanation types for a multilayer perceptron trained on ADNI data.
  • Because feature dependencies are relatively weak, marginal and symmetric causal Shapley values are quite similar.
  • Asymmetric causal Shapley values separate APOE4 values more clearly than marginal and symmetric causal values.
  • The asymmetric values emphasize APOE4 as a root cause by assigning it the indirect effect transmitted through ABETA.
Loading 2011.01625v1…