PaperlayerView exact revision on arXiv

Source-linked AI summary

Shapley Flow: A Graph-based Approach to Interpreting Model Predictions

Jiaxuan Wang, Jenna Wiens, Scott Lundberg

arXiv:2010.14592v3cs.LGstat.ML

TL;DR

Feature dependencies challenge importance estimates, while graph-based attribution can leave the full system unexplained when credit is assigned only to nodes. Shapley Flow generalizes Shapley value axioms to assign credit to causal-graph edges, with a unique allocation satisfying relaxed axioms across explanation boundaries. The paper reports that Shapley Flow unifies three prior Shapley-based methods and supports joint understanding of direct and indirect feature impacts, with examples on real data.

  • Problem

    Feature dependencies challenge importance estimates, while graph-based attribution can leave the full system unexplained when credit is assigned only to nodes.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Method

    Shapley Flow generalizes Shapley value axioms to assign credit to causal-graph edges, with a unique allocation satisfying relaxed axioms across explanation boundaries.

    Source: §3.1 · pp. 4–5

  • Results

    The paper reports that Shapley Flow unifies three prior Shapley-based methods and supports joint understanding of direct and indirect feature impacts, with examples on real data.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Shapley Flow shows both direct and indirect feature impacts and traces those impacts to their source nodes.

    Source: §10.4 · p. 19

    Shapley Flow relies on access to a complete causal graph, and the authors identify partially defined graphs as an open question.

    Source: §5 · p. 9

Abstract

from arXiv · show

Many existing approaches for estimating feature importance are problematic because they ignore or hide dependencies among features. A causal graph, which encodes the relationships among input variables, can aid in assigning feature importance. However, current approaches that assign credit to nodes in the causal graph fail to explain the entire graph. In light of these limitations, we propose Shapley Flow, a novel approach to interpreting machine learning models. It considers the entire causal graph, and assigns credit to \textit{edges} instead of treating nodes as the fundamental unit of credit assignment. Shapley Flow is the unique solution to a generalization of the Shapley value axioms to directed acyclic graphs. We demonstrate the benefit of using Shapley Flow to reason about the impact of a model's input on its output. In addition to maintaining insights from existing approaches, Shapley Flow extends the flat, set-based, view prevalent in game theory based explanation methods to a deeper, \textit{graph-based}, view. This graph-based view enables users to understand the flow of importance through a system, and reason about potential interventions.

1 Introduction

Feature correlations and causal dependencies complicate feature attribution: ignoring indirect effects or assigning credit only to graph nodes can leave important impacts unexplained. Shapley Flow instead assigns credit to edges, aiming to represent direct and indirect effects across the causal graph.

1 Introduction

  • When feature dependencies are ignored, the season can receive zero importance even though changing it would affect the predicted probability that the pavement is slippery.The example’s model takes season, rain, sprinkler, and pavement as inputs.Source: §1 · p. 1
  • Existing causal node-based methods can miss either direct or indirect effects: ASV gives season all the credit, while Causal Shapley values divide credit between upstream and downstream features.In the sprinkler example, this can leave variables such as the sprinkler or pavement without credit despite their effect on the outcome.Source: §1 · p. 2
  • Shapley Flow assigns credit to causal-graph edges to account for both direct and indirect effects, rather than choosing a single explanation boundary.The introduction presents this as a comprehensive approach to interpreting a model or system.Source: §1 · p. 2
  • The paper presents Shapley Flow as a graph generalization of Shapley feature attribution and says it unifies three earlier game-theoretic approaches.The authors describe its edge-credit allocation as the unique one conforming to several natural axioms.Source: §1 · p. 2
  • Examples on real data are used to demonstrate how Shapley Flow facilitates understanding feature importance.Figure 2 compares attribution outputs and the explanation boundaries used by the methods.Source: §1 · p. 2, Figure 2 · p. 2

2 Problem Setup & Background

The setup measures an input’s effect relative to a background sample and seeks edge credits that explain the output difference. The section then uses explanation boundaries to frame attribution choices and motivates assigning credit across graph edges.

2.1 Problem Setup

  • The target is the difference f(x) − f(x′), where x is the sample being explained and x′ is a background sample that may represent typical feature values.The formalism also allows multiple background values or a background distribution.Source: §2.1 · pp. 2–3
  • Given a causal graph, the method assigns credit to each edge so the credits collectively explain the output difference; node importance can be recovered by summing outgoing-edge credits.This edge-based assignment matches classic Shapley axioms when one source node connects directly to all input features.Source: §2.1 · p. 2

2.2 Feature Attribution with a Causal Graph

  • A causal graph alone does not resolve attribution: summarizing many counterfactuals into a comprehensive account remains nontrivial, and graph nodes may themselves be black-box models.The paper addresses this by generalizing game-theoretic fairness principles to graphs.Source: §2.2 · p. 3
  • An explanation boundary is a graph cut separating context nodes D from model nodes F; its crossing edges represent the model’s direct inputs.Different boundaries correspond to different choices about which part of the graph is treated as the model.Source: §2.2 · p. 3
  • Considering all explanation boundaries simultaneously through edge attribution captures direct and indirect effects without forcing the user to select one boundary.The paper contrasts this with node-based methods that must split credit between parents and children.Source: §2.2 · p. 3

3 Proposed Approach: Shapley Flow

Shapley Flow assigns credit to causal-graph edges, rather than nodes, to represent both direct and indirect effects. It defines edge attributions through history-based games and axioms generalized across explanation boundaries.

3 Proposed Approach: Shapley Flow

  • Shapley Flow assigns credit to causal-graph edges, providing a system-level view that can capture both direct and indirect effects.Edge credits also help users reason about what would happen if a causal link breaks.Source: §3 · p. 4, §3.1 · p. 4

3.3 Axioms

  • Efficiency, dummy, and linearity extend Shapley value axioms, while boundary consistency requires shared-edge credits to agree across boundaries using compatible histories.Boundary consistency restricts attention to histories that can be expanded into detailed histories consistent with the boundaries.Source: §3.1 · p. 4, §3.3 · p. 5

3.1 Assigning Credit to Edges: Intuition

  • Edge removal means a target continues to use its source’s background value, and credit averages the edge’s output difference over boundary-consistent histories.Considering histories avoids assigning both inputs zero credit in an OR example when either edge alone is removed.Source: §3.1 · pp. 4–5

3.2 Model explanation as value assignments in games

  • The explanation-boundary game treats external edges as players, histories as ordered edge transmissions, and coalition payoffs as evaluations of the boundary model.Unlike ordinary cooperative games, the ordering of players matters in this message-passing setup.Source: §3.2 · p. 5

3.4 Shapley Flow is the unique solution

  • The unique solution averages path attributions across depth-first-search configurations, then sums the path credits onto each edge.For each configuration, output differences are credited along source-to-sink search paths.Source: §3.4 · p. 6
  • On flat graphs, edge attributions equal SHAP feature attributions; with modeled dependencies they match ASV source-node attributions, and on trees they relate to the Owen value.The Owen value uses the tree as its coalition structure.Source: §3.4 · pp. 6–7
  • Shapley Flow is implementation invariant, conserves attribution between each node’s incoming and outgoing edges, and can explain arbitrary non-differentiable pipelines.Implementation invariance follows from boundary consistency; conservation follows from efficiency and boundary consistency.Source: §3.4 · p. 7

4 Practical Application

The section applies Shapley Flow to synthetic and nutrition data, using a provided causal graph to examine direct and indirect effects. Linear-model checks and nonlinear examples illustrate how its edge attributions compare with existing methods.

4.1 Experimental Setup

  • The nutrition-data causal graph is an illustrative oversimplification, built from assumed dependencies, deterministic relationships, and latent groupings.The method assumes a causal graph; the authors note that some relationships, including those between source variables and biomarkers, are more complex.Source: §4.1 · p. 7

4.2 Baselines

  • The comparison covers independent SHAP, on-manifold SHAP, and ASV, with 10,000 sampled orderings and a shared background sample across methods.ASV attributions for source nodes are represented by summing Shapley Flow's outgoing-edge attributions.Source: §4.2 · pp. 7–8

4.3 Sanity checks with linear models

  • Shapley Flow has zero mean absolute error for both direct and indirect effects in the linear-model checks.Errors are evaluated against ground-truth attributions over 1,000 test examples and all graph nodes.Source: §4.3 · p. 8, Table 1 · p. 8

4.4 Examples with non-linear models

  • Independent SHAP assigns age less importance than ASV because it misses age’s indirect effect through blood pressure.Shapley Flow traces that indirect path in the nutrition example.Source: §4.4 · p. 8
  • On-manifold SHAP can make age appear protective because it uses observational relationships rather than causal ones.When serum magnesium updates before age, the conditional expected age exceeds the foreground age, producing a decrease in the output when age updates.Source: §4.4 · p. 8
  • ASV assigns serum protein’s credit to its upstream blood-protein node, obscuring the possible impact of intervening on serum protein.The example contrasts ASV’s node attribution with the direct impact visible in Shapley Flow.Source: §4.4 · pp. 8–9
  • Shapley Flow shows direct and indirect feature impacts, tracing effects to source nodes and supporting reasoning about changes to causal links.The authors note that causal links can change or break over time.Source: §4.4 · p. 9

5 Discussion and Conclusion

Shapley Flow extends Shapley-value axioms to causal graphs and assigns credit to edges. The authors present it as a way to understand direct and indirect feature impacts, while noting that it relies on access to a complete causal graph.

5 Discussion and Conclusion

  • Shapley Flow extends classic Shapley-value axioms to produce a unique edge-attribution method that unifies three prior Shapley-based approaches.The authors say it jointly represents direct and indirect feature impacts and applies to both black-box and interpretable models.Source: §5 · p. 9
  • The method assumes access to a complete causal graph; working with partially defined graphs remains an open research question.The authors also identify handling complex causal graphs and using the method for feature selection as future questions.Source: §5 · p. 9

6 Explanation boundary for on-manifold methods without a causal graph

The on-manifold perturbation method can be represented with an auxiliary imputed variable for each feature. Perturbing a feature updates model inputs to respect data correlations, while the resulting credit is split among correlated features.

6 Explanation boundary for on-manifold methods without a causal graph

  • On-manifold perturbation is represented by auxiliary imputed variables that update model inputs to preserve data correlations after a feature is perturbed.When a feature has not been perturbed, other imputed variables treat it as missing and sample conditionally on non-missing predecessors.Source: §6 · p. 12
  • The method splits credit equally among correlated features, even when credit to some features does not reflect model f but does reflect the model F.The figure distinguishes the causal links from the black edges used by on-manifold perturbation.Source: §6 · p. 12

7 The Shapley Flow algorithm

The algorithm can approximate Shapley Flow by sampling random edge orderings and averaging their results.

7 The Shapley Flow algorithm

  • Instead of evaluating every edge ordering, approximate Shapley Flow by averaging over sampled random orderings.The paper’s pseudocode highlights this approximation option.Source: §7 · p. 12

8 Shapley Flow’s uniqueness proof

The uniqueness proof shows that at most one edge attribution can satisfy the axioms, then establishes that Shapley Flow satisfies them. It relies on boundary consistency and depth-first-search histories across the graph.

8 Shapley Flow’s uniqueness proof

  • For uniqueness, the proof assumes a single source without loss of generality because independent noise nodes can be treated as one source without changing existing boundaries.The proof begins by assuming a solution exists and establishes that it must be unique.Source: §8 · p. 12, §8.1 · p. 12

8.1 At most one solution satisfies the axioms

  • Carrier games form a basis for boundary payoffs, so the axioms determine a unique attribution for each boundary and therefore for every edge.In a carrier game, only the last edge in the coalition changes the payoff; the other cut-set edges are dummy. The linear independence argument establishes that the coefficients, and thus the attribution, are unique.Source: §8.1 · pp. 12, 14

8.2 Shapley Flow satisfies the axioms

  • A directed acyclic graph lets each valid boundary be grown from the source in topological order, supporting the proof of boundary consistency.The proof also shows boundaries can be shrunk back to the source; depth-first search preserves consistent edge attributions as nodes are added.Source: §8.2 · pp. 14–15
  • Shapley Flow satisfies efficiency, linearity, and the dummy-player axiom by averaging output changes over depth-first-search histories.The boundary-edge credits sum to the target output change, f(x) − f(x′).Source: §8.2 · p. 15

9 Causal graphs

This section describes the causal graphs used for the income and nutrition datasets and details how the income graph is estimated. For income, the graph is an assumed structure and its variable relationships are learned from data.

9 Causal graphs

  • The section introduces the nutrition dataset and explains that causal relationships are estimated to produce the graphs shown for both datasets.The resulting causal structures are visualized in Figure 9.Source: §9 · p. 16

9.1 The Census Income dataset

  • The income dataset has 32,561 samples and 12 features, and the task predicts whether annual income exceeds 50k using an assumed causal graph.Birth attributes serve as source nodes; remaining features receive edges from their causal ancestors, and each feature points to the model.Source: §9.1 · p. 16

9.2 Causal Effect Estimation

  • XGBoost models estimate each node’s relationships using an 80/20 train-test split, with cross-entropy for categorical nodes and mean squared error for continuous nodes.Models use 100 trees of maximum depth 3 for up to 1,000 epochs; independent noise nodes represent residual effects.Source: §9.2 · p. 16
  • Noise is represented differently by variable type: continuous-variable noise is the prediction residual, while discrete-variable noise represents a random draw consistent with the predicted category probabilities.For a binary prediction of [0.3, 0.7] and an observed value of 0, the noise is sampled from 0 to 0.3 and attributions are averaged across samples.Source: §9.2 · p. 16

10 Additional Results

Additional experiments compare Shapley Flow with feature-attribution baselines on synthetic and real-data examples. They report that it captures both direct and indirect effects, including in global explanations and analyses using multiple background samples.

10.1 Additional Sanity Checks

  • In a four-variable chain where only X4 affects the output, independent SHAP misses upstream effects, ASV misses X4’s direct effect, and on-manifold SHAP fails to capture both fully.The chain’s nodes are identical copies of their predecessors; the accompanying causal graphs are described as oversimplifications used as proof of concept.Source: §10.1 · p. 16, Figure 10 · p. 17, Figure 9 · p. 17
  • Shapley Flow makes no errors in estimating both direct and indirect effects on the income linear-model results, while independent SHAP performs well only for direct effects and ASV only for indirect effects.The reported trend aligns with the nutrition and synthetic datasets; the tables report mean absolute error (std) for direct and indirect effects.Source: §10.1 · p. 18, Table 2 · p. 18, Table 3 · p. 18

10.2 Additional examples

  • In the nutrition example, Shapley Flow traces race’s indirect effects through blood pressure, serum magnesium, and blood protein, while independent SHAP assigns race low attribution.The passage reports that changing race from Black to White increases serum magnesium by 0.07 meg/L on average in the dataset.Source: §10.2 · p. 18
  • For the single nutrition example, on-manifold SHAP assigns age more credit and flips its sign relative to the other approaches through partial histories that disregard causal ordering.The passage explains that this attribution arises when serum magnesium updates before age, a history that differs from the causal ordering considered by the other baselines.Source: §10.2 · p. 18
  • Shapley Flow retains direct impacts comparable to independent SHAP while tracing effects to source nodes, and its edge-level detail can help reason about changing or broken causal links.The section also reports that ASV’s node-based view can miss direct impacts, while Shapley Flow exposes them alongside indirect effects.Source: §10.2 · pp. 18–19

10.3 A global understanding with Shapley Flow

  • On the income dataset, Shapley Flow shows that age and sex can have opposite direct and indirect impacts, even when the baselines’ relative importance is similar.The global explanation averages attributions for the predicted probability of the actual class across 1,000 randomly selected examples, using one shared background sample.Source: §10.3 · p. 19

10.4 Example with multiple background samples

  • With 100 background samples, Shapley Flow presents a holistic picture, while independent SHAP and ASV omit effects; the nutrition example highlights systolic blood pressure’s indirect impact on pulse pressure.ASV assigns no credit to systolic blood pressure because it is an intermediate node, although Shapley Flow indicates that intervening on it has a large impact on the outcome.Source: §10.4 · p. 19
  • On-manifold SHAP flips age’s attribution sign in the nutrition example because it considers partial histories regardless of causal structure, making the interpretation reflect observational rather than causal relationships.When systolic blood pressure updates before age, the expected age is below the foreground age, so the subsequent age update appears to increase risk.Source: §10.4 · p. 19

11 Considering all histories could lead to boundary inconsistency

The example shows why considering all message-transmission histories can violate boundary consistency. A shared edge receives different attributions under two cuts of the same causal graph.

11 Considering all histories could lead to boundary inconsistency

  • The example compares histories across two cuts that share edge a; the eight allowed message orders require c and b to follow d.The red and green cuts are presented as two boundaries of the same causal graph.Source: §11 · p. 19, Figure 19 · p. 27
  • Under the red game, linearity gives a positive attribution to a, negative attributions to b and c, and zero to d.The two cuts represent boundaries for the same causal graph.Source: §11 · p. 21, Figure 19 · p. 27
  • Edge a receives nonzero credit under the red boundary but zero under the green boundary, so considering all histories does not preserve boundary consistency.The passage states that the green boundary treats A and Y as a black box, making a a dummy edge there.Source: §11 · p. 21
Loading 2010.14592v3…