PaperlayerView exact revision on arXiv

Source-linked AI summary

Shapley explainability on the data manifold

Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, Ilya Feige

arXiv:2006.01272v4cs.LGcs.AIstat.ML

TL;DR

A general, performant solution for on-manifold Shapley values has been lacking. The paper introduces generative-imputation and direct value-function learning methods, and reports that off-manifold explanations can be incorrect and hide sensitive-feature dependence.

  • Problem

    A general, performant way to compute Shapley values on the data manifold has been lacking, despite the use of local, model-agnostic explanations.

    Source: §1 · p. 1

  • Method

    The paper develops a generative method that learns conditional distributions and a supervised method that directly learns the Shapley value-function.

    Source: §1 · p. 2

  • Results

    Off-manifold explanations are often incorrect and can hide implicit model dependence on sensitive features.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors conclude that on-manifold Shapley values offer a viable approach to AI explainability in real-world contexts.

    Source: §5 · p. 9

    The formal comparison assumes that input features arise from latent variables through feature-specific functions.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Explainability in AI is crucial for model development, compliance with regulation, and providing operational nuance to predictions. The Shapley framework for explainability attributes a model's predictions to its input features in a mathematically principled and model-agnostic way. However, general implementations of Shapley explainability make an untenable assumption: that the model's features are uncorrelated. In this work, we demonstrate unambiguous drawbacks of this assumption and develop two solutions to Shapley explainability that respect the data manifold. One solution, based on generative modelling, provides flexible access to data imputations; the other directly learns the Shapley value-function, providing performance and stability at the cost of flexibility. While "off-manifold" Shapley values can (i) give rise to incorrect explanations, (ii) hide implicit model dependence on sensitive attributes, and (iii) lead to unintelligible explanations in higher-dimensional data, on-manifold explainability overcomes these problems.

1 INTRODUCTION

The introduction motivates local, post-hoc, model-agnostic explanations and identifies unrealistic feature splicing as a problem for Shapley methods when features are dependent. It presents two on-manifold methods and highlights shortcomings of off-manifold explanations.

1 INTRODUCTION

  • The paper focuses on local, post-hoc, model-agnostic explainability to explain individual predictions without limiting model expressiveness.The authors also value comparisons across model types.Source: §1 · p. 1
  • Shapley values provide axiomatically grounded attributions that capture feature interactions and sum to the model prediction (Shapley, 1953).Source: §1 · p. 1
  • Standard Shapley implementations splice data points to marginalize over features, producing unrealistic inputs when features are correlated.Such splicing is justified only if all features are independent.Source: §1 · p. 1
  • Prior empirical and kernel-based approaches to on-manifold Shapley estimation do not scale to complex data, leaving a general, performant solution lacking.The introduction identifies Aas et al. (2019) and Sundararajan & Najmi (2019) as examples.Source: §1 · p. 1
  • Off-manifold explanations can be incorrect and hide implicit dependence on sensitive features; the paper develops generative and direct value-function learning methods for on-manifold Shapley values.The authors report demonstrating the methods’ effectiveness on higher-dimensional data.Source: §1 · pp. 1–2

2 BACKGROUND ON SHAPLEY EXPLAINABILITY

This section defines Shapley attribution by treating features as players and model output as coalition value. It contrasts unconditional splicing with conditional, on-manifold evaluation and explains local and global interpretations.

2 BACKGROUND ON SHAPLEY EXPLAINABILITY

  • A Shapley value distributes a team’s total value among its players, assigning each player its average marginal contribution across all team orderings.Source: §2 · p. 2
  • For model explainability, features are players, the predicted class probability is their value, and a coalition value function is needed for partial inputs.Source: §2 · p. 2
  • The standard implementation samples omitted features unconditionally and evaluates spliced inputs, which generally lie off the data manifold.Source: §2 · p. 2
  • Conditioning omitted features on the observed coalition defines an on-manifold value function, but its conditional distribution is difficult to access with high-dimensional or many-valued features.Source: §2 · p. 2
  • Local Shapley values explain an individual prediction, while their aggregation yields global values interpreted as feature contributions to model accuracy.Global values obey a sum rule; the class-balance offset is not attributable to an individual feature.Source: §2 · p. 2
  • The framework also applies to regression by interpreting the model output as its predicted value rather than a predicted probability.Source: §2 · p. 2

3 EVIDENCE IN FAVOUR OF ON-MANIFOLD EXPLAINABILITY

On-manifold Shapley values evaluate models on inputs consistent with the data distribution, while off-manifold splicing can produce misleading attributions. The section shows how this distinction affects explanation accuracy, sensitive-feature attribution, and the interpretation of feature importance.

3.1 ON- VERSUS OFF-MANIFOLD DIFFERENCES MADE PRECISE

  • With small feature noise, data generated from d latent variables lies on a d-dimensional manifold in n-dimensional feature space, and on-manifold evaluations preserve its constraints.Spliced off-manifold inputs generically break those constraints.Source: §3.1 · p. 3

ALGEBRAIC MODEL DEPENDENCE CAN BE MISLEADING

  • On-manifold explanations agree for models that behave identically on the data distribution, whereas off-manifold splicing can give those models different explanations.Adding manifold constraints to model input slots creates an n(n −d)-dimensional equivalence class of models indistinguishable on the data.Source: ALGEBRAIC MODEL DEPENDENCE CAN BE… · p. 3

HIDDEN MODEL DEPENDENCE ON SENSITIVE ATTRIBUTES

  • Off-manifold attribution can assign zero importance to a sensitive feature even when an equivalent model retains the same on-data behavior and bias.In the Census Income example, the original and suppressed models make identical predictions on 98.5% of data; off-manifold sex attribution is zero for the suppressed model, while on-manifold explanations are similar.Source: HIDDEN MODEL DEPENDENCE ON SENSITIVE… · pp. 3–4, Figure 2 · p. 4

ON-MANIFOLD SHAPLEY VALUES IN THE OPTIMAL-MODEL LIMIT

  • In the optimal-model limit, on-manifold Shapley values attribute value according to each feature’s predictivity of the label.The authors report that this limit is approached with abundant data and high model capacity, and that the resulting values agree with average accuracy gains from adding features to retrained coalition models.Source: ON-MANIFOLD SHAPLEY VALUES IN THE… · p. 4
  • Tree SHAP roughly tracks the off-manifold explanation because it privileges early-splitting, high-predictivity features in an otherwise off-manifold calculation.In the Drug Consumption example, the paper contrasts fixed-model explanations with retraining on each feature coalition.Source: ON-MANIFOLD SHAPLEY VALUES IN THE… · pp. 4–5

OUTLIER DETECTION EXPLAINED INCORRECTLY OFF-MANIFOLD

  • One-quarter of off-manifold explanations are erroneous at minimal noise, rising to two-thirds at σ = 0.15, while on-manifold errors are dramatically lower.For the synthetic outlier task, the correct explanation identifies the first five features as breaking correlations shared by 99% of the data; off-manifold ranking can misorder those features.Source: Figure 3 · p. 5, OUTLIER DETECTION EXPLAINED… · p. 6

UNCONTROLLED MODEL BEHAVIOUR OFF-MANIFOLD

  • Off-manifold Shapley evaluates models outside their data-supported domain, where behavior may be uncontrolled, particularly for complex nonlinear models.The paper characterizes this as a garbage-in-garbage-out problem rather than a controlled measurement of algebraic model dependence.Source: UNCONTROLLED MODEL BEHAVIOUR… · p. 5

BREAKDOWN IN GLOBAL SHAPLEY VALUES OFF-MANIFOLD

  • Off-manifold global Shapley values can be negative on synthetic and real data, misleadingly suggesting a feature harms model performance.The synthetic example attributes a negative value to x1 because splicing overweights combinations rare in the data; the real-data example also has a large negative off-manifold value.Source: Figure 4 · p. 6, BREAKDOWN IN GLOBAL SHAPLEY VALUES… · pp. 6–7

4 SCALABLE APPROACHES TO ON-MANIFOLD SHAPLEY VALUES

The section develops unsupervised conditional-distribution learning and supervised direct value-function learning for scalable on-manifold Shapley values. Experiments find that the supervised approach is more accurate, stable, and efficient, while on-manifold MNIST explanations are more concentrated and interpretable than off-manifold explanations.

4.1 THEORETICAL DEVELOPMENT OF ON-MANIFOLD METHODS

  • The section introduces two approaches to learn the on-manifold value function: unsupervised conditional-distribution learning and supervised direct value-function learning.The unsupervised approach learns conditional distributions; the supervised approach learns the value function directly.Source: §4 · p. 7, §4.1 · p. 7

UNSUPERVISED APPROACH

  • The unsupervised approach uses variational inference to learn conditional distributions, with a VAE and masked encoder composing a model of p(x′|xS).A regularisation term penalises differences between the full-input encoder distribution and a smooth prior.Source: UNSUPERVISED APPROACH · p. 7

METRIC FOR THE LEARNT VALUE FUNCTION

  • The unsupervised model is evaluated by estimating the mean-squared error between its value-function estimate and the on-manifold value function.The metric is estimated using the empirical data distribution and the learned conditional model.Source: METRIC FOR THE LEARNT VALUE FUNCTION · pp. 7–8

PERFORMANCE AND STABILITY

  • Across the studied datasets, the supervised method achieves higher accuracy and stability than the unsupervised method, and both improve significantly over off-manifold splicing.The supervised method also requires roughly 10 times fewer model evaluations to match the unsupervised method’s standard error.Source: Table 1 · p. 8, PERFORMANCE AND STABILITY · p. 9

SUPERVISED APPROACH

4.2 ADDITIONAL EXPERIMENTS

  • Additional experiments examine performance, stability, and effectiveness on higher-dimensional data, including prior applications to Census Income and Abalone.Those applications found hidden sensitive-feature dependence on Census Income and positive, interpretable global Shapley values on-manifold for Abalone.Source: §4.2 · p. 8

PERFORMANCE AND STABILITY

  • On Drug Consumption data, both learned methods produce broadly similar explanations compared with the empirical ground-truth computation.Figure 5 compares the methods with the empirical computation.Source: PERFORMANCE AND STABILITY · p. 8, Figure 5 · p. 8

EXAMPLE ON MNIST

  • On MNIST, on-manifold explanations are more concentrated and interpretable than off-manifold explanations despite having the same pixel sum and explaining the same prediction.Off-manifold explanations rely primarily on large pixel coalitions, whereas fewer pixels are required on-manifold.Source: EXAMPLE ON MNIST · p. 9, Figure 6 · p. 9

5 CONCLUSION

The conclusion presents experimental evidence of shortcomings in off-manifold Shapley values and introduces two approaches for computing on-manifold values on general data. It argues that these results support on-manifold Shapley values as a viable approach to explainability in real-world contexts.

5 CONCLUSION

  • The paper reports experimental evidence for novel shortcomings of off-manifold explainability and distinctions between on- and off-manifold approaches.It characterises the evidence as supporting the case against off-manifold explainability.Source: §5 · p. 9
  • The two proposed approaches either learn feature imputations on the data manifold or learn the Shapley value function directly.Both approaches are intended for general data.Source: §5 · p. 9
  • The conclusion presents on-manifold Shapley values as a viable approach to explainability in real-world contexts.This conclusion follows the paper’s study of off-manifold shortcomings and its introduction of the two methods.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

The implementation uses neural-network components for both the conditional-distribution and direct value-function approaches, with validation MSE used for hyperparameter tuning.

A IMPLEMENTATION DETAILS

  • The unsupervised approach uses an encoder, decoder, and masked encoder to model conditional distributions.The encoder is a diagonal normal, the decoder a product distribution, and the masked encoder a Gaussian mixture.Source: A IMPLEMENTATION DETAILS · p. 12
  • The masked encoder accepts variable-size coalitions by replacing out-of-coalition features with −1, a value absent from the data.Source: A IMPLEMENTATION DETAILS · p. 12
  • Both approaches tune their network and optimisation hyperparameters to minimise validation-set MSE.The unsupervised approach also tunes β and the number of mixture components.Source: A IMPLEMENTATION DETAILS · p. 12

B DETAILS OF EXPERIMENTS

This section provides numerical details for the experiments reported in the paper.

B DETAILS OF EXPERIMENTS

B.1 DRUG CONSUMPTION EXPERIMENT

The Drug Consumption experiment predicts LSD use from 10 binary drug-use features and compares several Shapley estimation approaches. Its empirical conditional-distribution estimate is feasible here because the dataset has few all-binary features.

B.1 DRUG CONSUMPTION EXPERIMENT

  • 82.2% test-set accuracy was achieved by a default-parameter random forest predicting LSD use from 10 binary drug-use features, with a 57:43 class balance.The features included Mushrooms and Ecstasy.Source: §B.1 · p. 12
  • Global off-manifold values used 10^6 Monte Carlo samples, while each test point used one permutation and one sampled reference point.Bars show Monte Carlo means and error bars show standard errors of the mean.Source: §B.1 · p. 12
  • Empirical conditional-distribution estimation was possible only because this dataset has a small number of all-binary features.For sampled coalitions, a test-set point matching the observed coalition was used to estimate p(x′|xS).Source: §B.1 · p. 12
  • The on-manifold comparisons included empirical, supervised, unsupervised, tree SHAP, and model-retraining values.The retraining comparison fit 210 coalition-specific random forests and used stochastic-label accuracy; supervised values used an MSE-trained network, while the unsupervised approach used a variational autoencoder.Source: §B.1 · pp. 12–13
  • The supervised and unsupervised Shapley estimates were selected by held-out validation MSE, and their reported values used 10^4 and 10^6 Monte Carlo samples, respectively.The unsupervised method scanned latent dimensions, latent modes, and regularisation β; the supervised method selected its lowest-validation-MSE model.Source: §B.1 · p. 13

B.2 CENSUS INCOME EXPERIMENT

The Census Income experiment predicts whether income exceeds $50k using 13 features from 49,000 individuals. It also compares the original model with a fine-tuned model that penalizes the output’s finite difference with respect to sex.

B.2 CENSUS INCOME EXPERIMENT

  • 85% test-set accuracy was achieved on the Census Income task, predicting income above $50k from 13 features for 49,000 individuals.The dataset came from the 1994 US Census and had a 76:24 class balance.Source: §B.2 · p. 13
  • The original model was a fully connected network with hidden layer size 50, trained with default sklearn parameters and early stopping.Its Shapley values were computed using the procedure described in Appendix B.1.Source: §B.2 · p. 13
  • The sex-suppressed fine-tune agreed with the baseline on over 98.5% of data and had the same test-set accuracy.Its loss penalized the finite difference in output with respect to sex; α controls the trade-off between accuracy and reducing sex’s effect.Source: §B.2 · pp. 13–14

B.3 ABALONE EXPERIMENT

The Abalone experiment classifies age relative to the median from eight physical-measurement features. It reports the neural-network task setup and how the Shapley values were computed.

B.3 ABALONE EXPERIMENT

  • The task used eight physical-measurement features to classify abalone as younger or older than the median age.The data came from the UCI repository.Source: §B.3 · p. 14
  • 78% test-set accuracy was obtained with a neural network using hidden layer size 100, default sklearn parameters, and early stopping.Source: §B.3 · p. 14
  • Shapley values were computed using the Appendix B.1 procedure, with the supervised method trained for 5,000 epochs.Optimised hyperparameters are listed in Table 2.Source: §B.3 · p. 14

B.4 MNIST EXPERIMENT

The MNIST experiment evaluates Shapley explanations for a neural network trained on binary digits. The on-manifold pixel explanations use 16,000 Monte Carlo samples per pixel, with average uncertainty around 0.002 relative to each image’s maximum Shapley value.

B.4 MNIST EXPERIMENT

  • 98% test-set accuracy was achieved by a fully connected network on binary MNIST.The network used hidden layer size 512, default sklearn parameters, and early stopping.Source: §B.4 · p. 14
  • On-manifold explanations for sampled test-set digits used the supervised method and 16,000 Monte Carlo samples per pixel.The supervised method was trained for 2,000 epochs; the unsupervised method was also run to complete Table 1 but is not shown in the figure.Source: §B.4 · p. 14
  • Average uncertainty was roughly 0.002 as a fraction of each image’s maximum Shapley value.This uncertainty is not shown in Fig. 6(a).Source: §B.4 · p. 14
Loading 2006.01272v4…