PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Causal Faithfulness of SHAP as a Candidate Attribution Method for SAE Features

Samuel Xie, Noura Al-Moubayed

arXiv:2610.17898v1cs.LGcs.AIcs.CL

TL;DR

SAE feature interpretation is largely correlational, and attribution faithfulness to causal effects has not been systematically evaluated in this setting. The paper compares attribution scores with direct interventions and finds that most measured associations do not survive correction, with controlled differences between directional and importance faithfulness and between insertion and deletion edits.

  • Problem

    SAE feature interpretation is predominantly correlational, and attribution methods have not been systematically evaluated against measured causal effects in this setting.

    Source: §1 · p. 1

  • Method

    The study compares SHAP, Integrated Gradients, Gradient × Input, and probe-weight baselines with feature ablation and activation steering measured through the model’s logit difference.

    Source: §1 · p. 2, §3.5 · p. 4

  • Results

    Most faithfulness associations do not survive multiple-comparison correction, and attribution methods with ρ = 0.42–0.80 agreement correspond poorly with measured causal effects.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Importance and directional faithfulness are separable, and agreement among attribution methods is not a substitute for causal validation.

    Source: §5 · p. 8, §6 · p. 9

    The evaluation uses a single model, SAE, layer, and task, so whether the patterns generalize remains open.

    Source: §5 · p. 9

Abstract

from arXiv · show

Sparse autoencoders decompose language model representations into interpretable features, but feature interpretation in this literature remains predominantly correlational. Attribution methods offer more direct estimates of feature influence, yet their faithfulness to actual causal effects has not been systematically evaluated in the SAE setting. We apply SHAP-based attribution to SAE latent features and evaluate it against direct causal intervention, using sentiment classification on SST-2 with GPT-2 Small and a 32,768-feature layer-7 SAE as a testbed. We compare SHAP to Integrated Gradients, Gradient$\times$Input, and probe-weight baselines across linear and MLP probe arms, scoring each against feature ablation and activation steering measured through the language model's own logit difference. Most faithfulness associations we measure do not survive multiple-comparison correction, and attribution methods that agree strongly with one another ($\rho = 0.42$--$0.80$) agree poorly with measured causal effects, indicating that inter-method agreement is not evidence of causal faithfulness. Two effects hold under control. First, local per-example attribution retains a directional advantage over global averaged attribution, while its apparent importance advantage is explained by feature activity rather than superior causal ranking. Second, under known-ground-truth minimal pairs, attribution recovers inserted sentiment signal well above chance (AUROC $0.792$) but fails to localize deletions (AUROC $0.520$), suggesting that these methods detect the presence of a feature's influence more reliably than its absence. We argue that importance faithfulness and directional faithfulness are separable properties requiring separate evaluation, and that attribution agreement should not substitute for causal validation.

1 Introduction

SAE feature interpretation is largely correlational, and attribution methods have not been systematically tested against causal interventions in this setting. This work evaluates SHAP and related methods against interventions, reporting controlled differences between local and global attribution and between insertion and deletion edits.

Motivation

  • Feature interpretation commonly relies on activating examples that reveal contexts but do not establish causal inputs or downstream effects; attribution methods may miss nonlinear interactions.The paper identifies no prior systematic evaluation of SHAP-based SAE attribution against direct interventions or of SAE feature rankings against measured causal effects.Source: §1 · pp. 1–2

Approach

  • The study tests SHAP, Integrated Gradients, and Gradient × Input against feature ablation and activation steering on SST-2 sentiment classification.It uses intervention effects as the causal reference rather than relying on correlational proxies.Source: §1 · p. 2

Contributions

  • Local per-example SHAP retains a directional advantage over global mean attribution, while known-ground-truth edits show above-chance recovery for insertions but not deletions.The authors frame these findings as distinct properties of attribution faithfulness.Source: §1 · p. 2

2 Related Work

Prior SAE interpretation is predominantly correlational, while attribution and probing methods provide tools for describing features or predictions. This work evaluates attribution faithfulness using direct causal interventions.

Sparse Autoencoders for Interpretability

  • SAEs decompose residual-stream activations into sparse, nearly monosemantic features, but feature interpretation remains largely based on examples or automated labeling rather than causal validation.Prior work scales SAE feature recovery to frontier models.Source: §2 · p. 2

Attribution Methods

  • Shapley attribution and Integrated Gradients identify prediction-driving inputs, while attribution patching approximates activation patching for transformer internals.The cited work had not systematically applied these methods to SAE latent features or evaluated their causal faithfulness.Source: §2 · p. 2

Faithfulness Evaluation

  • Faithfulness evaluations often use input perturbations rather than direct intervention, which can conflate correlation with causation.This study instead uses feature ablation and activation steering, treating intervention effects as ground truth.Source: §2 · p. 2

Probing

  • Linear probes test whether a concept is encoded at a layer; here, probes over SAE activations define the prediction functions to which attribution methods are applied.The SAE activations form the probe input feature space.Source: §2 · p. 2

3 Methodology

The methodology evaluates attribution methods on SAE activations using linear and MLP probes, then compares their scores with intervention effects and targeted validation analyses. It also examines local attribution, known-ground-truth edits, and whether SST-2 behavior is lexically simple.

3.1 Experimental Setup; 3.2 Probing Models

  • Experiments use frozen GPT-2 Small (124M) with a layer-7 SAE yielding 32,768 latent features; held-out data are reserved exclusively for attribution scoring and causal evaluation (ablation and steering).The logistic-regression probe uses the full 32,768-dimensional activation vector with L1 penalty (C = 1, liblinear); the one-hidden-layer MLP is fit on the filtered feature subset.Source: §3.1 · p. 3, §3.2 · p. 3

3.3 Attribution Methods

  • SHAP, Integrated Gradients, and Gradient × Input explain each probe’s target scalar, with raw probe weights and MLP saliency as baselines.The linear probe’s constant gradient makes IG and Gradient × Input analytically identical to each other and SHAP.Source: §3.3 · p. 3

3.4 Feature-Ranking Agreement

  • Method agreement is measured with pairwise Spearman correlations on features with non-zero probe scores; linear-arm IG, Gradient × Input, and SHAP are identical.MLP agreement uses absolute scores because raw probe saliency has no directional sign.Source: §3.4 · p. 3

3.5 Causal Evaluation

  • Causal faithfulness is tested by ablating features or steering their activations, then measuring changes in the model’s logit difference.The evaluation includes last-token-only steering to match attribution scope and reports importance and direction separately.Source: §3.5 · p. 4

3.6 Local Faithfulness

  • Local SHAP is compared with global mean attribution on matched per-example candidate sets, including analyses that restrict candidates to active features and control for activation strength.The protocol also tests whether candidate activity explains local-versus-global differences.Source: §3.6 · pp. 4–5

3.7 Multi-Feature Steering

  • A secondary analysis aggregates decoder directions from top-ranked features into steering vectors and tests effects across intervention strengths.Full details and results are provided in Appendix H.6.Source: §3.7 · p. 5

3.8 Known-Ground-Truth Validation

  • Known-ground-truth validation uses minimal sentiment-word deletions and appended sentiment clauses, assessing readout validity, feature recovery, and causal recovery while controlling SHAP-background and distractor-selection confounds.The canonical training background and randomly sampled distractors address the specified confounds.Source: §3.8 · p. 5

3.9 Lexical Simplicity Control

  • A lexical-simplicity control examines feature co-activation, activation tracking of sentiment words, and held-out accuracy against a bag-of-words logistic-regression baseline.The evaluation uses the full 10,103-example held-out split.Source: §3.9 · p. 5

4 Results

Across both probe arms, only a minority of causal-faithfulness tests survived correction, and importance faithfulness was detected more consistently than directional faithfulness. Local attribution retained a directional advantage under activity control, while minimal pairs showed stronger recovery for inserted than removed sentiment signals.

4.1 Probe Performance

  • Both probe architectures achieved substantial above-chance performance, but MLP validation was mildly optimistic because its input features were pre-filtered by the feature mask.Table 1 reports mean probe accuracy and AUROC for the linear and MLP probes.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Attribution Method Agreement

  • IG, GA, and SHAP formed a correlated cluster (ρ = 0.42–0.80), while probe weights were essentially uncorrelated with all three (ρ ≤0.04).Both linear and MLP probes showed substantial above-chance performance, although MLP validation was mildly optimistic because inputs were pre-filtered by the feature mask.Source: §4.2 · pp. 5–6, §4.1 · p. 5, Figure 1 · p. 6

4.3 Causal Faithfulness

  • 4/8 linear-arm tests and 2/7 MLP-arm tests survive FDR correction, with importance faithfulness more consistently detected than directional faithfulness.In the linear arm, probe weights show directional faithfulness (ρdir = 0.410, q < 0.0001); in the MLP arm, SHAP and IG show importance faithfulness (ρimp = 0.279 and 0.278, both q = 0.0088).Source: Table 2 · p. 6, §4.3 · p. 6

4.4 Local Faithfulness

  • In Block B, restricting candidates to active features removed the local importance advantage but preserved the local directional advantage in both arms.Activity explained only a modest fraction of between-example variation in the local–global gap (R2 = 0.019 linear; 0.081 MLP; 0.120/0.215 controlling for candidate-set size).Source: Table 3 · p. 7, §4.4 · p. 7

4.5 Known Ground-Truth Validation and Lexical Simplicity Control

  • AUROC was 0.792 for local SHAP and global SHAP/IG/GA on inserted edits, while probe weights reached 0.610.Causal recovery followed the same ordering (ρ = 0.536, 0.458, 0.167). Readout sign accuracy was 0.700 for insertions.Source: Table 4 · p. 8, §4.5 · p. 8
  • Deletion recovery was near chance: AUROC was 0.520 for local SHAP, 0.509 for global SHAP/IG/GA, and 0.503 for probe weights.Readout sign accuracy for removals was 0.770; causal recovery was weak across the board (ρ = 0.254, 0.163, −0.099 respectively).Source: Table 4 · p. 8, §4.5 · p. 8
  • The lexical baseline reached 0.629 accuracy versus 0.761 for the SAE probe, while top-feature activation had only weak association with sentiment-lexicon presence (mean |r| = 0.063).The authors conclude the probe signal is not reducible to simple sentiment-word detection.Source: §4.5 · p. 8

5 Discussion

Most measured attribution–causal faithfulness associations do not survive correction, and strong agreement among attribution methods does not reliably track intervention effects. Controlled analyses distinguish importance from directional faithfulness, while deletion localization remains weak and the study’s scope is limited.

5 Discussion

  • Most faithfulness associations do not survive multiple-comparison correction, and none of the k = 20 associations survive pooled correction across the multi-seed sweep.The k = 116 evaluation scores the entire filtered pool without resampling, avoiding seed sensitivity; a single k = 20 draw is insufficient for a faithfulness claim.Source: §5 · p. 8
  • IG, GA, and SHAP correlate in the MLP arm (ρ = 0.42–0.80) but correspond poorly with intervention effects, so inter-method agreement is not evidence of causal faithfulness.Top-feature activation is only weakly associated with sentiment-lexicon presence (mean |r| = 0.063), while the SAE probe exceeds a bag-of-words baseline by 13.2 accuracy points.Source: §5 · p. 8
  • Under activity controls, local attribution retains a directional advantage, while its apparent importance advantage disappears and reverses in the MLP arm.There, global attribution outperforms local (+0.371 vs. +0.077, p = 9.6 × 10−9); the preferred attribution depends on the faithfulness criterion.Source: §5 · p. 8
  • Attribution recovers inserted sentiment signal (AUROC 0.792) but poorly localizes deletions (AUROC 0.520 and 0.509), despite a readout registering deletion correctly in 77% of pairs.The authors suggest deletion redistributes activation across many features, each moving too little to detect; attribution may identify influence more reliably than its absence.Source: §5 · pp. 8–9
  • The study evaluates a single model, SAE, layer, and task, so whether these patterns generalize remains open.The decomposition-and-control methodology is not specific to SHAP and could extend to other settings.Source: §5 · p. 9

5.1 Limitations

  • Attributions use one token position, steering magnitude is calibrated against a different readout than the evaluation readout, and feature-count correlations are underpowered for modest effects.Although several linear-arm associations appear in the powered seed-0 analysis, none survive pooled multi-seed FDR correction, indicating candidate-sampling sensitivity.Source: §5.1 · p. 9

5.2 Future Directions

  • Whether local attribution’s directional advantage generalizes beyond sentiment classification and this specific SAE remains an open question.The authors note that the decomposition-and-control methodology could extend to other settings.Source: §5.2 · p. 9

6 Conclusion

The paper evaluates SAE-feature attribution against direct causal interventions and finds that most measured faithfulness associations do not survive correction. It concludes that attribution agreement is not a substitute for causal validation and that importance and directional faithfulness require separate evaluation.

6 Conclusion

  • The study evaluates SHAP against direct intervention, comparing it with Integrated Gradients, Gradient×Input, and probe-weight baselines across linear and MLP probe arms.Most measured faithfulness associations do not survive correction, and attribution-method agreement does not track agreement with causal effects.Source: §6 · p. 9
  • Local attribution retains a directional advantage under control, while its apparent importance advantage reflects feature activity rather than superior causal ranking.Attribution recovers inserted sentiment signal well above chance but fails to localize deletions.Source: §6 · p. 9
  • Attribution agreement should not substitute for causal validation, and importance and directional faithfulness warrant separate evaluation.These conclusions are drawn from the study’s SAE-feature sentiment-classification evaluation.Source: §6 · p. 9

7 Use of AI Tools

The authors disclose AI assistance with experimental-code generation and paper prose, stating that they reviewed, tested, and verified the assisted work. The appendix describes its role as documenting experimental setup choices relevant to the validity of reported numbers.

7 Use of AI Tools

  • Claude Sonnet 5 and Opus 5 assisted with portions of experimental-code generation and with drafting and editing the paper’s prose.The authors reviewed, tested, and verified the AI-assisted code and text and take responsibility for the reported experiments, results, and conclusions.Source: §7 · p. 9

A Experimental Setup Details

  • The appendix provides implementation details on the dataset and splits, model and SAE, probe architectures, and feature selection.It focuses on choices bearing directly on the validity of the reported numbers.Source: A Experimental Setup Details · p. 11

A.1 Task and Dataset

The study uses a fixed split of SST-2 for sentiment classification and reserves held-out data for attribution scoring and causal evaluation. It applies a GPT-2 Small layer-7 SAE, compares linear and MLP probe arms, and uses a validation-derived feature mask for downstream components.

A.1 Task and Dataset

  • The task is binary sentiment classification on SST-2, split once with a fixed seed into three disjoint subsets.Attribution and causal evaluation use the held-out split, on which the researchers fit or tune nothing.Source: §A.1 · p. 11
  • Canonical sampling uses 2,000 held-out evaluation rows and 100 training background rows, with experiments using prefixes of those samples.The 100-row local sample is the first 100 rows of the 2,000-row global sample, enabling joins by row position.Source: §A.1 · p. 11

A.2 Model and Token Position

  • The testbed uses GPT-2 Small with a frozen 32,768-feature SAE at layer 7, using the last real token as the sentence representation.The same token-position definition is used for extraction, probing, and causal readouts; decoder rows are approximately unit norm, so no decoder-norm correction is applied.Source: §A.2 · p. 11

A.3 Probing Models

  • The study duplicates attribution and causal-evaluation scripts across linear and MLP probe arms; the MLP’s input-dependent gradients allow IG, GA, and SHAP to differ substantively.The linear probe’s constant gradient makes IG and GA collapse to closed-form identities with SHAP, serving as a sanity check rather than a substantive comparison.Source: §A.3 · pp. 11–12
  • The MLP’s input mask uses a validation-derived, label-free activation-frequency filter, making its reported validation accuracy mildly optimistic.The authors describe a small but real dependency between feature selection and reported accuracy.Source: §A.3 · p. 12

A.4 Filtered Feature Set

  • The downstream feature mask combines non-zero linear-probe coefficients with activation frequency of at least 5% on validation data.The reported mask contains 116 features; it gates the MLP inputs and steering candidate pools, while linear-arm attribution is computed across the full feature space.Source: §A.4 · p. 12

B Attribution Method Definitions

The section defines matched attribution targets and specifies SHAP and gradient-based methods for linear and MLP probes. For a linear probe, matched-target IG, GA, and interventional SHAP reduce to the same score; the substantive method comparison therefore lies in the nonlinear MLP arm.

B Attribution Method Definitions

  • The methods compared per arm are SHAP, Integrated Gradients, Gradient×(Input−Baseline), and raw probe weight or saliency as a zero-cost baseline.The linear probe's constant gradient makes the first three methods analytically coincide under a matched target.Source: B Attribution Method Definitions · p. 13, §B.6 · p. 14

B.1 Target-Function Matching

  • Every method within an arm explains the same scalar function; attribution magnitudes are compared only within an arm because the arms use different target scales.Table 6 identifies the target function attributed by each arm.Source: §B.1 · p. 13, Table 6 · p. 13

B.2 SHAP — Linear Arm

  • Linear-arm SHAP uses exact shap.LinearExplainer and reports each feature's mean signed SHAP value across 2,000 held-out examples.The explainer uses an Independent masker and the full 100-row training background.Source: §B.2 · p. 13

B.3 SHAP — MLP Arm

  • MLP-arm SHAP uses DeepExplainer on a weight-matched clone of the fitted network, targeting its pre-sigmoid logit and filtered input features.The clone preserves the supported Sequential operations required by DeepSHAP's operation-level attribution.Source: §B.3 · p. 13

B.4 Integrated Gradients

  • Integrated Gradients follows the baseline-to-input path; its constant-gradient linear form is computed directly, while the MLP path uses a Riemann sum.The baseline is the mean of the 100-row training background, not a zero vector.Source: §B.4 · p. 13

B.5 Gradient × (Input −Baseline)

  • Gradient × (Input − Baseline) uses a single endpoint gradient, matching IG for a linear probe but differing from path-integrated IG for the nonlinear MLP.The shared baseline preserves a meaningful input difference for sparse SAE features.Source: §B.4 · p. 13, §B.5 · p. 14

B.6 The Linear-Arm Identity

  • With a matched target, IG, GA, and interventional SHAP are analytically identical for a linear probe, so their perfect rank agreement is an implementation check, not an empirical comparison.The identity breaks under the MLP's ReLU nonlinearity, where the methods can diverge.Source: §B.1 · p. 13, §B.6 · p. 14

C Causal Evaluation Protocol

The protocol tests attribution rankings against interventions on SAE features, using GPT-2's logit difference rather than the probe's prediction as the causal readout. It also specifies controls for selection, activity, and statistical testing, alongside local, group-steering, and known-ground-truth evaluations.

C Causal Evaluation Protocol

  • Ablation zeros a chosen feature and steering adds a signed activation at every token; both are compared with an encode-decode reconstruction baseline.The causal readout is the change in GPT-2's last-token logit difference, “wonderful” minus “awful,” rather than the probe's prediction.Source: §C.1 · p. 15, §C.2 · p. 15, C Causal Evaluation Protocol · p. 15

C.3 Steering-Strength (α) Calibration

  • Steering strength is calibrated on a layer-11 probe probability shift, not the later GPT-2 logit-difference readout, making α a rough unsaturated magnitude.The primary per-feature mode scales α by each feature's mean positive training activation relative to the median scale.Source: §C.3 · p. 15

C.4 Candidate Selection Strategies

  • Random feature selection is the headline strategy; scoring all 116 filtered features provides a powered global test, while top-|SHAP| selection can mechanically depress SHAP's correlation.At k = 20, the confidence interval for Spearman ρ can span most of [−1, 1].Source: §C.4 · p. 16

C.5 Faithfulness Metric and Sign Convention

  • The protocol distinguishes importance faithfulness, which ranks absolute attribution against absolute causal effect, from directional faithfulness, which compares attribution sign with oriented intervention effect.It corrects ablation's sign so positive correlation denotes faithfulness for either intervention mode.Source: §C.5 · p. 16

D Local (Per-Example) Faithfulness Protocol

  • For local comparisons, each sentence's candidate set combines local and global top-k features, and per-example effects are retained; among active features, Block B tests ranking beyond activity.Globally top-ranked features are often inactive in an individual sentence, creating zero-effect ties that can make local attribution appear better simply because it tracks activity.Source: §D.1 · p. 17, §D.2 · p. 17, D Local (Per-Example) Faithfulness… · p. 17

E Multi-Feature (Group) Steering Protocol

  • The secondary group-steering experiment builds a signed decoder-weighted direction from each method's top-k features and measures mean logit-difference effect against perplexity cost.It adds the vector to a clean residual stream without SAE encode-decode, so curve shapes—not effect magnitudes—are comparable with per-feature experiments.Source: E Multi-Feature (Group) Steering… · p. 18

F Known-Ground-Truth Validation Protocol (Minimal Pairs)

  • Known-ground-truth pairs delete a single lexicon word or append a sentiment clause, then assess readout validity, recovery of changed features, and causal recovery by ablation.The one-word deletion constraint avoids leaving a second sentiment word to carry the pair's signal.Source: §F.2 · p. 19, §F.1 · p. 19, F Known-Ground-Truth Validation… · p. 19

H.4 Multi-Seed Replication

  • After pooling ten seeds, no faithfulness tests survive FDR correction: 0/80 in the linear arm and 0/70 in the MLP arm.The sweep uses steering, random selection, and scaled α at k = 20.Source: §H.4 · pp. 21–22

H.5 Underpowered k = 20 causal validation results

The reported controls show a residual local-SHAP association after continuous activity adjustment, weak lexical tracking by top SAE features, and different corrected-test counts across probe arms. The full group-steering grid is described, but its caption cautions against comparing effect magnitudes with per-feature interventions.

H.5 Underpowered k = 20 causal validation results

  • +0.184 for linear and +0.150 for MLP are the partial Spearman correlations retained by local SHAP after controlling continuously for activation.This control keeps the full candidate set rather than splitting active from inactive features.Source: §H.5 · p. 22

H.6 Group-Steering Full Sweep

  • The group-steering sweep spans k values 5–65 and seven α multipliers, but its clean-forward-pass baseline means only curve shapes—not magnitudes—are compared with per-feature interventions.The figure presents a k × α grid with one panel per method per arm.Source: §H.6 · p. 22, Figure 3 · p. 23

H.7 Lexical Simplicity Control

  • Mean |r| = 0.063 is below the prespecified 0.25 strong lexical-tracking threshold, while the 0.132 accuracy gap exceeds the 0.03 near-tie threshold.The authors interpret this as evidence that top SAE features are not simply lexical sentiment-word detectors.Source: §H.7 · p. 23, Table 13 · p. 23

H.8 All-token global faithfulness

  • 4/8 linear-arm tests and 0/7 MLP-arm tests survive FDR 0.05 in the all-token global faithfulness analysis.The supplied table caption identifies this as a powered replication using all filtered features and correction within arm.Source: §H.8 · p. 23, Table 14 · p. 23

I Known Limitations

The reported faithfulness comparisons have several scope and measurement caveats. Some probe statistics are unsuitable for directional interpretation, and some apparent agreements or rankings should not be read as empirical evidence.

I Known Limitations

  • Attributions cover one sequence position, whereas interventions edit every token, so a faithfulness gap may reflect scope mismatch rather than attribution error.Source: I Known Limitations · p. 24
  • Faithfulness ρ estimates are small-n and, by default, lack multiple-comparison control.Source: I Known Limitations · p. 24
  • Calibration uses a layer-11 residual probe’s P(positive), but evaluation uses GPT-2 logit difference.Source: I Known Limitations · p. 24
  • MLP probe_saliency uses magnitude-only ∥W1[i, :]∥2, making it valid for importance rankings but invalid for directional or causal-sign comparisons.It is also excluded from group-steering vector construction.Source: I Known Limitations · p. 24
  • Top-selection range-restricts only SHAP’s scored variance, biasing its faithfulness ρ downward relative to the other three methods.The authors do not present a top number without this caveat.Source: I Known Limitations · p. 24
  • The MLP probe’s validation accuracy is mildly optimistic because its inputs use a validation-derived, label-free frequency filter.Source: I Known Limitations · p. 24
  • The linear arm’s rank-agreement ρ = 1.000 is a closed-form identity, not empirical evidence that SHAP and gradient methods generally agree.Source: I Known Limitations · p. 24

Related papers

Loading 2610.17898v1…