PaperlayerView exact revision on arXiv

Source-linked AI summary

Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition

Devesh Tiwari, Camille Davis, Shivank Sinha, Talia Weaver, Aditya Shah, Maheep Chaudhary

arXiv:2609.18080v1cs.AI

TL;DR

Probe accuracy can show that information is decodable without establishing that it drives model behavior. The paper compares probe-aligned and gradient-sensitive SAE features through interventions. In the studied truth-probe setting, shared features produce larger coherent behavioral changes than probe-only features, which primarily affect the probe readout.

  • Problem

    Linear probes can decode behavioral information from activations, but decodability alone does not establish that the decoded representations causally contribute to model decisions.

    Source: §1 · p. 1

  • Method

    The framework decomposes activations into SAE features, ranks them by probe alignment and gradient sensitivity, and intervenes on feature groups while tracking output coherence.

    Source: §3.3 · p. 3

  • Results

    Shared probe-model features produce larger coherent behavior changes, while probe-only features primarily affect the probe readout.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Geometric projection of probe weights alone is insufficient to identify causally used features, whereas activation-aware selection recovers substantially more behaviorally causal features in this setting.

    Source: §6 · p. 9

    Most causal analysis concerns one model, layer, SAE, and behavioral setting, so generalization across model families, architectures, and domains remains unclear.

    Source: §7 · p. 10

Abstract

from arXiv · show

Linear probes can decode safety-relevant concepts such as truthfulness from language-model activations, but probe accuracy may show only decodability, not that the features the probe weights causally drive model behavior. We demonstrate that this gap cannot be closed from the geometry of probe weights alone: the features geometrically aligned with probe direction need not be the ones the model uses, so causal relevance requires intervention. We introduce a feature-level diagnostic that decomposes a deployed True/False probe into sparse-autoencoder (SAE) features, ranks those features by both probe alignment and by gradient sensitivity of the model's behavior, and ablates the resulting shared, probe-only, and random feature sets under a coherence gate. On the truth probe of Buerger et al. (2024) (TTPD), applied in the instructed truth/deception setting of Long et al. (2025) for Gemma2-9B-Instruct, the two rankings overlap only weakly (about 12%, Spearman rho = 0.10), and ablation dissociates them sharply: features the probe shares with the model flip the output far more (up to 27%) than equally sized probe-only (6%) or random (1%) features at full coherence, while probe-only features instead perturb the probe's own readout. The dissociation holds across five seeds and a held-out split, and an activation-aware selection of features flips behavior nearly three times as often as the probe's geometric top features (17.6% vs. 6.1%). In this setting, therefore, the geometric projection of a probe's weight vector alone does not identify the features the model causally uses; however, combining probe information with feature activation statistics recovers substantially more behaviorally causal features, and coherence-gated SAE intervention is needed to separate them from probe readouts.

1 Introduction

Probe accuracy establishes that information is decodable, not that it drives model behavior. We compare probe-aligned and model-sensitive SAE features through intervention, finding that their overlap predicts which features change behavior versus only the probe readout.

1 Introduction

  • Linear-probe accuracy alone does not establish that decoded representations play a causal role in model decisions.Distinguishing merely predictive features from causally relevant ones remains an open challenge.Source: §1 · p. 1
  • The study decomposes activations into SAE features, ranks them by probe alignment and gradient sensitivity, then intervenes while tracking output coherence.The framework compares probe-aligned features with features whose perturbation most affects model output.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Probe-aligned and model-sensitive features overlap weakly, and ablating shared features flips outputs more than equally sized probe-only or random sets while preserving coherence.Probe-only features instead perturb the probe readout without changing behavior.Source: §1 · p. 2
  • The framework compares probe-aligned and gradient-sensitive features through direct behavioral interventions, quantifying divergence and identifying shared features that influence both readouts.Shared probe-model features provide stronger evidence of behavioral relevance than probe-only features.Source: §1 · p. 2

2 Related Work

Prior work distinguishes information that probes decode from information used by a model, and evaluates representations through interventions. This work asks whether geometrically probe-aligned SAE features carry the behavioral effect of a probe direction.

Probing, truth representations, and behavioral use

  • Probe accuracy need not track task importance, motivating interventions that are complete for the targeted property and selective about unrelated information.Amnesic probing made the decodability-versus-use distinction explicit by removing probe-identified information and measuring downstream behavior.Source: §2 · p. 2
  • Prior truth-probe work identifies approximately linear truth representations and studies their behavioral relevance through activation interventions.Bürger et al. introduce TTPD, while Long et al. examine truth-related representations under instructed deception.Source: §2 · p. 2
  • The study asks whether SAE features geometrically aligned with a behaviorally causal probe direction are the features carrying its behavioral effect.Related work also reports probes that exploit task-format confounds (Sahoo et al., 2026).Source: §2 · p. 3

Sparse autoencoders and causal feature evaluation

  • SAE reconstruction quality and interpretability alone do not guarantee causal or task-level utility, motivating task-specific interventions and checks that reconstruction preserves the probe signal.Prior evaluations report weaker behavioral control, difficulty disentangling factual attributes, and unreliable transfer from proxy metrics.Source: §2 · p. 3
  • Prior work uses SAE ablation, patching, and steering to identify relevant features, while contrastively selected reasoning features can reflect lexical correlates rather than reasoning computations.Source: §2 · p. 3

3 Methodology

The methodology compares probe-aligned SAE features with gradient-sensitive features, then validates their effects through interventions while monitoring coherence. It also uses an activation-aware mask and partitions features into shared, probe-only, and random groups for matched comparisons.

3.1 Overview

  • The workflow covers probe replication, SAE attribution, gradient ranking, interventions, and overlap decomposition, with coherence monitored throughout and an auxiliary deception-probe diagnostic.The target probe is replicated using the original authors’ procedure and matches their reported performance.Source: §3.1 · p. 3, §3.2 · p. 3

3.3 Stage 2: SAE Feature Attribution

  • Probe attribution projects the probe weight vector into the SAE decoder basis, using each feature’s alignment with the probe direction to rank what the probe reads.The probe output is approximated by the sum of feature activations weighted by their decoder-direction alignment.Source: §3.3 · p. 3

3.4 Stage 3: Gradient-Based Sensitivity Ranking

  • Gradient sensitivity ranks SAE features by the dataset-averaged derivative of the True/False behavior margin with respect to each feature activation.Source: §3.4 · p. 4

3.5 Stage 4: Causal Interventions

  • Interventions measure both behavior-flip rate and probe-margin shift, separating effects on model behavior from effects on the probe readout.The comparison tracks output coherence as well.Source: §3.5 · p. 4

3.5.1 Feature Ablation

  • Feature ablation subtracts selected SAE features’ reconstructed contributions from the residual stream before the model’s forward pass continues.The evaluated sets include probe-attributed, gradient-sensitive, and matched random features.Source: §3.5.1 · p. 4

3.5.2 Learned Sparse Mask

  • The learned sparse mask combines each feature’s decoder alignment with its activation, fitting weights to reconstruct the probe margin under an L1 penalty.Thirty features exceed a weight of 0.5; the top 16 are ablated to match other feature-set sizes.Source: §3.5.2 · p. 4

3.6 Stage 5: Overlap and Decomposition

  • The decomposition measures ranking overlap and partitions features into size-matched shared, probe-only, and random groups for ablation.Shared features are the intersection of the top-K probe-aligned and gradient-sensitive features; the shared-set size determines the matched comparison sizes.Source: §3.6 · pp. 4–5

3.7 Coherence Gating

  • Behavioral effects are interpreted only when outputs remain valid True/False responses, since interventions that disrupt well-formed output cannot isolate feature-specific causal roles.The SAE reconstruction has mean cosine similarity 0.89, and shared-set ablation removes 5.7% of residual-stream norm on average.Source: §3.7 · p. 5

4 Experimental Setup

The experiments apply the framework to instructed factual truth and deception using a True/False probe, Gemma-2-9B-Instruct, and matched feature ablations. Behavior flips are defined against the baseline margin, with coherence tracking whether outputs remain valid True/False responses.

4.1.1 Task and Concept

  • The task presents factual statements under Truthful, Deceptive, or Neutral instructions; prior work found instructed outputs linearly decodable across all three conditions.An auxiliary diagnostic uses a reward-hacking code probe trained on hardcoded versus correct code completions.Source: §4.1.1 · p. 5

4.1.2 Probe Relevance

  • The TTPD probe offers a binary behavioral signal for truthful versus deceptive instruction-following in factual verification and uses fixed pre-generation activations with matching GemmaScope SAEs.Source: §4.1.2 · p. 5

4.1.3 Model, Dataset and SAE

  • The study uses Gemma-2-9B-Instruct and 1,600 factual statements across twelve affirmative and negated sets, extracting final-prompt-token activations at layer 20.The held-out analysis derives rankings on one half of the data and measures ablations on the other.Source: §4.1.3 · pp. 5–6

4.1.4 Behavioral Signal and Coherence

  • A behavior flip is a change in the sign of the True/False margin under intervention relative to baseline, while coherence is the fraction of valid True/False outputs.Source: §4.1.4 · p. 6

4.1.5 Intervention Configurations

  • Feature ablations compare probe-attributed, gradient-sensitive, learned-mask, shared, probe-only, and size-matched random sets at K values from 32 to 1,024.Source: §4.1.5 · p. 6

5 Results

Geometric probe alignment and gradient sensitivity identify largely different SAE features, with only a small overlap across ranking scales.

5.1 Probe-aligned and gradient-sensitive features exhibit minimal overlap

  • The top 16, 50, 100, and 200 probe features share only 2, 7, 12, and 22 features, respectively, with the gradient-sensitivity ranking.The corresponding Spearman ρ is 0.10.Source: §5.1 · p. 6
  • The shared fraction stays near 11 to 14 percent across scales, one to two orders of magnitude above chance but a small minority of either set.Chance overlap values for the top 16, 50, 100, and 200 are 0.02, 0.15, 0.61, and 2.44.Source: §5.1 · p. 6
  • Geometric alignment with the probe direction and gradient sensitivity of the True/False margin identify largely different features.Source: §5.1 · p. 6

Appendix A.1.3 plots the full overlap curve.

Under coherence-preserving ablation, shared probe–model features alter behavior more than probe-only or random features, while activation-aware selection outperforms geometric ranking. Ablating the full probe direction confirms a causal effect that shared features localize.

5.2 Probe alignment alone does not predict behavioral effect.

  • Shared-feature ablation flips outputs on 13–27% of statements, versus 6% for probe-only and 1% for random features.The effect is reported at 16 and 54 shared features; output coherence remains 1.00, while probe-only ablation shifts the probe readout more than shared ablation (0.24 versus 0.05).Source: §5.2 · pp. 6–7, Figure 2 · p. 7
  • Shared-ablation effects peak near 96 features, then decline as lower-ranked features partially cancel flips, while outputs remain confident and well-formed.The reported set-level results of at most 54 shared features fall within the rising regime.Source: §5.2 · p. 7
  • The shared > probe-only > random ordering holds across five seeds and a held-out split.The held-out rankings are derived on one half of the data and ablation is measured on the other.Source: §5.2 · p. 7, Table 2 · p. 7
  • At 16 features, shared ablation flips 86 outputs True→False and only 4 False→True, shifting decisions toward False rather than restoring factual correctness.Baseline accuracy under the deceptive prompt is 0.40 and does not increase under ablation.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Geometric probe attribution understates behavioral relevance.

  • The activation-aware mask flips 17.6 ± 0.7% of statements, nearly three times the geometric top-16’s 6.1 ± 0.5%, despite only 3 overlapping features.The mask combines decoder alignment with activation; coherence is 100% throughout, and the random control flips 0.8 ± 0.6%.Source: §5.3 · p. 8
  • Geometric top-16 features shift the probe margin more than shared features (0.16 versus 0.05) but move behavior less, even while removing more residual-stream norm (13.9% versus 5.7%).The comparison indicates that behavioral relevance depends on which features are removed, not simply perturbation magnitude.Source: §5.3 · p. 8
  • Gradient sensitivity alone is a weak ablation guide: gradient-top features flip behavior at 0.054 ± 0.009, and gradients are local first-order proxies in nonlinear models.The intersection retains features that are both active and aligned, concentrating causal effect that neither single ranking achieves alone.Source: §5.3 · p. 9

5.4 The probe direction is causal, and the shared features localize it.

  • Removing the full TTPD direction flips behavior on 18.1 ± 1.1% of statements; 16 shared SAE features recover 12.1 ± 1.1%, versus 6.8 ± 0.6% for geometric top-16.The shared set comprises 16 of 16,384 SAE features and recovers two-thirds of the direction-level effect.Source: §5.4 · p. 9, Table 5 · p. 9

6 Conclusion

The paper introduces a feature-level framework for testing whether probe-attributed SAE features are behaviorally relevant. In its True/False truth-probe setting, shared probe–model features produce larger coherent behavior changes, while activation-aware selection recovers more behaviorally relevant features than geometry alone.

6 Conclusion

  • The paper introduces a framework that compares probe-aligned and gradient-sensitive SAE features through direct behavioral interventions.It uses these interventions to assess whether probe-attributed features are behaviorally relevant.Source: §6 · p. 9
  • Shared probe–model features produce larger coherent behavior changes than probe-only features, which primarily affect the probe readout.The conclusion states that geometric projection of probe weights alone is insufficient to identify features the model causally uses.Source: §6 · p. 9
  • Activation-aware selection that combines probe information with feature activations recovers substantially more behaviorally causal features than activation-free geometric alignment.The conclusion limits this finding to the studied setting.Source: §6 · p. 9

7 Limitations

The analysis is primarily a single case study, and its gradient-derived importance ranking and ablation validation do not establish deception as a general computational mechanism.

7 Limitations

  • The causal analysis focuses primarily on one model, layer, SAE, and behavioral setting, leaving generalization to other settings unclear.Most of the causal analysis is derived from a single primary case study.Source: §7 · p. 10
  • Whether the observed probe–behavior dissociation generalizes across model families, layers, SAE architectures, and behavioral domains remains unclear.The study includes auxiliary obfuscation-probe diagnostics, but most causal analysis remains in one primary case study.Source: §7 · p. 10
  • Gradient-derived feature importance and feature ablations identify influences on True/False behavior in this setting but do not establish a general computational mechanism for deception.The authors call for evaluating alternative attribution methods, interventions, and behavioral settings.Source: §7 · p. 10

A Appendix

Appendix A is headed by a section on the TTPD probe, citing Bürger et al. (2024) and Long et al. (2025).

A Appendix

A.1.1 Replication

The replication uses the TTPD truth direction as its fixed probe for attribution analyses. It reports strong separation on deceptive-instruction activations and uses layer 20, while excluding the accurate but regularization-sensitive LR probe from the main attribution experiments.

A.1.1 Replication

  • AUROC 0.98: the reproduced TTPD truth direction separates true from false statements on deceptive-instruction activations.The probe classifies statements by the sign of their projection onto tG, the retained truth component of centered layer-20 activations regressed on truth label and polarity.Source: §A.1.1 · p. 12
  • Figure 3 shows layer-wise LR and TTPD accuracy rising from chance to a high-accuracy plateau, with layer 21 marked as Long et al.’s reported peak.The experiments analyze layer 20 because of SAE constraints.Source: Figure 3 · p. 13
  • LR-probe training accuracy remains 0.980–1.000 across the tested regularization strengths, but its top-feature overlap with the unregularized solution varies from 0.62 to 0.07.The reported Jaccard indices are 0.62, 0.43, and 0.07 at C = 1.0, 0.1, and 0.01, respectively; the paper excludes LR from the main experiments because attribution requires a fixed probe direction.Source: §A.1.1 · pp. 12–13
  • The TTPD direction tG is hyperparameter-free, so the authors use it for all attribution analyses instead of the regularization-dependent LR probe.The probe is replicated using Long et al.’s released code and associated factual-statement datasets.Source: §A.1.1 · pp. 12–13

A.1.2 Learned Sparse Mask Details

The section details an activation-aware sparse mask and reports that shared-feature effects vary with ablation depth and dataset category. A Qwen reward-hacking probe serves as a boundary case where the SAE reconstruction captures only part of the raw probe signal.

A.1.2 Learned Sparse Mask Details

  • Top-N overlap between probe and gradient rankings exceeds chance by one to two orders of magnitude for small N, then both curves approach 1 near the full dictionary.Source: Figure 4 · p. 14

A.1.4 Margin dip

  • Flip rate peaks at 0.31 at N=96, then falls to 0.10 after all 126 shared features are ablated; lower-ranked features reverse the margin shift while confidence remains high.The signed margin falls from 2.28 to 0.10 at N=96, then rebounds to 1.87 at N=126; mean absolute margin stays between 1.7 and 2.8.Source: §A.1.4 · p. 14, Figure 5 · p. 15

A.1.7 Category Breakdown

  • Shared-feature ablation flips occur in 10 of 12 categories (0.014–0.243), with True→False dominating 61 to 7; two categories show no flips at m=16.The strongest movement is in translation (Sp–En, 0.243) and inventor (0.186) statements; the authors suggest magnitude differences likely reflect category-level baseline confidence.Source: §A.1.7 · pp. 14–15, Table 6 · p. 15

A.2 Probe from Taufeeque et al.

  • In three factual-verification examples, shared-feature ablation always flips True→False, while probe-only ablation leaves the output unchanged.The direction can either overturn a correct answer or restore correctness, depending on factual truth; across the evaluation set, True→False flips dominate.Source: Figure 6 · p. 16
  • For the Qwen reward-hacking probe, SAE reconstruction preserves 26% of raw margin (74% lies in reconstruction error); top-50 mask ablation flips 32.7% reconstructed versus 2.1% raw predictions.The authors treat this as a boundary case and say SAE-feature interventions require verifying that the probe signal is represented in the SAE basis.Source: §A.2 · pp. 16–17

Related papers

Loading 2609.18080v1…