PaperlayerView exact revision on arXiv

Source-linked AI summary

How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attribution

Vedant Palit, Florent Draye, Terry Jingchen Zhang, Bernhard Schölkopf, Zhijing Jin

arXiv:2608.27510v1cs.CLcs.LG

TL;DR

Existing logit-targeted attribution and linear probes leave a gap between token-generation explanations and mechanistic explanations of internal concept representations. CTA targets a linear probe direction with attribution graphs, and its graphs show predictive structure and functionally distinct concept and generation mechanisms. The paper therefore connects probe performance with interpretable circuit structure while identifying scope limits from transcoder quality and probe validity.

  • Problem

    Logit-targeted graphs explain generated tokens, while linear probes show concept decodability without identifying the sparse computations supporting it.

  • Method

    CTA uses a linear probe score as the scalar target for Cross-Layer Transcoder attribution graphs, then analyzes graph structure, local features, and causal ablations.

  • Results

    Across four concepts, graph-level features predict probe accuracy (ρ = 0.91, R2 = 0.84), local features identify per-prompt drivers, and ablations dissociate concept encoding from generation.

  • Takeaways & Limitations

    CTA provides mechanistic evidence for auditing internal concept representations by linking probe performance to causal features and graph structure.

  • Takeaways & Limitations

    CTA inherits Cross-Layer Transcoder omissions and conflations, while linear probes may capture correlated stylistic features rather than conceptual features.

Abstract

from arXiv · show

Transcoder attribution graphs are usually trained to explain why a model assigns high probability to a particular next token. We introduce Concept-Targeted Attribution (CTA), which instead trains attribution graphs with respect to a linear probe direction. CTA therefore yields probe-specific circuits that explain why an internal concept representation arises in a prompt, independently of whether it is expressed in the generated token. Using Cross-Layer Transcoders, we show that these probe-targeted graphs contain predictive structure: graph-level features predict probe accuracy across four widely studied concept categories ($ρ= 0.91$, $R^2 = 0.84$), while local features identify the sparse components driving per-prompt classification. This connects probe performance to interpretable circuit structure, allowing us to ask not only whether a probe works, but which internal computations make it work. Causal ablations further show that probe-targeted and logit-targeted graphs capture functionally distinct mechanisms. Removing probe-relevant features reduces internal concept scores while largely preserving generated tokens, whereas removing logit-relevant features changes the generated token in 92% to 100% of cases with near-zero effect on probe scores. CTA provides a framework for moving from behavioral probe accuracy to mechanistic explanations of probe performance, enabling more detailed audits of internal concept representations, including safety-critical ones. Our code is available at https://github.com/vedantpalit/concept-targeted-attribution

1 Introduction

Existing attribution graphs explain next-token logits, while probes reveal concept decodability without identifying the computations behind it. CTA targets the probe direction to construct concept-specific attribution graphs and tests their structure and functional distinction from generation circuits.

  • Motivation: Next-token attribution graphs can miss internally represented concepts that are not expressed in the generated token.A safe continuation may conceal recognition of toxic content in the model’s internal representation.
  • Motivation: Linear probes show whether concepts are linearly accessible but not which sparse features support detection or how concept signals relate to generation.
  • Contribution: CTA constructs attribution graphs with respect to internal concept representations rather than output tokens.It uses a linear probe score as the scalar attribution target.
  • Evaluation: CTA is applied to toxicity, sentiment, reasoning, and truthfulness probes.
  • Findings: ρ = 0.91 and R2 = 0.84: graph-level features predict probe validation accuracy across four concepts, while local features identify sparse latents driving per-prompt classification.
  • Findings: A causal double dissociation separates concept and generation circuits: probe-exclusive ablations reduce concept scores while largely preserving tokens, whereas logit-exclusive ablations change tokens in 92%–100% of cases with near-zero probe effects.

2 Related Work

Circuit discovery traces causal computational flow, while dictionary-learning methods provide sparse feature-level representations. Attribution graphs usually target output logits, and linear probes establish concept decodability without explaining the circuit producing it.

  • Circuit discovery: Activation patching tests whether replacing an activation changes model output, while automated methods search across components to recover larger computational flow graphs.
  • Dictionary learning: Sparse autoencoders represent residual-stream vectors with sparse latent features, and transcoders learn sparse features that reconstruct MLP outputs.
  • Dictionary learning: Cross-layer transcoders extend transcoder representations across layers for analyzing model computations.
  • Logit-targeted attribution: Standard attribution graphs target a next-token logit and trace sparse features and paths that increase its probability.
  • Diagnostic probing: Linear probes classify concepts from hidden activations, establishing when a concept is linearly decodable at a layer.
  • Open gap: Probes and related concept-direction methods do not by themselves explain which sparse features assemble the readout or how it relates to token-generation paths.

3 Methodology

The methodology replaces MLP computations with sparse Cross-Layer Transcoder features and traces either a concept-probe score or a next-token logit. It evaluates concept emergence, causal dissociation, and whether graph structure predicts probe quality.

  • 3.1 Transcoder Attribution Graphs: Transcoder attribution graphs replace MLP modules with sparse feature models whose activations reconstruct MLP outputs.
  • 3.1 Transcoder Attribution Graphs: Attribution graphs represent sparse features as nodes and attribution weights as edges, enabling tracing through linearized model computations.
  • 3.1 Transcoder Attribution Graphs: CTA changes the scalar attribution target from a next-token logit to the linear probe score Sθ̄(xℓ) = θ̄⊤xℓ.
  • 3.2 Concept Representation via Probing: Each concept has a distinct belief point Lb, defined as the layer with the greatest single-step rise in mean positive-class probe score.
  • 3.1 Transcoder Attribution Graphs: A CTA graph traces sparse features and paths that make a concept detectable for a particular prompt, distinct from dataset-level probe accuracy.
  • 3.2 Concept Representation via Probing: The study trains residual-stream linear classifiers for toxicity, sentiment, reasoning, and truthfulness, orthogonalizing concept directions with maximum |cos(θ̄)| = 0.069.
  • 3.3 Causal Dissociation: Causal dissociation ablates CTA-exclusive and LTA-exclusive features to test whether concept scores and generated tokens depend on distinct pathways.
  • 3.4 Predicting the Quality of a Probe: The graph-structural predictor uses 35 structural features per graph, averages them by concept-layer pair, and predicts probe validation accuracy under group-aware cross-validation.

4 Experiments

Across four concepts, CTA reveals distinct emergence patterns and separates concept-detection from token-generation circuits. Graph structure predicts probe quality, while feature-level analyses identify shared, concept-specific, and cross-layer mechanisms underlying probe performance.

  • 4.1 Concept Emergence and the Belief Point: Concept representations consolidate at different layers: Toxicity and Sentiment later, Truthfulness mid-network, while Reasoning shows transient suppression from L6 to L14 before recovering at L15.These trajectories indicate that concepts differ in when their representations consolidate, not merely in whether they are linearly decodable.
  • 4.2 Structural and Functional Dissociation of Concept and Generation Circuits: Probe-exclusive ablation reduces concept scores while preserving generated tokens, whereas logit-exclusive ablation flips tokens in 100% of cases, or 92% for Reasoning, with ΔS ≈0.Together, the interventions demonstrate a functional double dissociation between internal concept encoding and output production.
  • 4.3 Graph-Structural Features Predict Probe Quality: Graph-structural features predict within-concept layer variation in probe accuracy beyond layer depth, with bootstrap ρ remaining above 0.72 and permutation tests rejecting trivial explanations.High-accuracy layers show concentrated, stronger activations, sharper routing, stronger read-out edges, and tighter graph density.
  • 4.4 Feature-Level Prediction of Probe Classification Power: Feature-level analyses find concept-specific mechanisms: sentiment features partly track domain and polarity, reasoning markers remain prompt-specific, and truthfulness features more sharply detect falsehood than truth.These findings show that predictive features can be informative while retaining dataset- or prompt-specific limitations.
  • 4.5 Cross-Prompt Consistency of Concept-Attribution Graphs: Top predictive features recur 6–21 times more often than random features, and circuits across prompts converge on a shared predictive core despite peripheral variation.Neutral, analytic, or factual-register features can suppress toxicity while supporting reasoning or truthfulness, indicating distinct lexical cues across concepts.
  • 4.6 Cross-Layer Stability of Concept-Attribution Graphs: Circuit composition changes rapidly in early layers before stabilizing at approximately J = 0.9 from intermediate layers through the final layers.Highest-weight predictive features concentrate early and form a semantic core, while later layers add refinements without displacing it.

5 Discussion

CTA provides mechanistic grounding for probe-based concept detection by identifying causal features and graph structures associated with encoding quality. Across models and prompts, concept circuits show separable, structured, and reproducible organization.

  • The features causally responsible for probe scores largely do not overlap with those driving generation across Gemma-2-2B and Llama-3.2-1B.This supports targeted interventions on concept representations while preserving broader generative behavior.
  • Circuit structure as a diagnostic of concept encoding: Top predictive CLT features are concept-specific and appear 6–21 times more frequently than random features across prompts.Their repeated recruitment forms a stable feature core for concept encoding.
  • Circuit structure as a diagnostic of concept encoding: Attribution-graph topology can serve as a probe-free indicator of concept internalization at a given layer.The proposed diagnostic could compare model behavior across training checkpoints or model variants.
  • Circuit stability as evidence of structured encoding: Shared feature cores and early circuit stabilization indicate that concept encoding is structured and reproducible across prompts and layers.The discussion presents this consistency as necessary for causal claims about concept encoding to generalize beyond individual inputs.

6 Conclusion

The paper introduces CTA to explain internal concept representations through causal circuit structure rather than probe scores alone. Its findings indicate that concept encoding and token generation use distinct, separable circuits with shared feature cores.

  • CTA grounds probe-based findings in causal mechanistic evidence across two model families.The framework makes probe-targeted concept representations objects of causal interpretability and intervention.
  • Concept-encoding and generation recruit causally non-overlapping circuits, while concept circuits converge on a shared feature core across prompts.The shared core strongly drives concept encoding, and graph structure predicts encoding quality without labeled supervision.
  • Knowing and saying recruit mechanistically distinct and causally separable circuits.This conclusion makes probe-targeted concept representations tractable targets for causal interpretability and intervention.

Limitations

CTA’s mechanistic explanations are bounded by the representational quality of the underlying CLT and by limitations of linear probes and feature-presence analysis.

  • CTA cannot expose features that the CLT conflates, omits, or represents polysemantically.The CLT’s sparsity–reconstruction trade-off also shapes which features appear in circuits without concept-specific tuning.
  • Linear probes are necessary but not sufficient for establishing conceptual validity.Correlated label distributions may cause probes to capture stylistic rather than conceptual features.
  • Feature-presence analysis discards activation magnitude and may miss concepts whose discriminative signal lies in activation geometry.
  • Future work should test broader architectures, deeper fine-grained interventions, and richer concept taxonomies beyond the four studied concepts.

A.1 Dataset Curation

The appendix describes the concept datasets, probe construction, CLT representation, and attribution-graph analysis used to study four semantic concepts across transformer layers. It also specifies the graph features, prediction targets, pruning, and model comparisons used in evaluation.

  • Dataset Curation: The study evaluates Gemma-2-2B and Llama-3.2-1B on Toxicity, Sentiment, Reasoning, and Truthfulness.
  • Probe Training Details: The probes use residual-stream linear classifiers with IID corrections to obtain orthogonal concept directions for accuracy evaluation.Raw DIM directions are used for CTA attribution, while IID-corrected directions are restricted to evaluation.
  • Dataset Curation: Each concept uses 5,000 positive and 5,000 negative examples, an 80/20 within-class train/validation split, and inputs limited to 60 tokens.Sampling is balanced, shuffled with seed 42, and intended to reduce class-imbalance bias and aid graph interpretation.
  • Cross-Layer-Transcoder: Cross-Layer Transcoders represent sparse features whose effects can persist, change, and be reused across layers.Unlike layer-local transcoders, CLTs allow reconstructed later-layer MLP outputs to depend on features from all preceding layers.
  • Experiments: Graph-level analysis extracts 35 structural features per prompt to predict layer-specific probe validation accuracy, while local analysis predicts probe margins from binary feature presence.The local predictor uses top-100 activation-ranked CLT features and Ridge regression; the graph analysis averages features over 100 prompts per concept-layer pair.
  • Probe Validation: Figure 7 shows that toxicity is linearly decodable from early layers in both models, while Sentiment and Truthfulness have weaker Llama-3.2-1B trajectories.

B.1 Belief layer selection.

CTA attribution uses a belief layer selected by encoding velocity rather than simply peak probe accuracy, then evaluates causal dissociation between probe- and logit-targeted circuits. The ablations show concept-specific separation, including a dominant shared pathway for Truthfulness and an anti-aligned Reasoning probe.

  • Belief layer selection: The belief layer is the steepest rise in mean positive-class probe score within a 2–8% peak-accuracy plateau, rather than necessarily the highest-accuracy layer.This targets active concept construction while requiring stable adjacent-layer circuits.
  • Causal dissociation: Probe-exclusive ablation reduces internal concept scores while preserving generated tokens, whereas logit-exclusive ablation changes tokens with near-zero probe-score effects.The causal subsets are probe-exclusive, logit-exclusive, and shared nodes.
  • Reasoning: Reasoning’s anti-aligned L15 probe reverses the sign interpretation: probe-exclusive ablation raises the score toward zero, while logit-exclusive ablation causes a 92% token-flip rate with near-zero internal effect.The baseline probe score is −21.44, so ablation reduces magnitude rather than ordinary score direction.
  • Truthfulness: Truthfulness has a dominant shared subgraph: ablation causes a 93% probe-score drop and a 67% token-flip rate, while logit-exclusive ablation remains cleanly separable.The shared component carries effects on both concept and generation signals.
  • Structural predictability: Graph-level predictability is statistically supported by cross-validation and permutation testing, with the observed association approximately six standard deviations above the null mean.The layer baseline yields negative R2, while GradientBoosting substantially exceeds it.

B.3 Detailed Feature-Level Study

The feature-level study finds that CTA circuits differ substantially by concept: toxicity is driven by stable early lexical features, while sentiment and reasoning require broader structure or show weaker feature recovery. Truthfulness uses sparse mid-layer features with asymmetric sensitivity to falsehood markers.

  • Toxicity: Toxicity’s predictability exceeds 0.66 at every layer and peaks at ρ = 0.86 at L10, driven largely by two early lexical features.L0/F8116 and L0/F4345 appear in roughly half the prompts and shift predicted probe margins by over 17 points.
  • Sentiment: Sentiment’s Ridge R2 peaks at 0.23 at L10 before falling to 0.08 at its best probe layer, because individual features capture register or affective cues rather than polarity.Sentence-level metaphor, hedging, and negation require integration beyond individual feature presence.
  • Reasoning: Reasoning reaches only 0.14 peak Ridge R2, but mid-to-late features are enriched in prompts where the probe transitions from incorrect to correct, with odds ratios up to 8.8×.An early feature is consistently present in reasoning graphs but absent from memorisation graphs.
  • Cross-concept feature stability: Top predictive features appear 8–21× more often than random features at the highest-validation-accuracy layer, indicating a stable mechanistic core despite prompt-level variation.This frequency comparison is reported across concept circuits.
  • Truthfulness: Truthfulness peaks at ρ = 0.66 at L12, with sparse mid-layer features producing sharp margin shifts when active and negative predictors outweighing positive predictors in magnitude.L1/F3007 is a frequent exception associated with factual claims and neutral encyclopedic register.
  • Cross-concept features: Some features recur across concept circuits with interpretable coefficient signs, including features negative for Toxicity but positive for Reasoning or Truthfulness.These shared features suggest limited semantic reuse with concept-dependent directions.

B.5 Case Study: Truthfulness vs. Reasoning

Truthfulness and Reasoning are semantically close and have adjacent belief layers, yet their attribution graphs are largely disjoint. Shared features remain a small minority, with overlap concentrated in similarly aligned depth regimes.

  • Concept relationship: Truthfulness and Reasoning have adjacent belief layers, L14 and L15, making them the strongest candidates for shared circuit vocabulary.Both concepts concern the epistemic quality of language.
  • Layer alignment: Cross-concept Jaccard overlap is elevated in early layers at approximately 7 or below and late layers at approximately 16 or above, while off-diagonal layer pairs approach zero.Layer alignment is necessary but not sufficient for sharing.
  • Circuit composition: Shared features remain below approximately 40 per prompt, compared with roughly 460 Reasoning-exclusive and 250 Truthfulness-exclusive features by the final layers.The overlap therefore represents only a small fraction of either circuit.
  • Overall comparison: Even this semantically proximate pair recruits largely disjoint attribution graphs, with overlap dependent on layer alignment and never exceeding a small minority of circuit features.The result extends the many-circuits finding across concept boundaries.
  • Causal replication: On Llama-3.2-1B, the core dissociation replicates: probe-exclusive ablation causes measurable probe effects with 0% token flips, while logit-exclusive ablation causes 100% flips near baseline probe scores.The pattern holds across all four concepts.

D.2 Graph-Structural Predictability: Llama-3.2-1B

In Llama-3.2-1B, graph structure predicts raw probe accuracy but separates layers within concepts less reliably than in Gemma-2-2B. Concept transfer is generally poor, although late-layer holdout remains strong because one structural feature is consistent across depth.

  • Raw structural predictability: GradientBoosting reaches ρ = +0.808 and R2 = +0.850 on raw validation accuracy, exceeding a layer baseline with negative R2.This replicates the direction of the Gemma-2-2B Channel 2 finding.
  • Within-concept separation: Within-concept structural separation is weaker, with ρ = +0.34 and bootstrap CI [−0.055, +0.743], versus ρ = +0.908 [+0.817, +0.962] for Gemma-2-2B.The comparison is attributed to fewer layers and near-flat Sentiment and Truthfulness accuracy trajectories.
  • Concept transfer: Held-out concept ρ ranges from −0.74 to +0.48, indicating poor concept transfer compared with Gemma-2-2B’s +0.93–+0.95 for Sentiment and Reasoning.Late-layer holdout is an exception, reaching ρ = +0.90 and R2 = +0.905.
  • Topological fingerprints: Toxicity and Reasoning show clearer high- versus low-accuracy separation on global graph features, while Sentiment and Truthfulness show little separation across feature families.The cited distributions include graph density and mean indegree among the separating features.
Loading 2608.27510v1…