PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Language Models to Explain Their Own Computations

Belinda Z. Li, Zifan Carl Guo, Vincent Huang, Jacob Steinhardt, Jacob Andreas

arXiv:2511.08579v3cs.CLcs.AIcs.LG

TL;DR

Verbalized explanations can sound plausible without describing the computations behind model decisions, raising the question of whether training can make them faithful. The paper fine-tunes explainer models to describe target models’ internal procedures and finds evidence across three question types that explanations leverage privileged access to the explainer’s own internals.

  • Problem

    Existing verbalized explanations can be plausible yet fail to describe computations that determine model decisions, leaving open whether training can make them faithful.

    Source: §1 · p. 1

  • Method

    The authors fine-tune explainer models to predict three aspects of target models’ internal procedures using explanations consistent with interpretability procedures.

    Source: §1 · p. 1, §7 · p. 8

  • Results

    Across three question types, generated explanations showed evidence of leveraging models’ privileged access to their own internal mechanisms.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Self-consistency objectives may offer an avenue toward more scalable interpretability methods and faithful explanations.

    Source: §7 · p. 8

    Self-verbalizations remain complementary to internal or counterfactual interpretability methods and require validation in high-stakes scenarios.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Can language models (LMs) learn to faithfully describe their internal computations? Are they better able to describe themselves than other models? We study the extent to which LMs' privileged access to their own internals can be leveraged to produce new techniques for explaining their behavior. Using existing interpretability techniques as a source of ground truth, we fine-tune LMs to generate natural language descriptions of (1) the information encoded by LM features, (2) the causal structure of LMs' internal activations, and (3) the influence of specific input tokens on LM outputs. When trained with only tens of thousands of example explanations, explainer models exhibit non-trivial generalization to new queries. This generalization appears partly attributable to explainer models' privileged access to their own internals: using a model to explain its own computations generally works better than using a *different* model to explain its computations (even if the explainer model is significantly more capable than the target). Our results suggest not only that LMs can learn to reliably explain their internal computations, but that such explanations offer a scalable complement to existing interpretability methods. Code and data at https://github.com/TransluceAI/introspective-interp

1. Introduction

The paper investigates whether fine-tuning can teach language models to faithfully describe their internal computations, and whether self-access improves those explanations.

1. Introduction

  • The paper is motivated by the possibility that prompted explanations may sound plausible yet fail to reflect the computations and decision-relevant factors that produced an output.It proposes training models to describe internal representations and mechanisms, not only their output distributions.Source: §1 · p. 1
  • The study trains explainers on three tasks: describing activated internal features, predicting activation-patching outcomes, and identifying input tokens important to decisions.The tasks use auto-interpretability, activation patching, and input ablation, respectively.Source: §1 · p. 1, Figure 1 · p. 2
  • The target model remains fixed while only the explainer is fine-tuned, so the paper uses “self-explaining” in a looser-than-strict sense.The authors choose this setup to explain pretrained models rather than enforce strict self-consistency.Source: §1 · p. 1
  • Explainers generally describe a target model’s computations better when explaining themselves, with privileged-access patterns across feature descriptions, activation patching, and input ablation.The advantage can hold even when the alternative explainer is larger or instruction-tuned.Source: §1 · pp. 1–2
  • Fine-tuning can teach models internal-feature explanations that outperform zero-shot methods and nearest-neighbor SAE features; explanation ability also correlates with explainer–target similarity.The paper notes that these capabilities are not always present without training.Source: §1 · p. 2
  • 0.8% of the training data yielded results comparable to a nearest-neighbor baseline, making self-explaining approximately 100 times more sample-efficient.The comparison is reported for the nearest-neighbors baseline.Source: §1 · p. 2

2. Methods

The methods train language-model explainers to verbalize internal computations using interpretability procedures as supervision. They cover feature descriptions, activation-patching effects, and input-token influence.

2.1. Definitions and Task

  • Explainer models learn to answer questions about target models by training on explanations generated by an external procedure T.The framework minimizes cross-entropy loss on the explanation.Source: §2 · p. 2, §2.1 · p. 3

2.2. Feature Descriptions: What does a model-internal feature represent?

  • Feature explainers describe inputs that activate a residual-stream direction, with descriptions selected to correlate simulated and actual activations.The explainer receives the direction as a continuous token; experiments train on sparse autoencoder features and test generalization to full activations and activation differences.Source: §2.2 · p. 3

2.3. Activation Patching: What internal components affect the prediction?

  • Activation patching tests causal effects by replacing an activation with one from a counterfactual run and measuring the resulting output change.Explainers predict both whether the output changes and the content of the patched output; the activation is inserted as a continuous token in the explainer’s embedding layer.Source: §2.3 · pp. 3–4, Figure 3 · p. 4

2.4. Input Ablation: What parts of the input matter?

  • Input ablation measures token influence by removing a subset of input tokens and observing whether and how the model’s answer changes.Explainers predict whether removal changes the answer and the content of the new output.Source: §2.4 · p. 4

3. Privileged Access Improves Explanations

Fine-tuned explainers generalize to new SAE features and other activation representations, and feature descriptions are generally better when a model explains its own computations than when another model does. The section evaluates this capability against baselines and finds that alignment between explainer and target activations improves performance.

3.1. Models and Training Data

  • The target is Llama-3.1-8B; five explainer models include it, related Llama variants, a larger Llama model, and Qwen3-8B, trained on 131K residual-stream SAE features.Ground-truth feature explanations come from Neuronpedia.Source: §3.1 · p. 5

3.2. Evaluation Set and Metrics

  • Evaluation tests 1,550 held-out SAE features and out-of-distribution full activations and counterfactual activation differences, using gold-description similarity and simulator scores.The LM judge scores similarity from 0 to 1 in 0.25 increments and has 81.25% human agreement.Source: §3.2 · p. 5

3.4. Results

  • Fine-tuned Llama-3.1-8B significantly outperforms every baseline on every feature type and metric, even gold SAE labels, whose ground-truth noise limits simulator scores.Table 1 reports the comparisons; the paper attributes the result against gold labels to noise in the feature labels.Source: Table 1 · p. 5, §3.4 · p. 6

Alignment between activations improves verbalization

  • Pretraining Llama-3.1-70B’s projection improves its explanations by 14% on SAE features, 2.7 times on real activations, and 1.7 times on activation differences.The comparison is against the randomly initialized projection; activation alignment improves explainer performance.Source: Table 1 · p. 5, Alignment between activations improves… · p. 6

3.3. Baselines

  • Baselines include nearest-neighbor descriptions from training features, SelfIE prompting with patched activations, and simulator-scored gold SAE labels.SelfIE uses the best explanation across five activation scales.Source: §3.3 · pp. 5–6

Alignment between activations improves verbalization

  • Among trained explainers, Llama-3.1-8B and Llama-3-8B beat Llama-3.1-8B-Instruct by a few points and significantly outperform Qwen3-8B and larger Llama-3.1-70B.The authors propose activation similarity between explainer and target as a predictor of explanation performance.Source: Alignment between activations improves… · p. 6

4. Privileged Access Confers Data-Efficiency

Self-explanation is more data-efficient than training another explainer or using nearest neighbors, and recovers much of the performance of other explanation techniques with orders of magnitude less data.

4. Privileged Access Confers Data-Efficiency

  • At 0.8% of training features, self-explanation retained substantial end-performance across held-out SAE and activation-explanation metrics.For the two held-out-SAE metrics, ACT simulator score, and ∆ACT simulator score, respectively, Llama reached 71%, 80%, 89%, and 81%; Qwen reached 35%, 24%, 46%, and 66%; nearest neighbors reached 55%, 56%, 55%, and 75%.Source: §4 · p. 6
  • Because Llama’s end-performance already exceeds Qwen’s and nearest neighbors’, the data-efficiency gap widens in low-data regimes.The study compares performance across training-data subset sizes against the SelfIE baseline.Source: §4 · p. 6
  • Self-explanation training can recover most of other explanation techniques’ performance with orders of magnitude less data, addressing the expense of annotating feature descriptions.The paper frames this as a practical benefit of training a model to describe its own features.Source: §4 · p. 6

5. Generalization Across Tasks

Across activation patching and input ablation, Qwen3-8B best explains its own intervention outcomes; untrained Qwen3-8B rarely predicts a change in the input-ablation task.

5. Generalization Across Tasks

  • The evaluation reports Has-Changed F1 for intervention-change predictions, Content Match for predicted output content, and Exact Match for correctly predicting both parts.Has-Changed F1 is macro F1 over changed and unchanged classes.Source: §5 · p. 6
  • Qwen3-8B best explains its own outcomes on both tasks, and trained explainers outperform untrained versions on activation patching.The study reports that untrained models generally guess the has-changed value randomly.Source: Table 2 · p. 7, §5.1 · p. 7, §5.2 · pp. 7–8

5.1. Activation Patching: Training LMs to Predict the Outcomes of Activation Interventions

  • Activation-patching training uses CounterFact factual subject–relation–object inputs and counterfactual pairs with the same relation but different subjects and objects, balancing changed and unchanged predictions.The method tests whether patching an activation changes the model’s most likely prediction.Source: §5.1 · p. 7

5.2. Input Ablations: Training LMs to Describe their Decision Rules

  • Input-ablation training uses MMLU questions with a randomly selected answer hint appended, balancing examples where the hint changes the model’s most likely next token and those where it does not.The task trains explainers to predict whether the hint changes the output and, if so, the changed content.Source: §5.2 · p. 7
  • Untrained Qwen3-8B predicted “changed” only 8.6% of the time; its content match was 96.4% for unchanged answers versus 10.3% for changed answers.The authors interpret this pattern as evidence that explicit fine-tuning is essential to elicit faithful explanations of decision rules.Source: §5.2 · p. 8

6. Related Work

The related work positions the study against challenges in faithful chain-of-thought, scalable mechanistic interpretability, and model introspection, and describes self-verbalization as an alternative way to make explanations accessible.

6.1. Chain-of-Thought Faithfulness

  • Prior chain-of-thought can support external monitoring, but has been found unfaithful to models’ decision processes; this motivated the hint ablation’s training on models’ own decision rules.Source: §6.1 · p. 8

6.2. Mechanistic Interpretability Methods

  • Mechanistic interpretability connects feature descriptions into causal circuits, but manual methods are hard to scale and generalize, while automated description and circuit-discovery methods often require substantial compute.Researchers continue to develop scalable methods that balance correctness and efficiency (Nanda, 2023; Syed et al., 2024; Shaham et al., 2025).Source: §6.2 · p. 8
  • LogitLens (nostalgebraist, 2020), PatchScopes (Ghandeharioun et al., 2024), and SelfIE (Chen et al., 2024a) enable zero-shot self-interpretation but require significant hyperparameter tuning.Source: §6.2 · p. 8
  • Latent Interpretation Tuning trains models to answer questions about other models’ inputs via activations, and later work scaled this approach to explanation settings (Karvonen et al., 2026; Huang et al., 2025).Goel et al. (2025) trained a LoRA adaptor to verbalize fine-tuning weight differences; this study trains models to self-verbalize internal computations using finer-grained interpretability data.Source: §6.2 · p. 8
  • Interpretability-based explanations can be difficult for non-experts to access and may require specialized interfaces; this work offers an “in-band” alternative (Viégas & Wattenberg, 2023; Chen et al., 2024b).Source: §6.2 · p. 8

6.3. Introspection & Metacognition

7. Discussion: Self-Consistency as a Framework for Interpretability

Across three question types, the paper finds evidence that generated explanations leverage models’ privileged access to their internal mechanisms. The authors propose self-consistency objectives as a possible route to scalable interpretability and alignment.

7. Discussion: Self-Consistency as a Framework for Interpretability

  • Across three question types, generated explanations show evidence of leveraging models’ privileged access to their own internal mechanisms.How explainer capacity, target complexity, and task difficulty affect this privileged access remains an open question.Source: §7 · p. 8
  • Self-consistency objectives (Anonymous Authors, 2026) could support scalable interpretability, faithful explanations, and alignment guarantees.The authors suggest applying generalized objectives to descriptions of inferred user traits, adversarial triggers, and influential training examples.Source: §7 · p. 8
  • Fine-tuning models to match generated explanations could offer a form of “unsupervised alignment” without large preference datasets or reinforcement learning.The authors note this could be normatively correct even if the explanations are inaccurate as self-descriptions.Source: §7 · p. 8

Impact Statement … A.2. Human Validation

Human preferences aligned strongly with the LM judge’s feature-description preferences, while the paper presents self-descriptions as a scalable aid to interpretation that still requires rigorous validation in high-stakes settings.

Impact Statement

  • Self-descriptions may support debugging, red-teaming, decision understanding, and broader explanation access while reducing the compute and human costs of interpretability experiments.The authors also propose that self-descriptions may make explanation access more accessible to non-experts and enable greater trust and interactivity.Source: Impact Statement · p. 9
  • Over-reliance on self-verbalizations could mislead users about trustworthiness, and improved self-understanding could increase deceptive capabilities, including behavior differing between evaluation and deployment (Laine et al., 2024).The authors say verbalizations should complement internal or counterfactual interpretability methods and be validated with more rigorous techniques in high-stakes scenarios.Source: Impact Statement · p. 9

A. Feature Description: LM Judge Results

  • GPT-4.1-mini judged predicted SAE feature descriptions against Neuronpedia gold descriptions on a 0–1 similarity scale in 0.25 increments.The judge was used to assess similarity between each predicted description and its gold description.Source: A. Feature Description: LM Judge Results · p. 14

A.1. Prompt

  • The LM judge prompt assessed similarity between ground-truth and predicted feature descriptions on a 1–5 scale, from completely unrelated (1) to same as expected or highly similar (5), treated as a correct match.Source: §A.1 · p. 14

A.2. Human Validation

  • For human validation, an annotator unfamiliar with the examples judged which prediction was better across 100 paired comparisons constructed from 200 test-set queries.The three choices were prediction 1, prediction 2, or equal; preference judgments were used because human and LM judges might have different scoring baselines.Source: §A.2 · p. 15
  • Across 100 pairs, human and LM-judge preferences had 64% exact agreement; weighting near-agreements of 0.25 points as 0.75 correct and 0.5 points as 0.5 correct raised agreement to 81.25%, with complete divergence in 3%.Source: §A.2 · p. 15

A.3. Error Analysis … C. Feature Descriptions: Additional Results

Many zero-scored outputs reflect flawed labels or judge assessments rather than explainer failures; the appendices also describe the explainer models and a layerwise nearest-neighbor baseline.

A.3. Error Analysis

  • Some explainer outputs improve on noisy or incorrect original labels, while judge errors can arise from slight differences between expected and predicted labels.A feature with comparison and contrast patterns was mislabeled as computational terminology by Neuronpedia, while the explainer identified its contrast function; sparse-activation features activate on fewer than 0.001% of examples and are difficult to characterize.Source: §A.3 · p. 15
  • Figure 6 attributes 43.5% of low-scoring cases to low-quality auto-interpretation labels, 11% to LM judge error, and 18% to sparse activation.The figure describes the remaining 27% as genuine explainer model errors.Source: Figure 6 · p. 15
  • 27.7% of cases scored 0 by the LM judge were genuine, unexplainable errors from the fine-tuned explainer, suggesting automated metrics may underestimate its performance.Figure 6 reports the rounded share as 27%.Source: §A.3 · p. 15, Figure 6 · p. 15

B.1. Explainer models

  • The feature-description study trains five explainers, including Llama-3.1-8B and the architecturally identical Llama-3-8B, whose context length and training data differ.Llama-3.1-8B received longer-context continual pretraining and multilingual inputs beyond English.Source: §B.1 · p. 15

3. Llama-3.1-8B-Instruct: took Llama-3.1-8B and did ad-

  • The comparison selects models to vary architecture, size, or training regimen as minimally as possible; Llama-3.1-70B shares Llama-3.1-8B’s regimen and basically its architecture, but is much larger.Qwen3-8B and Llama-3.1-8B both have 4096 hidden dimensions and 32 attention heads; Qwen3-8B has 36 layers and a 12,288 versus 14,336 MLP intermediate size, and was trained on approximately 36T tokens across 119 languages versus Llama’s 15T.Source: §3 · p. 16

B.2. Layerwise Nearest Neighbors

  • The layerwise nearest-neighbor baseline retrieves explanations only from training features in the corresponding test layer, rather than from all layers.The passage defines D as the function mapping training features to their explanations.Source: §B.2 · p. 16

C.1. Gemma-2-9B as Target Model … D.1. Llama-3.1-8B as Target Model

Across target models and activation-patching evaluations, explainers generally perform best on their own computations, while activation alignment tracks explanation quality. Feature descriptions change little when layer information is removed or made incorrect.

C.1. Gemma-2-9B as Target Model

  • For Gemma-2-9B, Gemma-2-9B and Gemma-2-9B-Instruct explain features better than other models, nearest neighbors, and untrained SelfIE.Self-explanation training also outperforms nearest neighbors and untrained SelfIE.Source: §C.1 · p. 16, Table 3 · p. 16
  • Instruction-tuning significantly affects Gemma-2-9B explanation ability, and Gemma-2-9B-Instruct’s added abilities matter more than its possible alignment gap with the base model.Source: §C.1 · p. 16

C.2. Quantifying privileged access via activation alignment

  • The study measures activation similarity using corresponding-position dot products and Pearson correlations of SAE-feature activation patterns on top exemplars.Source: §C.2 · p. 16
  • Activation alignment is generally positively correlated with explainer quality across metrics, with a few outliers such as Llama-3.1-70B on held-out SAE features.The authors interpret this as evidence that activation alignment contributes to verbalization capability.Source: §C.2 · pp. 16, 18, Figure 7 · p. 17

C.3. Is Layer Annotation Necessary to Describe SAE Features?

  • 77.3% of generated tokens matched when layer annotations were removed, and 75.2% when replaced with an incorrect layer; cross-layer nearest-neighbor retrieval also outperformed same-layer retrieval.The match rates use 100 test prompts and fixed 10-token rollouts; they do not count semantically equivalent wording or token-order differences.Source: §C.3 · p. 18

D.1. Llama-3.1-8B as Target Model

  • Llama-3.1-8B and Qwen3-8B each explain their own activation-patching outcomes better than the other model and an untrained baseline.Source: §D.1 · p. 18

D.2. Ablation Analysis · E. Input Ablation: Additional Results

The ablations suggest that layer and token annotations can often be omitted because their information is recoverable from the activation and input context. Llama-3.1-8B decoded layer and token information from those inputs with 98.7% test-set exact-match accuracy.

D.2. Ablation Analysis

  • The ablations removed one of three explainer inputs—activation, layer, or token—while conditioning on the other two to generate ground-truth explanations.The input components are the activation to insert, layers ℓ1:i, and the target-model token to patch into.Source: §D.2 · p. 18
  • The authors attribute layer and token robustness to their being recoverable from the activation and input context.Source: §D.2 · p. 18, Table 5 · p. 19
  • Llama-3.1-8B was trained to decode layer and token information using only the input context and activation value.Source: §D.2 · p. 18
  • Ablating layer or token annotations generally had negligible effect on activation-patching outcomes.Table 5 reports the results across target models and metrics; its caption notes negligible effects for these two ablations.Source: §D.2 · p. 18, Table 5 · p. 19
  • 98.7% exact-match accuracy on the test set was achieved when Llama-3.1-8B decoded layer and token information from activation and input context.The test set used the same train/test division as the main experiments in Section 5.1.Source: §D.2 · pp. 18–19
  • The ablation study used the same train/test split as the main experiments in Section 5.1.Source: §D.2 · p. 19

E.1. Full Results by Explainer … H.3. Input Ablations

The appendix reports full input-ablation results and documents the prompts, dataset construction, and untrained baselines used for the explanation tasks.

F. Explainer Model Prompts

  • Prompts cover feature descriptions, activation patching, and input ablations, with output formats conditioned on whether a patch or hint changes the prediction.The appendix lists the explainer prompts and output formats for each explanation type.Source: F. Explainer Model Prompts · pp. 19–20

E.1. Full Results by Explainer

  • Table 6 reports evidence of privileged access in input ablations, even though Llama-3.1-8B is a weaker explainer overall than Qwen3-8B.The table groups results by explainer and compares self-explanation, explaining the other model, and untrained models across exact match, has-changed F1, and content prediction.Source: §E.1 · p. 19, Table 6 · p. 20

G. Data Generation Pipeline Details; G.1. Feature Descriptions

  • Feature-description data uses LlamaScope’s Llama-3.1-8B-LXR-32x SAE features, with Neuronpedia explanations and 50 held-out features per layer.The model has 32K features per layer and 131K total; layer 3 has no Neuronpedia features.Source: §G.1 · p. 20, G. Data Generation Pipeline Details · p. 20

G.2. Activation Patching

  • Activation-patching data uses CounterFact counterfactual pairs, searches layer chunks and token positions, and balances has-changed categories to limit spurious correlations.Prompts include five answer options shared across each counterfactual pair; the dataset categorizes tokens including subject-final, relation, and answer-option tokens.Source: §G.2 · pp. 20–21

G.3. Input Ablations

  • Input-ablation data injects hints into MMLU questions and uses target-specific hint prompts to ensure samples where answers both change and remain unchanged.Table 8 reports changed/unchanged dataset breakdowns for the Llama-3.1-8B-Instruct and Qwen-3-8B targets across splits.Source: §G.3 · p. 21, Table 8 · p. 23

H. Untrained Model Prompts; H.1. Feature Descriptions (SelfIE)

  • Untrained models serve as baselines for all three tasks; feature-description prompts use SelfIE, adapted from prior zero-shot self-explanation work.Prompts for the other explanation types are hand-crafted to elicit the correct response format and avoid invalid-response errors.Source: H. Untrained Model Prompts · p. 21, §H.1 · p. 21

H.2. Activation Patching

  • The untrained activation-patching prompt asks whether patching a token changes the output and requires one of two exact response forms.The target output is specified as the next token following the input.Source: §H.2 · p. 21

H.3. Input Ablations

  • The untrained input-ablation prompt asks how removing a hint changes a multiple-choice answer and requires one of two exact output options.The answer is represented by a single letter, with no explanation or reasoning in the response.Source: §H.3 · p. 23

Related papers

Loading 2511.08579v3…