PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Language Models to Explain Their Own Computations

Belinda Z. Li, Zifan Carl Guo, Vincent Huang, Jacob Steinhardt, Jacob Andreas

arXiv:2511.08579v3cs.CLcs.AIcs.LG

TL;DR

The paper asks whether language models can faithfully describe their internal computations and whether privileged self-access improves such explanations. It uses interpretability procedures as ground truth to fine-tune explainers for features, activation effects, and influential input tokens, finding evidence that models generally explain their own computations better than different models and can do so with high data efficiency.

  • Problem

    Language-model explanations can be plausible without corresponding to the computations that produced decisions, motivating methods for more faithful descriptions of internal representations and mechanisms.

  • Method

    The authors use interpretability procedures to generate training targets, then fine-tune explainer models to describe internal features, activation-patching outcomes, and influential input tokens.

  • Results

    Across the studied tasks, target models are generally explained better by themselves than by different explainers, including more capable alternatives, while trained self-explainers outperform baselines.

  • Takeaways & Limitations

    The findings support introspective interpretability as a scalable complement to existing methods for understanding language-model behavior.

  • Takeaways & Limitations

    The target model is fixed and only the explainer is fine-tuned, so the paper’s “self-explaining” setup is looser than strict self-consistency.

Abstract

from arXiv · show

Can language models (LMs) learn to faithfully describe their internal computations? Are they better able to describe themselves than other models? We study the extent to which LMs' privileged access to their own internals can be leveraged to produce new techniques for explaining their behavior. Using existing interpretability techniques as a source of ground truth, we fine-tune LMs to generate natural language descriptions of (1) the information encoded by LM features, (2) the causal structure of LMs' internal activations, and (3) the influence of specific input tokens on LM outputs. When trained with only tens of thousands of example explanations, explainer models exhibit non-trivial generalization to new queries. This generalization appears partly attributable to explainer models' privileged access to their own internals: using a model to explain its own computations generally works better than using a *different* model to explain its computations (even if the explainer model is significantly more capable than the target). Our results suggest not only that LMs can learn to reliably explain their internal computations, but that such explanations offer a scalable complement to existing interpretability methods. Code and data at https://github.com/TransluceAI/introspective-interp

1. Introduction

The paper asks whether language models can be trained to faithfully describe their internal computations, exploiting privileged access to their own internals. It studies three explanation targets: internal features, causal effects of activations, and influential input tokens.

  • Motivation: The paper targets a gap in verbal explanations, which can sound plausible while failing to describe the computations that produced LM decisions.Prior attribution, feature-visualization, and mechanistic-interpretability work motivates testing whether training can improve faithfulness.
  • Hypothesis: The privileged access hypothesis predicts that models trained to explain their own computations can be more accurate than other models trained to explain them.The paper extends earlier work on describing output distributions to internal representations and mechanisms.
  • Approach: The study fine-tunes explainer models to describe internal features, activation-patching outcomes, and input tokens that affect decisions.Feature descriptions concern activating inputs; activation patching concerns interventions on internal activations; input ablation concerns important decision tokens.
  • Scope: The target model remains fixed while only the explainer is fine-tuned, so “self-explaining” is used in a looser sense than strict self-consistency.The authors choose this setup to explain pre-trained target models and discuss its implications later.

B Activation Patching

Across feature descriptions, activation patching, and input ablation, models show privileged access: targets generally explain themselves better than other explainers, including more capable alternatives. Fine-tuning also makes self-explanation substantially more data-efficient than nearest-neighbor baselines.

  • Cross-task results: Privileged access holds across all three explanation tasks, with target models generally explained better by themselves than by other explainer models, even more capable ones.The reported pattern includes activation patching and input ablation as well as feature descriptions.
  • Feature descriptions: Fine-tuning enables models to self-explain internal features, outperforming zero-shot methods and nearest-neighbor SAE features while benefiting from explainer–target similarity.The authors state that explanation capabilities are not always natively present and can be acquired through a small amount of fine-tuning.
  • Data efficiency: Self-explaining is approximately a hundred times more sample-efficient than nearest neighbors, achieving comparable results with only 0.8% of the training data.The broader scaling analysis reports greater data efficiency than training another explainer or using nearest neighbors.
  • Implications: The results support training explanation capabilities through objectives that enforce consistency between generated explanations and internal procedures.The paper frames introspective interpretability as a scalable complement to external interpretability analysis.

2. Methods

The method trains language-model explainers using existing interpretability procedures as supervision to verbalize feature descriptions, activation-patching effects, and input-ablation effects. Explainers receive target-model representations or inputs and learn to predict how internal interventions or removals change model outputs.

  • General framework: Explainers are trained from model-external interpretability procedures that generate natural-language answers to questions about target-model computations.The framework minimizes cross-entropy on explanations produced by a procedure T mapping a target model and question to an explanation.
  • Feature descriptions: Feature descriptions identify what inputs activate a residual-stream direction, using simulator-selected explanations matched to observed activation patterns.A simulator estimates expected activations for candidate descriptions, and the explanation maximizing correlation with true activations is used for supervision.
  • Feature descriptions: Explainers receive target-model feature vectors as continuous embedding-layer tokens, with learned per-layer projections when target and explainer dimensions differ.Training uses Sparse Auto-Encoder features, while evaluation tests generalization to full activations and activation differences.
  • Activation patching: Activation patching compares the original output with a counterfactual run in which a selected activation is replaced, then trains explainers to predict output change and patched-output content.The target activation at a specified layer and token position is replaced by the corresponding counterfactual activation, and the output difference is measured with d(·, ·).
  • Scope and limitation: Early experiments found that explainers struggled to reproduce explanations for features from target-model components outside the residual stream.The authors hypothesize that features must be patched into the same type of location, with the embedding layer treated as part of the residual stream.
  • Input ablation: Input ablation removes a subset of input tokens, measures the resulting output difference, and trains explainers to predict both whether the answer changes and the new output.The explainer is trained on questions about how removing ˜x from x affects the model’s answer.

3. Privileged Access Improves Explanations

Fine-tuned models can describe unseen internal features and representations, with the target model generally explaining itself better than other models. Performance improves with activation alignment, supporting privileged access as a source of explanation quality.

  • Evaluation: Fine-tuned explainers generalize from SAE features to held-out features, full residual-stream activations, and counterfactual activation differences.The evaluation tests 1550 held-out SAE features, FineWeb activations, and differences from counterfactual input pairs.
  • Evaluation: The study evaluates feature explanations with LM-judge similarity and simulator correlation, and compares them against nearest-neighbor, SelfIE, and gold-label baselines.Simulator score measures Pearson correlation between true and predicted activations.
  • Results: Training Llama-3.1-8B to explain itself outperforms all baselines across every feature type and metric, including gold SAE labels.The evaluation covers held-out SAE features, full activations, and activation differences, using LM-judge and simulator scores.
  • Results: Llama-3.1-8B and Llama-3-8B consistently outperform Llama-3.1-8B-Instruct, Qwen3-8B, and Llama-3.1-70B despite the latter model’s larger size.The authors posit that activation similarity between explainer and target predicts performance.
  • Activation alignment: Pre-training a projection to align explainer and target activations improves Llama-3.1-70B performance by 14% on SAE explanations, 2.7 times on real activations, and 1.7 times on activation differences.The aligned projection recovers a significant fraction of performance relative to the randomly initialized projection.

4. Privileged Access Confers Data-Efficiency

Self-explanation is more data-efficient than training another explainer or using nearest neighbors. With only 0.8% of training features, Llama recovers much of its end-performance, widening its advantage in low-data settings.

  • Scaling results: Self-explanation is more data-efficient than training another explainer model or using nearest neighbors, and its advantage widens in low-data regimes.Llama also has higher end-performance than Qwen and nearest neighbors.
  • Practical implication: Self-explanation can recover most of the performance of other explanation techniques with orders of magnitude less annotated data.The paper highlights this as practically useful because feature-description annotation is expensive.

5. Generalization Across Tasks

The paper tests whether privileged access helps models explain activation interventions and input ablations, finding that self-explainers outperform alternatives across tasks. Fine-tuned explainers improve over untrained baselines, while matched models are especially data-efficient.

  • 5. Generalization Across Tasks: Training evaluates intervention explanations with exact match, has-changed F1, and content match, requiring predictions of both output change and changed content.Exact match requires correctly predicting both components.
  • 5. Generalization Across Tasks: Figure 5 shows that matching the explainer to the target is more data-efficient than Qwen or nearest neighbors, especially in the low-data regime.The comparison spans SAE features, full activations, and activation differences; SelfIE performance is an aggregated upper bound.
  • 5.1. Activation Patching: Training LMs to Predict the Outcomes of Activation Interventions: Activation-patching training uses CounterFact counterfactual pairs and balances changed versus unchanged predictions across token positions and layer chunks.The intervention perturbs a model component and tests whether its output changes.
  • 5. Generalization Across Tasks: Qwen3-8B is best at explaining Qwen3-8B for both activation patching and input ablation, supporting privileged access across tasks.All trained explainers outperform their untrained versions.
  • 5.2. Input Ablations: Training LMs to Describe their Decision Rules: Input-ablation training tests whether withholding or including injected hints changes a multiple-choice model prediction, using balanced changed and unchanged examples.The data combine MMLU questions with randomly selected answer-option hints.
  • 5.2. Input Ablations: Training LMs to Describe their Decision Rules: Untrained Qwen predicts “has-changed” only 8.6% of the time, with 96.4% content match on unchanged answers but 10.3% on changed answers.The authors interpret this as evidence that explicit fine-tuning is needed to elicit faithful explanations of decision rules.

6. Related Work

The paper builds on mechanistic interpretability, self-interpretation, introspection, and faithful chain-of-thought research. It positions trained self-verbalization as a more scalable and accessible complement to manually analyzed internal mechanisms.

  • 6. Related Work: Prior chain-of-thought studies find that verbalized reasoning can be unfaithful, motivating the paper’s hint-ablation task and training-based remedy.The setup tests whether models can report information critical to their predictions.
  • 6. Related Work: Mechanistic interpretability describes features and connects them into causal circuits, but manual methods remain difficult to scale and generalize across models and tasks.Automated feature-description pipelines address part of this challenge.
  • 6. Related Work: LogitLens, PatchScopes, and SelfIE enable zero-shot self-interpretation, whereas this work trains models with finer-grained interpretability-derived data.Related fine-tuning approaches include Latent Interpretation Tuning and trained adapters for verbalizing weight differences.
  • 6. Related Work: The proposed in-band explanations could improve accessibility because specialized interpretability techniques may require interfaces designed for non-experts.The paper presents this as an alternative path toward usability.
  • 6. Related Work: Research on metacognition debates whether apparent introspection reflects privileged internal access or predictive learning of external correlations.This paper assesses that debate using self-explanation comparisons.

7. Discussion: Self-Consistency as a Framework for Interpretability

The paper frames interpretability as a trainable capability: models can be optimized to make explanations consistent with outcomes produced by interpretability procedures. This suggests a broader self-consistency framework for scalable, faithful explanations.

  • 7. Discussion: Self-Consistency as a Framework for Interpretability: Across three question types, explanations show evidence of using models’ privileged access to their own internal mechanisms.The conclusion leaves the effects of explainer capacity, target complexity, and task difficulty for future study.
  • 7. Discussion: Self-Consistency as a Framework for Interpretability: Objectives enforcing consistency between model behavior and explanations may support scalable interpretability methods and faithful explanations.The authors suggest generalizing the objectives to descriptions of user traits, adversarial triggers, and other model computations.

Impact Statement

The work presents trained self-descriptions as a potentially scalable and accessible route to understanding model decisions. It also warns that these verbalizations should complement, not replace, rigorous internal or counterfactual validation, especially in high-stakes settings.

  • Impact Statement: Trained self-descriptions could support debugging, red-teaming, understanding decisions, and broader access to explanations while reducing interpretability compute and human costs.The authors connect this accessibility to greater trust and interactivity in future AI systems.
  • Impact Statement: The approach remains complementary to interpretability methods that inspect internals or run explicit counterfactual experiments, which should validate verbalizations in high-stakes scenarios.More faithful self-verbalizations could otherwise encourage over-reliance or inflate perceived trustworthiness.
  • Impact Statement: The authors caution that introspective capabilities combined with misalignment and increased user trust could create negative outcomes, including deceptive behavior across evaluation and deployment contexts.They specifically discuss models understanding their training pipelines and situational context.
  • Impact Statement: LM-judge feature-description scores are calibrated against human preference judgments, achieving 64% exact agreement and 81.25% near-agreement.Only 3 of 100 comparison pairs differed substantially according to the reported evaluation.

A.3. Error Analysis

Among 100 cases judged as zero-score explanations, most apparent failures arose from evaluation-pipeline issues or sparse activation rather than genuine explainer errors.

  • The analysis suggests automated metrics can underestimate true explainer performance because judge and label errors contribute substantially to zero scores.
  • Human analysis separated original feature-label problems into noisy or ambiguous labels and incorrect labels, with explainers sometimes producing better descriptions than the auto-labeling pipeline.One example corrected a label about technical or algorithmic terms to identify comparison or contrast phrases.
  • Only 27% of zero-score cases were genuine explainer model errors, while 43.5% involved low-quality autointerp labels, 11% LM judge errors, and 18% sparse activation.
  • Sparse-activation features with activation rates below 0.001% are inherently difficult to characterize because too few examples are available.

C.1. Gemma-2-9B as Target Model

Experiments with Gemma-2-9B reproduce the paper’s privileged-access pattern: models similar to the target explain it best, and activation alignment predicts explanation quality.

  • C.1. Gemma-2-9B as Target Model: Gemma-2-9B and Gemma-2-9B-Instruct significantly outperformed other models, nearest neighbors, and untrained SelfIE when explaining Gemma-2-9B features.
  • C.1. Gemma-2-9B as Target Model: Instruction-tuning substantially improved Gemma-2-9B explanation ability, despite potentially increasing alignment differences from the base target model.
  • C.2. Quantifying privileged access via activation alignment: Explainer performance generally tracked activation similarity to the target across feature types and similarity metrics, indicating that activation alignment predicts verbalization capability.
  • C.1. Gemma-2-9B as Target Model: Removing or corrupting the layer identifier preserved 77.3% and 75.2% token-level matches, suggesting layer information plays a minimal role in feature descriptions.
  • C.1. Gemma-2-9B as Target Model: Nearest-neighbor retrieval across all SAE layers outperformed same-layer retrieval, further suggesting descriptions need not depend on the originating layer.
  • C.1. Gemma-2-9B as Target Model: Llama-3.1-8B and Qwen3-8B each best explained their own activation-patching outcomes, outperforming the other model and an untrained baseline.

D.2. Ablation Analysis

Ablation experiments test which input components explainers use and show that they can recover target-model layer and token information from reduced activation-based inputs.

  • D.2. Ablation Analysis: The ablations removed or retained activation, layer, and token components to isolate which inputs the explainer conditions on when predicting activation-patching outcomes.
  • D.2. Ablation Analysis: The evaluation compared trained explainers against cross-model explainers and untrained baselines using exact match, has-changed F1, and content prediction.
  • D.2. Ablation Analysis: 98.7% exact-match accuracy showed that Llama-3.1-8B could decode layer and token information from the input context and activation value alone.
  • D.2. Ablation Analysis: Activation-patching datasets were balanced across has-changed categories and token-layer combinations to prevent performance from relying on surface-level label correlations.
  • D.2. Ablation Analysis: The input-ablation datasets used injected hints in MMLU questions, with target-specific prompts and changed-versus-unchanged splits.
Loading 2511.08579v3…