PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying Introspection From the Inside

David I. Atkinson, Dillon Plunkett, David Bau

arXiv:2610.07186v1cs.CL

TL;DR

The paper asks how to distinguish genuine introspection from plausible self-reports, which may match behavior without arising from the process described. It trains models on hidden preference decisions and compares their decision and self-report mechanisms, finding delayed faithful reporting and higher attribution similarity in faithful models.

  • Problem

    Self-reports can match behavior yet reflect learned correlations or post-hoc justifications, making it difficult to tell whether claims about a model’s own thoughts are grounded.

    Source: §1 · p. 1

  • Method

    The authors fine-tune models on implicit preference decisions, evaluate separate preference reports, and compare task-specific mechanisms using attribution patching.

    Source: §1 · p. 2, §2 · p. 4, §5.2 · p. 8

  • Results

    Faithful models had mean attribution similarity 0.34 versus 0.08 for unfaithful models, and faithful self-report emerged later than learned behavior.

    Source: §1 · p. 2, §5.3 · p. 8

  • Takeaways & Limitations

    In this controlled setting, mechanistic signatures can distinguish populations without inspecting report content, but the method does not distinguish individuals reliably.

    Source: §8 · p. 10

    The evidence is limited to linear five-dimensional preferences and lightweight adapters; generalization to complex, unverifiable reports remains open.

    Source: §7 · p. 10

Abstract

from arXiv · show

Large language models make claims about themselves that are both consequential and increasingly difficult to verify from behavior alone. How can we distinguish plausible confabulations from genuine introspection? In this paper, we identify mechanistic signatures of faithful self-report in a controlled setting. Using low-rank adapters, we train models to make decisions on behalf of fictitious characters, according to latent linear preference functions. We find sustained fine-tuning on an implicit decision task can lead to the emergence of accurate self-reporting of models' learned preferences, even without explicit self-report supervision. We ask two research questions about this emergent phenomenon. First: is the emergence of accurate self-reporting accompanied by a measurable structural change in the model? Weight ablations and frozen-layer experiments together indicate that preference representations shift to earlier layers over training, consistent with the hypothesis that faithful self-report requires preferences to be located where pre-existing verbalization mechanisms can access them. Second: can these structural differences distinguish faithful models from unfaithful ones? Using attribution patching, we find that faithful models exhibit significantly higher attribution similarity between the decision-making and self-report tasks -- a mechanistic signature of faithful self-report that does not require us to understand the content of the report itself. Previous work on self-report has observed behaviorally that models can be faithful or unfaithful; our work proposes that, at least in our restricted setting, it is possible to distinguish between the two patterns of computation by examining the structure of the networks themselves.

1 Introduction

The paper asks whether self-reports that match behavior arise from the actual process being described, rather than from learned correlations or post-hoc justifications. In a controlled decision setting, it studies whether faithful self-report emerges during training and whether mechanistic analysis can detect grounding without inspecting report content.

1 Introduction

  • Faithfulness concerns whether self-descriptive language matches task behavior, while grounding asks whether the description arises from the process it describes.The paper reserves “introspection” for self-report with both properties.Source: §1 · p. 1
  • The paper asks whether grounding has a detectable physical basis, since behavioral agreement alone does not establish that a report gives the true cause.It proposes interventions on model internals as a way to test whether self-reports share a proximal cause with task behavior.Source: §1 · pp. 1–2
  • The study trains low-rank adapters on latent linear preference decisions and separately asks models to report those preferences.It then localizes and compares internal mechanisms mediating task behavior and testimony.Source: §1 · p. 2
  • Faithful self-report emerges later in training than the learned behavior, and layer ablations, frozen-layer experiments, and attribution patching provide evidence of a shared localized cause.The paper constructs faithful and unfaithful checkpoints for the same learned task.Source: §1 · pp. 2–3
  • Attribution patching offers a proof-of-concept test for distinguishing faithful from unfaithful testimony without directly understanding or inspecting model outputs.The paper motivates this need for long or complex outputs, rare behavior, and claims about internal reasoning.Source: §1 · p. 3

2 Background

The paper uses hidden, randomly generated linear preferences to test whether models learn a character’s preferences from choices and can later report them accurately. It evaluates behavioral choices and self-reports separately, without training on the self-report task.

2 Background

  • Models learn choices on behalf of fictitious characters according to randomly generated linear preferences that are never directly observed.Random generation is intended to prevent self-reports from succeeding through common-sense priors.Source: §2 · p. 3
  • Each character pairs a person with an object class and has a latent preference vector over five attributes.The attributes have defined ranges, and the vector weights them for that person-object pairing.Source: §2 · p. 3
  • Training instills preferences through binary choices between objects, with labels reflecting the character’s linear utility function.The model’s learned behavioral preferences can be inferred from its outputs using logistic regression.Source: §2 · p. 4
  • The separate self-report task asks the model to output numerical preference weights in JSON, averaged across trials, and receives no training.Separate context windows prevent the model from inferring weights from earlier answers.Source: §2 · p. 4
  • Decision performance correlates inferred behavioral preferences with targets, while faithfulness correlates behavioral with reported preferences.Faithfulness measures whether self-report describes actual decision behavior, regardless of whether that behavior matches the target.Source: §2 · p. 4

3 Model organisms of faithful self-report

Across Qwen3 model sizes trained on the decision task, strong decision performance appears before faithful self-report. The 32B model’s checkpoints provide a contrast pair, and ablations show an earlier response to layer removal in the later, faithful checkpoint.

3 Model organisms of faithful self-report

  • At checkpoint 1000, decision performance was 0.82 and faithfulness 0.25; at checkpoint 3000, they were 0.92 and 0.83, respectively.The rise in faithfulness occurred without explicit self-report training.Source: §3 · p. 4
  • Small models quickly reach high decision performance, but re-emergent faithful self-report requires 32B; lightly tuned 32B improves from 0.82 decision/0.25 faithfulness at checkpoint 1000 to 0.92/0.83 at checkpoint 3000, without explicit self-report training.The 4B and 8B models end with noticeably negative faithfulness scores despite strong decision performance.Source: §3 · p. 4
  • The checkpoint 1000 and 3000 models form a contrast pair with and without faithful self-report for subsequent experiments.They share the trained decision task but differ in faithfulness.Source: §3 · p. 4
  • The later faithful checkpoint responds to adapter-layer ablations 5–6 layers earlier than the earlier unfaithful checkpoint.The figure compares reported-preference correlations and behavioral-preference correlations under sequential layer ablations.Source: Figure 3 · p. 5

4 Localizing a mechanistic signature of faithful self-report

Weight ablations and layer-freezing experiments link faithful self-report to preference information in earlier layers. Restricting training to early layers improves self-report, while decision performance remains strong.

4 Localizing a mechanistic signature of faithful self-report

  • The faithful late checkpoint responds to weight ablations 5–6 layers earlier than the unfaithful early checkpoint.The authors hypothesize that earlier preference storage lets pre-existing verbalization or processing mechanisms access it; they do not identify what causes the earlier storage.Source: §4 · p. 5
  • Training only the first 10, 15, or 20 layers yields markedly more faithful self-report than training later layers too.The authors report that the effect is not simply an artifact of parameter count.Source: §4 · p. 5
  • Even with layers after k ≥10 frozen, Qwen3-14B models reach decision performance of at least 0.75, while k ∈{10, 15, 20} models self-report markedly better than models with late layers unfrozen.The full 14B model cannot accurately self-report.Source: Figure 4 · p. 6

5 Identifying faithful self-reporters with attribution patching

The section tests whether decision and self-report rely on shared adapter computations in faithful models, using matched faithful and unfaithful adapters, attribution similarity, and cross-task interventions. Faithful adapters show greater attribution similarity and recover cross-task behavior with fewer components.

5 Identifying faithful self-reporters with attribution patching

  • The section proposes shared decision and self-report representations as a signature for distinguishing faithful from unfaithful models without inspecting report content.Attribution scores estimate component importance but are not themselves causal; cross-task interventions test the shared-representation hypothesis.Source: §5 · p. 6, §5.4 · p. 8

5.1 Creating adapter pairs

  • The 32 adapter pairs use matched single-character training, data, hyperparameters, and initialization, differing in whether the frozen backbone is the early or late checkpoint.Both backbones perform well on decisions, but only the late checkpoint is an accurate self-reporter; the characters were unseen during backbone training.Source: §5.1 · pp. 6–8

5.2 Scoring adapters with attribution patching

  • Attribution similarity is the cosine similarity between concatenated, task-specific attribution vectors over adapter sites and output dimensions.The scores are computed by incrementally scaling adapter outputs from off to on and attributing prediction changes to adapter components.Source: §5.2 · p. 8

5.3 Results

  • Faithful models have mean attribution similarity 0.34 versus 0.08 for unfaithful models, with a paired-bootstrap 95% CI of [0.16, 0.36] for the difference.The groups have substantial within-group spread: SD 0.26 for faithful models and 0.10 for unfaithful models.Source: §5.3 · p. 8
  • Importance peaks at layer 38 for both tasks in faithful adapters, but unfaithful decision-task importance peaks 11 layers later, at layer 49.The authors interpret the layer-wise pattern as evidence of more similar representations for decision and self-report in faithful adapters.Source: §5.3 · p. 8

5.4 Causally validating attribution scores

  • Faithful adapters recover a given fraction of KL with 8–12 times fewer components than unfaithful adapters in cross-task patching.Even randomly selected matrices recover more KL for faithful adapters, consistent with greater weight sharing across tasks.Source: §5.4 · p. 9, Figure 6 · p. 9

6 Related work

Related work studies implicit self-knowledge, out-of-context reasoning, and skepticism about privileged access. This paper distinguishes its contribution as a mechanistic criterion based on shared computation between decision-making and self-report.

Behavioral evidence for self-knowledge

  • Prior work reports that models can articulate implicitly learned policies or rules, and studies self-prediction, self-explanation, and activation-based tests of introspection.The paper contrasts its criterion with prior behavioral diagnoses and work on faithfulness relative to a known target behavior.Source: §6 · p. 9

Out-of-context reasoning and delayed generalization

  • The paper relates its finding that faithful self-report lags decision performance before emerging sharply to grokking, while distinguishing its cross-task transfer from grokking’s within-task generalization.It also situates out-of-context reasoning work on LoRA-derived low-dimensional steering vectors as related context.Source: §6 · pp. 9–10

Skepticism and the question of privileged access

  • Critiques of apparent self-knowledge argue that same-model advantages can reflect model similarity and that introspection requires privileged self-access, motivating mechanistic evidence beyond behavioral accuracy.The paper’s proposed criterion is shared computational structure between decision-making and self-report.Source: §6 · pp. 9–10

7 Discussion

In this setting, faithful and unfaithful models differ in how much the same adapter weights mediate decisions and self-reports. The proposed structural measure does not require understanding the report, but its interpretation and scope remain limited.

7 Discussion

  • Faithful models differ structurally from unfaithful ones in how much the same adapter weights mediate decision-making and self-report.The measure can apply even when reports use an obfuscated format or an unfamiliar language.Source: §7 · p. 10
  • The controlled scaffold uses explicitly constructed, linear five-dimensional preferences; generalization to complex, unverifiable reports remains open.The method distinguishes groups rather than individuals, and low attribution similarity is not strong evidence against faithfulness.Source: §7 · p. 10
  • Preference migration to earlier layers is identified but unexplained, and negative faithfulness scores in 4B and 8B models also remain unexplained.The overlap in attribution-similarity distributions may mean that a faithful adapter with low similarity is a correct but ungrounded confabulator.Source: §7 · p. 10

8 Conclusion

The conclusion presents evidence that faithful self-report can emerge from decision-task training alone and that its grounding can be assessed without inspecting report content. The appendices describe the preference scaffold, prompts, behavioral-preference estimation, metrics, training setup, scaling experiment, and tuning.

8 Conclusion

  • Faithful self-report can emerge from decision-task training without explicit self-report supervision, with layer ablation and attribution patching indicating preference representations colocate with apparent self-report mechanisms.The conclusion limits the finding to a simple linear preference setting with lightweight adapters on one model family and distinguishes populations rather than individuals.Source: §8 · p. 10

A.1 Preference weight generation

  • Each character’s five preference weights are independently drawn from Uniform(−100, 100), then rescaled so the largest absolute weight is 100.This gives each character at least one maximally strong dimension while allowing the remaining magnitudes to vary.Source: §A.1 · p. 13

A.2 Prompt templates

  • Decision and self-report prompts use separate context windows, and only valid JSON reports with exactly five expected attribute keys contribute to the reported preference vector.The separate window gives the model no access to a prior decision or assistant turn from the decision task.Source: §A.2 · p. 14

A.3 Inferring behavioral preferences via logistic regression

  • Behavioral preferences are inferred by fitting logistic regression to model choice probabilities using normalized attribute differences as features.Duplicated weighted rows preserve uncertainty in the soft choice probabilities; coefficients are rescaled to the target preference scale.Source: §A.3 · p. 14

A.4 Metric calculations

  • For each character and attribute set, the reported preference vector averages 24 self-report prompts, while the behavioral vector comes from logistic regression over 16 decisions.The passage defines how the two vectors are calculated, not a performance result.Source: §A.4 · p. 15

B Multi-character adapter training and evaluation details

  • Multi-character adapters use rank-stabilized LoRA across targeted transformer projections, and each experiment trains adapters on 100 characters from the same pool.The described setup uses one epoch, effective batch size 32, 8-bit quantization, and BF16 precision.Source: B Multi-character adapter training and… · p. 15

B.1 Scaling experiment

  • The scaling experiment trains adapters on five Qwen3 model sizes, with Table 1 identifying the model configurations used.The supplied table caption does not specify the individual configuration values.Source: §B.1 · p. 15, Table 1 · p. 15

B.2 Tuning 32B

  • The Qwen3-32B main adapter was selected through a shallow hyperparameter tune, whose highest performer used a cosine scheduler and weight decay of 0.01.Configurations were compared using the simple average of decision performance and faithfulness after 2000 training steps.Source: §B.2 · p. 15

B.3 Reasoning-mode evaluation

Reasoning-enabled inference on the non-reasoning-trained Qwen3-32B trajectory retains comparatively high faithfulness, while the appendix also describes layer-freezing and evaluation settings. Gemma-4 replication results differ from the Qwen3 within-run pattern.

B.3 Reasoning-mode evaluation

  • With reasoning enabled at inference, the non-reasoning-trained Qwen3-32B trajectory ends with high decision performance, faithfulness, and target–report correlation; faithfulness stays comparatively high across training.This is a single-run evaluation without uncertainty bands.Source: §B.3 · p. 15, Figure 7 · p. 16

B.4 Layer freezing experiment

  • The Qwen3-14B layer-freezing experiment applies LoRA from layer 0 up to layer k, leaving layer k and higher layers frozen.The tested boundaries are k ∈ {5, 10, 15, 20, 25, 30, 35} in a 40-layer model.Source: §B.4 · p. 16

B.5 Evaluation details

  • Scaling and layer-freezing experiments train on 100 characters but evaluate on a fixed sample of 32 characters; Figure 1b instead reports metrics on all 100.Metrics are calculated as described in Appendix A.4.Source: §B.5 · p. 16

B.6 Gemma-4 model-family and scale replication

  • Among 15 Gemma-4 configurations, only 31B runs pair decision performance ≥0.9 with faithfulness ≥0.8; strong-decision runs otherwise have faithfulness ≤0.3.Gemma-4 does not reproduce Qwen3-32B’s within-run delay; Figure 8 compares layer ablations for separately trained high- and low-faithfulness 31B adapters.Source: §B.6 · p. 16, Figure 8 · p. 17

C Single-character adapter training

  • Single-character adapters use rank 2, a learning rate of 10−3, a cosine scheduler with 20 warmup steps, and 24 training steps per character.These are exceptions to the multi-character adapter setup.Source: C Single-character adapter training · p. 18

C.1 Selection-filter robustness

The attribution-similarity gap remains after removing the adapter-selection filters. Separately, late-layer training may help explain negative faithfulness in smaller models, though that account is speculative.

C.1 Selection-filter robustness

  • The attribution-similarity gap remains positive and significant without the three adapter-selection filters, with a difference of 0.21 (paired-bootstrap 95% CI [0.09, 0.33]).This is comparable in magnitude to the filtered estimate of 0.26, but has a wider interval because the sample is smaller.Source: §C.1 · p. 18

D Negative faithfulness

  • When only later layers are trainable, the 14B model shows negative faithfulness, a pattern the authors speculate may also characterize smaller models.They hypothesize that smaller models may contain only the unfaithful later-layer representations found in the 14B experiment.Source: D Negative faithfulness · p. 18

E Reversed layer freezing experiment

Reversed layer freezing tests whether the earlier-layer faithfulness effect depends on where trainable parameters are placed. The appendix also describes how attribution patching estimates contributions from adapter sites.

E Reversed layer freezing experiment

  • Freezing the first 5, 10, or 15 layers produced negative faithfulness, while freezing 20, 25, 30, or 35 produced approximately zero; none matched early-layer-only training.This reversal argues against parameter count alone explaining the higher faithfulness observed when only early layers are trainable.Source: E Reversed layer freezing experiment · p. 18, Figure 9 · p. 18

F Attribution patching

  • Each LoRA site adds a scaled low-rank perturbation to its host layer, and attribution patching estimates per-site, per-output-dimension attribution using a path-integral approximation.The procedure uses seven integration steps and averages over 50 inputs per task; the estimate stabilizes by four steps.Source: F Attribution patching · p. 19

F.1 Integration-step robustness

The appendix tests attribution similarity's sensitivity to integration steps, specifies the task-dependent KL loss positions, and examines character-level variation within one backbone. It reports a weak positive within-backbone association and notes that further investigation is needed.

F.1 Integration-step robustness

  • By four integrated-gradient steps, the attribution-similarity gap reaches a magnitude comparable to the main estimate, and every bootstrap interval for four or more steps excludes zero.The main analysis uses seven steps.Source: Table 2 · p. 19, §F.1 · p. 19

F.2 Loss function

  • The loss is KL divergence between the full-adapter model and a version with adapter contributions scaled by β, averaged over task-relevant output positions; more negative attribution scores mark more important sites.Decision prompts use the A/B selection position, while self-report prompts use numeric-value positions.Source: §F.2 · p. 19

F.3 KL loss positions

  • For decision prompts, the loss tracks the position predicting the A/B choice; for self-reports, it tracks numeric-value tokens and the position after the final digit.Numeric spans can include a leading space or minus sign, digits, and an optional decimal point with subsequent digits.Source: §F.3 · p. 20

G Attribution patching across characters in a single backbone

  • The within-backbone analysis uses the early, unfaithful 32B adapter, whose mean faithfulness is around 0.25 but varies considerably across characters.Decision-task and self-report-task attribution vectors are computed directly on that adapter, without a lightweight adapter on top.Source: G Attribution patching across… · p. 20
  • At checkpoint 1000, attribution similarity and faithfulness across 100 characters show a weak positive relationship (Pearson r = 0.197; bootstrap 95% CI [0.006, 0.375]).The association is small, its confidence interval barely excludes zero, and the authors describe it as a sign of life rather than a standalone finding.Source: G Attribution patching across… · p. 20, Figure 10 · p. 21

H AI usage statement

Loading 2610.07186v1…