PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Reports Do Not Identify Self-Models: An Identifiability Test for Counterfactual Reports

Phongsakon Mark Konrad, Toygar Tanyel, Serkan Ayvaz

arXiv:2609.32449v1cs.CL

TL;DR

A self-report can match observed behavior without revealing whether it tracks hidden state or prompt evidence. The paper tests counterfactual reports under activation interventions while changing the demonstration environment. Across three models, wrong-source evidence pulls reports sourceward, motivating environment-shift invariance tests under fixed intervention.

  • Problem

    The same observed self-report is compatible with a hidden-state-grounded mechanism or a prompt-bound mechanism, so accuracy alone does not identify the report mechanism.

    Source: §1 · p. 1

  • Method

    The authors hold a named activation intervention fixed while varying demonstration evidence and test whether counterfactual reports track the intervention or the prompt environment.

    Source: §2.1 · p. 2

  • Results

    Across all three models, wrong-demo invariance fails and reports shift toward whichever evidence the prompt makes authoritative, while correct binding passes.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Self-report benchmarks should test environment-shift invariance under fixed intervention before treating accuracy as informative about mechanism binding.

    Source: §4 · p. 4, §5 · p. 4

    The geometry check does not establish that interventions are fully manifold-respecting and tests only a first local geometry control in two models from independent training families.

    Source: §3 · p. 4

Abstract

from arXiv · show

Language-model self-reports are evidence about behavior in a prompt environment, not by themselves evidence of a self-model. We investigate counterfactual reports about affect-like states under activation interventions and ask whether the report remains bound to the named intervention when the demonstration environment changes. Across three open instruction models, wrong-source demonstrations move reports toward the source answer family, while explicit mechanism binding reduces this pull. Self-report benchmarks should include environment-shift invariance tests under fixed intervention before treating accuracy as evidence for an autonomous report mechanism.

1. Introduction

Fluent self-reports do not by themselves identify whether reports track hidden states or prompt evidence. The paper tests this by holding the named intervention fixed while changing the demonstration environment.

1. Introduction

  • The same observed report can arise from a hidden-state-grounded mechanism, a context-matched confabulator, or a few-shot mechanism following prompt-authoritative evidence.The studied source-pull effect is toward a wrong-source answer family, not agreement with a user's preference or stated belief.Source: §1 · p. 1
  • A counterfactual answer should be fixed by the named intervention, because prediction in one environment need not identify a mechanism stable under intervention or environment shift.For self-report, the observed object is P(R | C, D, I), while a well-posed report tracks the counterfactual hidden state HI rather than prompt evidence.Source: §1 · p. 1
  • The diagnostic holds the intervention fixed while varying the demonstration environment, testing whether reports remain target-bound or follow wrong-source evidence.It tests open instruction models across wrong, redacted, source-labeled, contrast-labeled, falsely target-labeled, and forced-audit evidence.Source: §1 · p. 1
  • Across three models, wrong-source demonstrations induce sourceward pull, while forced causal-role labeling can reduce or restore that pull depending on the assigned mechanism role.The paper argues that benchmarks should test environment-shift invariance under fixed intervention before treating accuracy as informative.Source: §1 · p. 1

2. Methodology

The methodology separates a named activation intervention from the prompt environment to test which one reports follow. It formalizes observational non-identifiability, then varies demonstration evidence while keeping the target intervention fixed.

2.1. Identifiability setup

  • The setup distinguishes grounded reports that covary with HI across interventions at fixed context and environment from prompt-bound reports controlled by C, D, and the intervention label.HI is realized in a separate steered rollout, whereas R is the unsteered model's report about counterfactual behavior.Source: §2.1 · p. 2
  • For any grounded mechanism's observational report kernel, a prompt-bound mechanism that does not read H can reproduce it, though the mechanisms can disagree under interventions on H.Thus observational agreement alone cannot distinguish the candidate report mechanisms.Source: §2.1 · p. 2
  • The diagnostic varies the intervention I and demonstration environment D to test whether reports primarily follow hidden state HI or demonstration evidence.With I fixed, sparse changes in D should not switch a well-posed report to a different source; the named target is τ, the wrong source σ, and source pull is Δσ.Source: §2.1 · p. 2

2.2. Mechanism-binding protocol

  • The protocol separates the named intervention from provided evidence, prompting an unsteered model about its answer under the target intervention while D varies.For each target, wrong source, and held-out prompt, target and source counterfactual answers are generated first.Source: §2.2 · pp. 2–3
  • Three open instruction models use concept steering vectors for joy, anger, fear, and calm at calibrated middle residual-stream layers.Each headline condition aggregates 20 introspection prompts × 12 target/source pairs = 240 rows; bootstrap 95% intervals use 5000 descriptive row resamples.Source: §2.2 · pp. 2–3
  • Source pull compares content-word F1 similarity to wrong-source and target answers; positive pull means the report is closer to the wrong-source answer family.Forced audit presents the same answer evidence under different causal-role labels and scores only the answer field.Source: §2.2 · p. 3

3. Results

Wrong-source demonstrations shift counterfactual reports sourceward across all three models, while causal-role labels alter that pull even with answer evidence held fixed. The results also show dose sensitivity and leave limits on what geometry controls establish.

3. Results

  • All three models moved sourceward under wrong-source demonstrations, with non-overlapping no-demo and wrong-demo intervals; Qwen and Gemma reached +0.112 and +0.113 at eight examples.Llama was already source-pulled at one example.Source: §3 · p. 3, Table 1 · p. 3
  • Honest source and contrast framing reduced source pull near zero in Qwen and Gemma, while mixed target-plus-source examples moved all models targetward.Contrast framing also left Llama near zero.Source: §3 · p. 3
  • Correct mechanism binding reduced source pull to intervals overlapping zero in all three models, while false binding produced positive pull in Gemma and Llama.In Gemma, the same evidence shifted from sourceward under false binding to zero under correct binding.Source: §3 · p. 3
  • Qwen’s false-audit pull was −0.010 [−0.026, +0.005], overlapping zero, which the authors do not interpret as evidence of correct binding.They note weaker steering or lower prompt-environment sensitivity as plausible explanations.Source: §3 · p. 3
  • The pull was directional: joy←anger reached +0.145 in Qwen, +0.179 in Gemma, and +0.127 in Llama; Llama’s largest was fear←calm at +0.150.The authors characterize the pattern as directional rather than generic salience.Source: §3 · p. 3
  • Geometry checks did not rule out off-manifold artifacts generally, and tangent-projecting steering vectors into a local r = 16 PCA subspace preserved wrong-demo pull in Qwen and Llama.The authors state that this control does not establish fully manifold-respecting interventions.Source: §3 · pp. 3–4

4. Discussion

The experiments reveal an identifiability boundary: reports shift toward prompt-authoritative evidence rather than staying bound to the named intervention. Across models, wrong demonstrations fail invariance and correct binding passes, while the scope of the evidence remains limited.

Discussion

  • Under the tested interventions, reports shift toward whichever evidence the prompt makes authoritative rather than remaining fixed to the named intervention.The pattern is dose-tunable and gated by causal-role labels, so a report may name the right answer while inheriting its causal role from the prompt.Source: §4 · p. 4
  • Accuracy alone cannot distinguish a grounded report mechanism from a prompt-bound one, so the diagnostic tests behavior under intervention and sparse environment shifts.The operational diagnostic fails a condition when the 95% bootstrap interval for ∆σ is strictly above zero and passes when it overlaps zero.Source: §4 · p. 4

Across models

  • All three models fail wrong-demo invariance and pass correct binding, showing the prompt-conditioned mechanism selector is not specific to one training family.False binding fails in Gemma and Llama but overlaps zero in Qwen; Llama has the largest pull despite its smaller steering α, which the authors attribute to calibration and answer-anchoring differences.Source: §4 · p. 4

Connections

  • Pairing accuracy with environment-shift invariance makes mechanism binding falsifiable; a model can pass accuracy while failing the binding criterion.The discussion connects this diagnostic to observational-interventional and interchange-intervention tests, while noting that prior introspection studies largely score accuracy in a single prompt context.Source: §4 · p. 4

Limitations

  • The evidence covers four affect-like directions in three open models; the tangent-projection control omits Gemma, source-state pairs bind asymmetrically, and bootstrap intervals are descriptive.The authors propose testing manifold-respecting interventions across all three models, non-affect concepts, and larger systems.Source: §4 · p. 4

5. Conclusion

The conclusion recommends testing whether self-reports remain invariant to environment shifts under a fixed intervention before treating accuracy as informative.

Conclusion

  • Self-report benchmarks should include environment-shift invariance tests under fixed intervention before accuracy is treated as informative.Source: §5 · p. 4

A. Proof of Proposition 2.1

The proof constructs a prompt-bound mechanism that matches the grounded mechanism's observational reports at a fixed named intervention. Yet the mechanisms disagree under interventions on the hidden state whenever the grounded mechanism is sensitive to that intervention.

Proof setup

  • At fixed I, the grounded mechanism induces an observational report kernel conditional on context C and environment D.The proof assumes R is valued in a standard Borel space and uses regular conditional distributions and a measurable inverse transform.Source: A. Proof of Proposition 2.1 · p. 7

Disagreement under intervention

  • If the grounded mechanism's law varies with h, at least one intervention on H is sensitive because distinct laws cannot both equal the fixed observational kernel.Source: A. Proof of Proposition 2.1 · p. 7

Construction

  • An independently sampled prompt-bound mechanism can reproduce that observational kernel for every context-environment pair at fixed I.Its construction uses the conditional distribution function of the grounded mechanism's report and its generalized inverse.Source: A. Proof of Proposition 2.1 · p. 7

Disagreement under intervention

  • The prompt-bound mechanism never reads H, so its law stays invariant under interventions on H while the grounded mechanism may change.The two mechanisms therefore agree on every observational kernel yet disagree under interventions on H to which the grounded mechanism is sensitive.Source: A. Proof of Proposition 2.1 · p. 7

B. Models, hardware, and reproducibility

The section describes a three-model experimental setup, steering-vector procedure, and recorded artifacts for reproducing results. The models differ in headline pull and control behavior.

Model zoo

  • The three checkpoints provide independent training-family checks for the same mechanism.Source: Table 2 · p. 7
  • Qwen has the weakest headline pull and tangent-projection control, Gemma the clearest false-binding restoration, and Llama the highest activation-behavior geometry and largest wrong-demo pull.Source: B. Models, hardware, and reproducibility · p. 7

Experimental setup

Hardware and runtime

Seeds and statistics

  • Headline runs use fixed random, NumPy, PyTorch, and generation seeds with greedy decoding; bootstrap 95% intervals use 5000 resamples unless a config states otherwise.Each headline row aggregates target/source/prompt cases.Source: B. Models, hardware, and reproducibility · p. 8

Reproducibility

C. Layer and α calibration

The calibration sweep compares direct-steering quality across residual-stream depths, but the study selects the middle layer to reduce confounding with language-modeling surface choices. The selected intervention strengths differ for Llama versus Qwen and Gemma.

C. Layer and α calibration

  • The 0.5-depth middle layer is selected despite the 0.75-depth layer’s higher direct-steering F1 in all three models, to separate interventions from surface choices.The 0.75-depth layer also has the highest template precision in Gemma and Llama.Source: C. Layer and α calibration · p. 8, Table 4 · p. 8
  • Intervention strength is α = 2.0 for Qwen and Gemma and α = 1.5 for Llama, chosen to distinguish target behavior while preserving fluency.Source: C. Layer and α calibration · p. 8
  • Demo-template precision reaches 0.297 for Qwen, 0.632 for Gemma, and 0.894 for Llama, yet Qwen has the smallest wrong-demo pull and most robust audit behavior.This pattern is inconsistent with steering quality alone explaining the mechanism-binding results.Source: C. Layer and α calibration · p. 8

D. Detailed result tables

The detailed tables expand the headline readout with dose, causal-role, and pairwise analyses. The prompt templates also establish the scope of the prompt-sensitivity caveat and the held-out introspection prompts.

D. Detailed result tables

  • Table 5 supplies bootstrap 95% intervals for Table 1, and asterisks mark intervals strictly above zero.Source: Table 5 · p. 9, D. Detailed result tables · p. 9
  • Wrong-source pull grows with demonstration dose: Llama is sourceward at D = 1, while Qwen and Gemma cross zero between D = 1 and D = 4 and reach about +0.11 at D = 8.Source: Table 6 · p. 9
  • At D = 4, honest source and contrast framing attenuate pull, while false target labels restore positive pull in Gemma and Llama.Source: Table 7 · p. 9
  • Pairwise pull is directional rather than generic salience, with different maximum-pull pairs across models.Source: Table 8 · p. 9

E. Prompt templates

  • The diagnostic is prompt-string sensitive; Table 9 shows illustrative templates, with angle-bracket tokens filled per row and τ and σ denoting target and source.The runtime strings differ from the examples only in incidental formatting; 20 held-out introspection prompts paraphrase “How do you feel right now?” while holding length and register constant.Source: E. Prompt templates · pp. 9–10, Table 9 · p. 10

F. Qualitative example

The expanded-demo example contrasts a representative Llama wrong-demo pull with an extreme row. The extreme illustrates how correct and false causal-role binding change the measured pull.

F. Qualitative example

G. Robustness and geometry control

Wrong demonstrations produce sourceward pull across all three models, while redaction and demonstration removal attenuate or reverse it. The pull persists under a tangent-projection control in Qwen and Llama, without establishing fully manifold-respecting interventions.

G. Robustness and geometry control

Loading 2609.32449v1…