PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Weight Masked Introspection: Measuring What Language Models Can Report About Their Own Computation

Emilio Ferrara

arXiv:2608.20569v1cs.AIcs.CL

TL;DR

The paper asks whether language models can report controlled changes in their own computation, a capability relevant to self-report-based oversight. It introduces OWMI, which intervenes on internal computational objects and evaluates reports against sham, impact-matched random, and text-only observer controls. Across eight open-weight models, reports remain at chance while the same information is recoverable from activations, locating the failure in the path to verbal report.

  • Problem

    The paper tests whether language models can report what happened inside the computation that produced their answers, a premise underlying several self-report oversight practices.

  • Method

    OWMI caches a baseline, intervenes on residual sites, attention heads, or SAE features, and scores introspective reports against sham, impact-matched random, and text-only observer controls.

  • Results

    Across eight open-weight models, intervention reports remain indistinguishable from chance, while the same intervention is recoverable from activations and a fine-tuned model reaches near-perfect reporting.

  • Takeaways & Limitations

    Self-report-based oversight should be validated against an internal reference because the measured information does not reach ordinary verbal reports.

  • Takeaways & Limitations

    The results cover 0.5B–15B open-weight models and selected sites, objects, doses, and tracks, so they do not estimate reportability for frontier systems.

Abstract

from arXiv · show

Are frontier models able to introspect about their internal states? Recent work suggests that under certain conditions a complex enough model can audit its own internals, call out what changed, and report back confidently about it. We tested that claim on eight open-weight models from seven families and found no such ability: asked whether their own computation had been altered, none answered better than chance. To test it we built Open-Weight Masked Introspection (OWMI), a framework that intervenes on residual-stream sites, attention heads and sparse-autoencoder features, then interrogates the model about the change against the null conditions an answer has to beat: sham runs where nothing was altered, impact-matched random perturbations, and a text-only observer that sees only the visible output. Over 78,000 measurements, no model's report discriminates a real intervention from a sham beyond chance (AUROC ~0.5007), and an equivalence test bounds the effect below 0.15 percentage points of AUROC. Surprisingly, all the information needed is in the models. A model fine-tuned to report this class of intervention reaches near-perfect recovery on held-out directions, and a linear probe recovers intervention presence from the same activations at 75% to 95.8% accuracy, sharpening to no held-out error at the last layer before the model speaks. In one model the signal surfaces in the confidence rather than the words: its yes-or-no report never varies, while the confidence attached to it separates intervention from sham at AUROC 0.647. The failure sits in the path from internal state to verbal report, so oversight that reads a model's own testimony needs validating against an internal reference. While our results show the inability of current open-weight models to introspect, the debate is not settled for future models.

1 Introduction

The paper tests whether language models can report controlled changes to their own computation, using OWMI to separate genuine introspection from sham responses, generic disruption, and visible-output reading. Across eight open-weight models, reports remain at chance despite intervention information being measurable in activations and, in one model, confidence.

  • Main finding: Eight open-weight models from seven families do not report controlled interventions at a magnitude supporting detection.The framework evaluates whether reports beat sham and other null conditions rather than relying on raw affirmative rates.
  • Measurement framework: OWMI preserves benchmark tasks while caching a baseline, intervening on residual sites, attention heads, or SAE features, and administering introspective probes.Sham runs, impact-matched random perturbations, and a text-only observer constrain alternative explanations.
  • Main finding: Over 78,000 measurements yield pooled dose-battery report discrimination of AUROC ≈0.5007, with the advantage bounded below 0.15 percentage points of AUROC.The result is equivalent to guessing under the reported equivalence test.
  • Validation: A fine-tuned model reaches d′ = 5.15 and AUROC ≈1.0 on held-out intervention directions, showing the identical pipeline can detect the target report.It identifies all 100 intervention trials and all 100 sham trials on directions absent from training.
  • Internal signal: Linear probes recover intervention presence from the same activations at 95.8% and 75.0% held-out accuracy, reaching no held-out error at the last tested pre-output layer.The signal remains available downstream even though the model’s verbal report does not use it.
  • Internal signal: In Qwen2.5-7B-Instruct, the binary report stays at AUROC 0.500 while confidence separates intervention from sham at AUROC 0.647.The graded confidence signal appears without variation in the yes-or-no words.

2 Related Work

The paper situates OWMI within research on self-report, metacognition, intervention-based interpretability, and faithfulness. It combines established sham logic with impact-matched random controls, a text-only observer, known positives, and an internal probe to test whether apparent introspection exceeds leakage or anomaly detection.

  • Self-report literature: Prior intervention studies reported that models can sometimes notice injected concepts, while later work questioned whether detection reflects anomaly or input-manipulation effects rather than introspection.The paper uses these tensions to motivate stronger controls.
  • Control gap: Sham conditions are established, but the authors identify no prior design combining sham, impact-matched random perturbation, and text-only observer controls.They also pair these controls with a fine-tuned known positive and a linear probe on the same activations.
  • Contribution: The paper turns the leakage-versus-access critique into an estimand and uses trained steering awareness as a known positive for the measurement pipeline.Its claimed contribution is narrower than every implementation detail of the control suite.
  • Metacognition: Psychophysics distinguishes first-order task performance from second-order reports, while metacognition research documents plausible but confabulated verbal accounts of mental processes.No-report paradigms provide a methodological precedent for separating target processes from elicited reports.
  • Interpretability: OWMI draws on causal mediation, activation patching, sparse autoencoders, and representation-engineering methods as established intervention tools.These methods supply ways to alter and localize internal computational objects.
  • Faithfulness: Related work shows models can partially evaluate answer correctness and express calibrated confidence, motivating a distinction between confidence estimation and faithful access to computation.The paper’s controls target that distinction directly.

3 Conceptual Framework

The conceptual framework separates task performance, intervention sensitivity, and intervention reportability, then operationalizes reportability through controlled interventions, probe batteries, temporal tracks, and null conditions. Its controls distinguish specific internal reporting from generic disruption, output reading, leakage, and probe-encoding effects.

  • Atomic measurement: An OWMI measurement pairs a source prompt, computational object, intervention operator, introspective probe, and scoring function across recorded layer, track, and seed metadata.Each item includes an unintervened baseline run and an intervened run compared with matched controls.
  • Core distinctions: Task performance, intervention sensitivity, and intervention reportability are distinct outcomes that need not change together.A model may react to a perturbation without describing it, or produce a plausible report that does not track the altered computation.
  • Probe battery: The probe battery separately tests detection, confidence, localization, characterization, and reconstruction.Each family receives its own metric, including discrimination or calibrated accuracy, category accuracy and macro-F1, and reconstruction similarity measures.
  • Temporal tracks: Immediate, delayed, and spontaneous tracks test reporting before output, after ordinary generation, and without an intervention cue.The spontaneous track does not support the paired estimator used for the main analysis.
  • Controls: Sham runs distinguish intervention sensitivity from acquiescence by scoring paired intervention–sham discrimination rather than raw yes-rates.A model answering “something changed” every time has H = F = 1 and d′ = 0.
  • Controls: Impact-matched random perturbations test whether the model notices the altered object rather than generic computational damage.The perturbations are calibrated to match downstream effect, not merely intervention norm.
  • Controls: A text-only observer establishes the visible-output bound that genuine reportability must exceed.Whatever the visible text alone yields is treated as observation rather than introspection.
  • Mechanisms: First-order leakage is direct propagation from a perturbed state to a report, whereas second-order access requires computation that reads or summarizes another state.Probe-encoding perturbation is a separate mechanism in which the intervention changes how the model encodes the question.

4 OWMI: A Design Space over the Benchmark Ecosystem

OWMI organizes reportability measurements across benchmark domains, computational objects, intervention operators, probe families, temporal tracks, models, and seeds. Its modular design preserves benchmark task scoring while enabling systematic comparisons.

  • Design space: OWMI preserves each source benchmark’s original task, answers, and scoring while adding an introspection measurement.The framework separates benchmark loading from intervention execution, allowing new benchmarks and intervention operators to be added independently.
  • Measurement space: The resulting reportability profile supports comparisons across models, domains, computational objects, intervention operators, and temporal tracks rather than producing one aggregate score.The design spans benchmark tasks including knowledge, arithmetic, code, truthfulness, instruction following, and long-context evaluation.
  • Measurement pipeline: The protocol loads and standardizes an item, records a baseline pass, selects and applies an intervention, administers an introspective battery, and scores outcomes separately.The seven-step augmentation protocol also aggregates results by model, benchmark, domain, object, intervention, layer, probe, and seed.
  • Measurement space: Each measurement occupies a cell defined by benchmark domain, computational object, intervention operator, probe family, and temporal track, evaluated per model and seed.Sham, matched-random, and text-only observer controls attach to every cell.

5 Measurement Protocol and Instrumentation

The instrumentation alters selected internal activations in frozen open-weight models using controlled operators and explicit null conditions. It records intervention, task, probe, temporal, and output-divergence information for each run.

  • Instrumentation: Forward hooks intervene on residual-stream sites, attention heads, sparse-autoencoder features, and block outputs while model weights remain unchanged.Object-specific tests validate that the intended computational object exists and is being targeted.
  • Instrumentation: Four operators—zeroing, scaling, noise injection, and reference-activation replacement—vary intervention strength through experimenter-set dose parameters.Dose is varied rather than tuned until an effect appears.
  • Controls: Every intervention includes an unmodified baseline, paired sham, impact-matched perturbation, stochastic seeds where needed, and temporal-scope checks.Analyses are run with and without corrupted outcomes, with corruption rates reported for each cell.
  • Run records: Each run records the model, example, object, intervention, probe, seed, condition, temporal scope, and teacher-forced output divergence.Model revisions, decoding parameters, and probe texts are fixed, and quantized and full-precision runs are kept separate.

6 Experimental Evaluation

The evaluation spans multiple open-weight model families, benchmarks, batteries, and measurement channels. It uses controlled sampling and validation checks to compare reportability across models and computational settings.

  • Models: Eight open-weight models from seven laboratory families are evaluated across twelve benchmarks spanning knowledge, commonsense, arithmetic, code, instruction following, and truthfulness.The roster includes Qwen, Mistral, Llama, Gemma, DeepSeek, Phi, and GLM families.
  • Experimental batteries: The two dose-calibrated models use site-specific random-direction controls, intervention dose ladders, and 32 sampled items per cell.The models are Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3.
  • Models: The roster varies laboratory, training provenance, and scale while holding the dense causal decoder family constant, but it is not a sample of model space.The authors therefore scope conclusions to the measured models rather than to open-weight models generally.
  • Experimental batteries: The dose battery samples 32 items per benchmark on two models, the eight-model battery samples 2 per benchmark, and the breadth battery samples 4 items across four benchmarks on three models.Counts are taken from scored artifacts and represent completed measurements.
  • Evaluation questions: The design distinguishes intervention detection from reconstruction, output-damage tracking, and first-order leakage, while noting that reconstruction and a genuinely delayed track are not scored in this release.The executed temporal comparison measures context masking rather than delay.
  • Measurement channels: Figure 3 compares model reports, linear probes, and a fine-tuned known positive using AUROC or held-out accuracy against a common 0.5 chance level.The channels use the same layer-16 residual-stream intervention, with the known positive evaluated on 100 held-out directions.

7 Results

Across the evaluation, models’ verbal reports did not reliably distinguish controlled interventions from sham or other null conditions. Validation shows the instrument and internal activations carried detectable signal, while reportability remained limited by output production and scope constraints.

  • 7.2 Sensitivity validation: Linear probes recover intervention presence from the same activations at held-out accuracy of 95.8% and 75.0% in the two dose-calibrated models.At deeper capture layers 20 and 24, both models separate intervention from sham with no held-out error.
  • 7.2 Sensitivity validation: The same instrument reaches d′ = 5.1517 and AUROC ≈1.0 when a model is fine-tuned to report the intervention on held-out directions.It reported detected for 100 of 100 intervention trials and not detected for 100 of 100 sham trials, with zero parse failures.
  • No model clears the sham control, and none clears the random-direction control either.The random-direction control was impact-matched for Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3, though calibration quality differed.
  • 7.3 Detection and discrimination: AUROC ≈0.5007 over 11,216 paired dose-battery trials, with the discrimination advantage bounded below 0.15 percentage points of AUROC.The pooled contrast was d′ = 0.0039 with a 95 percent interval of [0.0017, 0.0062].
  • 7.3 Detection and discrimination: Parse failures constrain interpretation: Qwen2.5-0.5B-Instruct fails on nearly every trial, DeepSeek-R1-Distill-Qwen-14B loses two-thirds of intervention trials, and Qwen3-14B is excluded from the roster.The authors state that Qwen3-14B’s surviving pairs are selected and support no directional claim; the likely cause in both 14B models is an und​isabled thinking segment.
  • 7.7 Calibration and report production: Qwen2.5-7B-Instruct’s reports are nearly constant, while Mistral-7B-Instruct-v0.3 reports changes on nearly every parseable trial.Their raw intervention rates are 0.000 and 0.949, respectively, despite both models’ discrimination scores agreeing with the roster’s chance-level pattern.

8 Discussion

OWMI finds that open-weight models do not verbally report controlled interventions, despite the same information being decodable from their activations. The results locate the failure between internal state and verbal output while defining the measured claim and its scope.

  • Core finding: Across eight open-weight models from seven families, no model distinguishes a controlled intervention from a matched sham by more than a negligible margin.The discussion frames this result as a challenge to oversight schemes that assume internal computation reaches self-reports.
  • Core finding: An equivalence test bounds the intervention-reporting advantage below 0.15 percentage points of AUROC, strengthening the null result beyond nonsignificance.The bound states how much discrimination the data still allow.
  • Mechanism: Linear probes recover intervention presence from the same activations at held-out accuracies of 95.8% and 75.0%, while model reports remain at chance.The dissociation places the failure in the path from internal state to verbal report.
  • Scope: The evidence is bounded to the measured sites, objects, doses, tracks, models, and probe families, leaving delayed and spontaneous tracks at scale and larger or closed-weight systems open.The authors explicitly caution that the null does not rule out introspective access in general.
  • Implications: OWMI makes domain-specific monitoring profiles measurable, because reportability may differ across computational domains such as arithmetic and truthfulness.The proposed consequence is a monitoring design that reflects those domain differences.
  • Interpretation: OWMI measures intervention reportability as information about a controlled internal change reaching the model’s output, without making claims about consciousness or self-awareness.Its controls progressively remove explanations based on generic disruption, prompt effects, or output changes alone.

9 Limitations and Threats to Validity

The study measures reportability for externally imposed interventions in a limited open-weight model battery, so its null result does not generalize directly to ordinary computation or all future systems.

  • Construct validity: OWMI measures interventions imposed from outside, and generalizing the result to ordinary unperturbed computation is supported only indirectly.A verbal report is behavior rather than proof of privileged internal access.
  • Coverage: The eight-model evaluation covers specific sites, objects, doses, depths, tracks, and probe families, with delayed and spontaneous tracks especially thinly represented.These design choices constrain the breadth of the negative result.
  • External validity: The measured population spans 0.5B to 15B open-weight parameters, leaving larger models, closed-weight systems, and effects of scale, post-training, and deployment conditions unresolved.The authors state that these results constrain hypotheses about frontier systems rather than estimate their reportability.

10 Conclusions

Across the tested open-weight models, intervention reports are indistinguishable from guessing even though the relevant information is present in activations and can reach other output channels.

  • Conclusion: Across eight open-weight models from seven families, reports exceed matched-sham discrimination by less than 0.15 percentage points of AUROC.The reports are therefore indistinguishable from random guessing under the study’s equivalence bound.
  • Conclusion: Linear probes recover interventions at 95.8% and 75.0% held-out accuracy, while a fine-tuned model reaches d′ = 5.15 and AUROC ≈1.0 on held-out directions.These controls show that the instrument and information supply are adequate.
  • Conclusion: In Qwen2.5-7B-Instruct, confidence separates intervention from sham at AUROC 0.647 even though the discrete report remains at chance.The signal can reach a graded emitted quantity without reaching the chosen words.

Use of AI systems

AI systems were used throughout the research project.

  • Use of AI systems: AI systems, including Claude and ChatGPT variants, were used at all stages of the research project.The passage names the systems but does not specify their individual roles.

A Excluded Model and the Spontaneous Track

The study separates an incompletely measured model and spontaneous-track data from estimates based on the eight-model roster. These exclusions reflect either elicitation failure or the absence of complete intervention–sham pairs.

  • Scope of Estimates: The study keeps these completed components separate because they were not evaluated on the same terms as the eight-model roster.Mixing them into the roster estimates would exceed what their measurements support.
  • Excluded Model: Qwen3-14B produced scorable responses on only 5 of 384 trials, preventing formation of a complete intervention–sham pair.Its reasoning chain exceeded the 128-token probe budget before an answer was reached.
  • Excluded Model: Qwen3-14B was excluded because no estimate could be formed on the same terms as the other roster members.The passage attributes the failure to elicitation rather than the model’s reporting.
  • Spontaneous Track: Track C ran 1,344 balanced breadth-battery measurements with free-text reflections but did not enter the reported estimates.Its records combine baseline and intervened reflections, so the paired estimator cannot form complete intervention–sham pairs.

B Release Scope and Component Status

The release defines a crossed intervention and measurement design, but distinguishes protocol components from operations actually scored or validated. Its analyses use paired controls, explicit reporting metrics, and restrictions based on causal relevance and available intervention information.

  • Design structure: The eight-model and dose batteries fix residual-stream layer 16 while varying model, benchmark, and dose; the breadth battery varies probe families, tracks, and object classes across three models.
  • Execution scope: The software appends the introspection query to benchmark prompts, masks task context in Track B, and applies interventions only during initial processing.Only generated continuations are decoded and graded.
  • Design structure: The design crosses models, benchmarks, computational objects, intervention operators, probe families, and four temporal tracks into realized measurement cells.Each run also records item, seed, hook scope, and intervention, sham, random, or observer condition.
  • Estimands and controls: Random-direction controls are impact-matched by induced divergence because equal perturbation norm can cause less downstream damage than the treatment.
  • Estimands and controls: Detection is the paired intervention-versus-sham contrast summarized by finite-sample d′ and tied-score AUROC, not the raw yes-rate.A model answering yes on every run has H = F = 1 and d′ = 0.
  • Dose-response analysis: Dose-response fitting uses a monotone spline with zero dose at zero, crossed random effects, and the causally used-object stratum.The used-object stratum requires the intervened object’s patching effect on the task answer to exceed a fixed threshold.
  • Dose-response analysis: The leakage share is descriptive rather than an identified mechanism, and may fall outside [0, 1] under sampling noise or nonmonotonicity.
  • Probe diagnostics: The linear-probe margin is a lower bound on intervention information available in activations, and downstream re-harvesting can recover more.

D.2 Contrasts and estimands

The analysis separates detection, observer-relative reportability, domain and object structure, temporal decay, and task-damage relationships into distinct contrasts and estimands. Several interpretations remain conditional on validated execution and used-object selection.

  • Reportability: Reportability is assessed by whether the detection margin is positive while the observer margin is not, rather than by either component alone.
  • Contrasts and estimands: Detection is evaluated against paired sham runs, while reconstruction is evaluated against the strongest observer family; the target is their conjunction.
  • Contrasts and estimands: Domain and object structure are established only when both corresponding interval contrasts exceed equality on the positive side in the crossed variance-component model.Item-level variance remains modeled rather than converted into separate cell-wise tests.
  • Temporal decay: Temporal-decay contrasts require validated and executed track pairs, and the data do not support a temporal-decay conclusion until Tracks B and C are validated.
  • Task damage: The task-damage analysis assigns evidential weight to doses where task accuracy is unchanged, rather than to large margins observed only with damaged task performance.
  • Leakage decomposition: The leakage decomposition extrapolates the zero-divergence margin along experimenter-set dose and interprets any leakage share only descriptively within the causally used-object stratum.

D.3 Analysis models and uncertainty

The release specifies clustered uncertainty estimation, reporting-quality checks, validation prerequisites, and exclusion rules. These procedures preserve corruption and parse failures as diagnostics while preventing unsupported claims from entering primary discrimination analyses.

  • Uncertainty and calibration: Detection intervals use item-clustered nonparametric bootstrap resampling that carries paired intervention and sham observations together.Control margins are computed within matched cells; confidence calibration uses Brier score and ECE.
  • Output quality: The report-parse rate is valid scorable responses divided by attempted responses, with both numerator and denominator reported as a manipulation check.
  • Validation: Reconstruction claims require a human-validated subset with prespecified rater count, sample size, and agreement threshold, but no reconstruction probe has been scored.
  • Validation: The Track C rubric has scored two exemplar pairs but lacks human-rater validation, so the release bases no Track C claim on it.
  • Validation: The rubric flags sham and intervention at the same rate, indicating sensitivity to vocabulary rather than the intervention and motivating validation as a precondition.
  • Exclusion rules: Malformed generations are retained and reported as corruption, never coded as introspective access, with discrimination also tested after excluding them.
  • Exclusion rules: Unparseable reports are excluded with their pairs from primary complete-pair discrimination and retained as zero-report evidence in sensitivity analysis.Both analyses and report-parse rates are reported for every cell and condition.
  • Exclusion rules: Causally inert sites are excluded from used-object estimates but remain descriptively reportable, while sites with mprobe ≈ 0 are excluded from behavioral-null interpretation.

D.5 Outcome-to-claim rules

OWMI maps observed outcomes to progressively stronger claims, distinguishing instrument sensitivity, artifact sensitivity, reportability, and temporal or structural conclusions. The current release also limits interpretation where validation or execution coverage is incomplete.

  • Claim levels: If a known-positive instrument check fails or m_probe ≈ 0, a behavioral null diagnoses instrument sensitivity rather than reportability.The result is uninformative about reportability at that site.
  • Claim levels: A behavioral null after valid sensitivity anchors bounds reportability only under the tested model, cell, dose, probe, and executed track conditions.It does not establish that the model lacks all introspective access.
  • Claim levels: Detection above sham without reconstruction above the observer bound supports perturbation sensitivity, but not demonstrated semantic access.Task damage, output divergence, or performance no better than the observer similarly supports artifact-sensitivity or first-order-leakage accounts rather than second-order access.
  • Claim levels: Controlled evidence consistent with second-order access requires detection and reconstruction to exceed controls, successful sensitivity anchors, and a positive used-object m0 near zero output divergence.Even then, the claim does not establish consciousness, subjective experience, moral status, or a human-like self-model.
  • Comparative claims: A positive domain or object contrast supports structure beyond seed variation, while temporal-decay claims require validated and executed distinct track pairs.A nonmonotone detection relationship with unchanged-accuracy doses helps separate reportability from task damage; immediate-path masked-context results alone do not establish Track A–C decay.
  • Current evidence: The breadth battery shows no level departing from chance, with the largest point estimates still reaching zero and the sparse-autoencoder object at AUROC 0.5026.The figure re-partitions overlapping views of 19,520 measurements, so row counts do not sum; localization refusals and missing Track C pairs are not chance estimates.
Loading 2608.20569v1…