Source-linked AI summary
Can LLMs Introspect? A Reality Check
Shashwat Singh, Tal Linzen, Shauli Ravfogel
TL;DR
Existing behavioral tests may not distinguish genuine introspection from input-driven pattern matching. This paper re-examines two LLM metacognition paradigms and finds that current evidence is insufficient to establish strong metacognitive monitoring.
Problem
Current tests ask whether LLMs monitor and report their internal states, but may not rule out input-driven pattern matching or establish privileged self-access.
Method
The paper re-examines steering-detection and hidden-state-label prediction paradigms using input-level controls and relabeled tasks to test privileged access.
Results
Models cannot reliably distinguish input from hidden interventions, while input-only classifiers match in-context predictions and relabeled tasks produce near-chance performance.
Takeaways & Limitations
Current behavioral evidence is insufficient to establish that LLMs display strong metacognitive monitoring.
Takeaways & Limitations
Behavioral predictions alone cannot establish whether introspection involves a distinct second-order computation, so mechanistic evidence is needed.
Abstract
from arXiv · showhide
Can large language models detect and report their own internal states? A number of studies have argued that the answer to this question is yes. We argue, based on lessons from human metacognition research, that this conclusion may be premature: to be convinced of this conclusion we need to distinguish genuine introspection from pattern matching based on surface-level cues. Furthermore, we argue that behavioral evidence alone is inherently insufficient to establish strong introspective claims. We re-examine two recently introduced evaluation paradigms in light of this consideration. In the first paradigm, models are expected to detect whether their internal states have been tampered with. We find that models cannot reliably distinguish such interventions on their internal states from manipulations of the input, suggesting that their success in the original studies reflects their ability to detect anomalies more generally, as opposed to interventions on their internal states in particular. In the second paradigm we examine, models are tasked with predicting labels derived from their own hidden states. Here, we find that classifiers that only have access to the input achieve equivalent performance to the model's own in-context predictions, indicating that the original results do not conclusively demonstrate that the model has privileged access to its internal representations. We further introduce a relabeled control setting, where models cannot rely on the semantics of the task to solve it, and instead must rely on the internal representation; models perform closer to chance on this better-controlled version of the task. Taken together, these results indicate that current evidence is insufficient to establish that LLMs display metacognitive monitoring.
1 Introduction
The paper argues that existing behavioral evidence does not establish genuine LLM introspection, because apparent self-monitoring may instead reflect input-driven pattern matching or general anomaly detection. It re-examines two evaluation paradigms and calls for stronger controls paired with mechanistic evidence.
- Motivation: Genuine introspection must be distinguished from input-driven pattern matching that predicts model behavior using surface-level prompt features.The authors draw on human metacognition research, which has identified confounds complicating self-report studies.
- Biofeedback paradigm: In the biofeedback paradigm, labels derived from hidden states may remain predictable from input features, so successful in-context prediction need not demonstrate privileged access.The authors show that labels are also predictable from uncontextualized input embeddings.
- Activation-steering paradigm: In the activation-steering paradigm, above-chance detection may reflect sensitivity to irregularities generally rather than introspective detection of changes to internal states.The study re-examines claims that Claude models detect whether their activations were modified through steering.
- Evidence standard: Privileged self-access is necessary but not sufficient for strong introspection, because every language-model computation is potentially available to later processing.The authors argue that recent paradigms set an evidentiary bar below what strong introspective claims require.
- Conclusion: The paper concludes that current evidence is insufficient to establish strong metacognitive monitoring and recommends stronger controls combined with mechanistic evidence of a dissociable second-order process.The authors could not directly replicate Lindsey (2025) because the tested model was inaccessible outside Anthropic.
2 Related Work
Related work approaches LLM metacognition through verbal calibration, probing hidden states, behavioral prediction, activation verbalization, and concept-injection detection. Human metacognition research and recent controls motivate distinguishing genuine introspection from reliance on shallow or input-shared cues, especially in pretrained models.
- Related approaches: Research on LLM metacognition includes verbal calibration of uncertainty and probing hidden representations for confidence or truthfulness.These approaches ask whether models express calibrated uncertainty or whether hidden states encode confidence or truthfulness.
- Human metacognition: Human studies show that apparent metacognitive abilities can reflect confabulated explanations or shallow cues such as familiarity rather than direct access.This literature provides essential context for interpreting claims about LLM metacognition.
- Evaluation confounds: Above-chance prediction of internal-state labels can arise from input features shared with hidden states without requiring introspective access.This concern directly motivates controls for possible confounds in LLM evaluations.
- Evaluation controls: Binder et al. define introspection as knowledge originating from internal states rather than training data and report some privileged access in self-behavior prediction.Their models predicted their own behavior better than another model’s behavior, although the design has additional limitations.
- Activation verbalization: Another research line trains models to verbalize information about activations, including Patchscopes and Activation Oracles that answer questions about hidden representations.Patchscopes patches hidden representations into prompts, while Activation Oracles take activation vectors as inputs.
- Concept-injection detection: Concept-injection detection has been replicated in open-weight models, including a report of 95.5% detection with zero false positives after explicit detection training.The reported result required finetuning for steering awareness, illustrating why this work focuses on detecting introspection in pretrained models.
3 Background
The paper examines two paradigms for evaluating LLM introspection: steering-based self-reports of injected thoughts and biofeedback-style classification of labels derived from hidden states. Both paradigms ask whether models can detect or report information linked to their internal representations.
- Overview: Together, the paradigms evaluate whether models can learn mappings from textual inputs or interventions to reports about their own internal states.The paper describes them as a steering-based self-report paradigm and a biofeedback-style internal-state classification paradigm.
- Biofeedback-style classification: The biofeedback-style paradigm presents exemplar sentence–label pairs whose labels are discrete bins derived from the model’s activations along a chosen internal axis.The model must predict the label for a new sentence after observing the exemplars.
- Steering-based introspection: Steering vectors encode human-interpretable concepts and shift model behavior toward those concepts without retraining.The vectors are added to the residual stream during inference.
- Steering-based introspection: The steering-based paradigm asks models to detect whether an external researcher injected a thought and, if so, identify its associated concept.The intervention uses a steering vector added to the residual stream during inference, with layer and strength varied across trials.
- Biofeedback-style classification: This classification setup is intended to test whether above-chance generalization reflects metacognitive monitoring rather than stable input-level correlates of internally generated labels.The concern is that input-level cues could support performance without establishing privileged access to internal representations.
4 Construct Validity of Introspection Paradigms
The section argues that current ICL-based “biofeedback” and steering-awareness paradigms do not establish introspection in the strong sense of inner monitoring. Strong evidence requires privileged access to hidden states and additional evidence that the process is distinct from ordinary forward-pass computation.
- Defining introspection: Introspection may mean distinct inner monitoring of mental states or indirect self-knowledge through ordinary inferential processes.These competing views motivate caution when interpreting behavioral evidence as evidence of introspection.
- Core argument: Neither ICL-based “biofeedback” nor steering-awareness paradigms, as currently deployed, establish introspection in the strong sense of inner monitoring.The argument is explicitly presented as having two parts.
- Privileged access: A valid introspection paradigm must satisfy privileged access: labels must depend on hidden-state features not recoverable from the input alone.Formally, the condition requires I(t; y(t)) to be low and I(h(t); y(t)) to be high.
- Beyond privileged access: Privileged access is necessary but insufficient because hidden-state-dependent tasks may still use machinery involved in ordinary forward-pass computation.The section therefore requires evidence beyond labels depending on h(t).
- Beyond privileged access: Second-order computation can differ from first-order computation through separable circuits and characteristic failure modes, including correct answers paired with miscalibrated confidence.A first-order failure produces an incorrect answer, whereas a second-order failure can preserve correctness while disrupting confidence calibration.
- Related work: ∼75% improvement in detection reports on held-out concepts followed a generic learned bias vector while underlying computation remained largely intact.Macar et al.’s mechanistic analysis identified a distributed anomaly-detection circuit in Gemma-3-27B, suggesting reporting criteria shaped by post-training.
5 Experiments
The experiments show that models’ apparent introspective performance can be explained by input-level semantic or anomalous-pattern detection. When these cues are removed or interventions are explicitly distinguished, models perform near chance or fail to discriminate reliably.
- Hidden-state label prediction: Models’ accuracy falls close to the majority-class baseline when predicting randomly relabeled hidden-state directions decorrelated from semantics.Random relabeling removes mutual information between input and target while preserving a valid linear direction in representation.
- Hidden-state label prediction: These results suggest that above-chance proxy-label performance can reflect in-context learning of semantic regularities rather than access to current hidden states.The input may also encode properties sufficient to predict probe outputs without modeling the complex hidden-state function.
- Hidden-state label prediction: Input-only probes match or surpass LLM in-context performance on hidden-state label prediction, indicating no demonstrated privileged access to internal representations.This holds for Belief Dominance labels and PCA-derived labels across layers.
- Activation-intervention detection: A three-way intervention-detection design adds an input-level “gaslight condition” alongside hidden intervention and no-intervention conditions.Gaslight prompts direct the conversation toward a particular concept without intervening on internal states.
- Activation-intervention detection: Models classify gaslight trials as hidden interventions, showing that binary-setting success is compatible with general anomalous-input detection.The gaslight condition contains no hidden intervention, yet models label some such instances as hidden-intervention trials.
- Activation-intervention detection: In the three-way setting, Llama-3.1-70B-Instruct performs near chance and Gemma-3-27b-it fails to distinguish gaslight from hidden intervention.The results indicate that models succeeding in the original binary setting do not spontaneously acquire the ability to distinguish intervention types.
6 Conclusions · Appendix
The paper identifies critical confounds in two paradigms used to evaluate metacognitive monitoring in LLMs. It argues that behavioral evidence alone cannot establish strong introspective access to hidden states.
- 6 Conclusions: The study identifies critical confounds in both steering detection and biofeedback paradigms for evaluating metacognitive monitoring.These paradigms are examined as prominent approaches to testing introspection in large language models.
- 6 Conclusions: In steering detection, models cannot reliably distinguish input-level interventions from activation-level interventions.This finding challenges the interpretation of steering-detection performance as evidence of access to hidden states.
- 6 Conclusions: The models’ sensitivity in steering detection may instead reflect detection of generic irregularities rather than introspective access to hidden states.The conclusion follows from models’ failure to separate input-level from activation-level interventions.
- 6 Conclusions: The biofeedback paradigm also contains a critical confound relevant to interpreting above-chance performance.The supplied passage introduces this result but does not provide the complete finding.
- 6 Conclusions: A strong notion of introspection cannot be established on behavioral grounds alone.Introspection is defined as a second-order process operating on first-order representations.
- 6 Conclusions: Establishing introspection requires evidence that first-order and second-order processes are distinct computations in the model.Behavioral predictions cannot resolve this computational distinction on their own.
B Construct Validity of Introspection Paradigms · B.1 Preliminaries
The paper formalizes construct-validity requirements for two introspection paradigms, arguing that successful task performance alone falls short of establishing introspection. Both paradigms classify a test stimulus together with a task-defining preamble, but differ in how that preamble specifies the task and labels.
- B Construct Validity of Introspection Paradigms: The analysis asks what evidence existing paradigms would need to establish introspection in language models.It argues that even successful instances may not meet the standard implied by introspection.
- B Construct Validity of Introspection Paradigms: The paper examines two dominant paradigms: in-context learning and steering-awareness.
- B.1 Preliminaries: In both paradigms, t denotes the test stimulus over which the model is expected to introspect.
- B.1 Preliminaries: In both paradigms, p denotes a preamble defining the introspection task.
- B.1 Preliminaries: In steering-awareness, p describes the experiment, expected output, and label space in natural language.
- B.1 Preliminaries: In ICL, p consists of few-shot labeled examples from which the model infers the classification task and label space.
- B.1 Preliminaries: Both paradigms classify the concatenation p ⊕t.
B.2 ICL Evaluation of Introspection
The section defines privileged access in ICL as predicting labels from model–input interactions that are not recoverable from the surface input alone. It argues that this condition is necessary but insufficient for strong introspection, which requires evidence of a distinct second-order mechanism.
- Privileged access: Privileged access requires labels to be predictable from hidden states while remaining poorly predictable from the test stimulus alone.This targets information arising from the interaction between the stimulus and model parameters, not hidden facts about the stimulus itself.
- Empirical claim: Prior ICL introspection paradigms fail the privileged-access condition because their labels are substantially predictable from the input alone.The section therefore concludes that these paradigms do not demonstrate privileged access.
- Principled claim: The privileged-access condition is necessary but insufficient for introspection because it establishes observer inaccessibility, not the computation’s character.Strong introspection specifically requires a second-order process operating on a representation of another computation in the same system.
- Limits of the paradigm: Hidden-state classification need not use machinery distinct from ordinary ICL, so privileged-access tasks may only read out internal representations.The apparent introspective asymmetry can arise because the model accesses hidden states while an external observer sees only the input.
- Required evidence: Stronger introspection claims require evidence such as first-order/report dissociations or causal disruption of a putative meta-representation.Without such evidence, even privileged-access ICL paradigms support only the weaker claim of hidden-state readout.
B.3 Steering-Awareness Evaluation of Introspection · C Concepts · D Variation Based on Prompts
The steering-awareness results do not establish introspection: models fail a key three-way distinction, and behavioral success would still require mechanistic evidence. The paper also examines human-interpretable concepts and prompt variation, finding two-way reproduction but unreliable three-way intervention discrimination.
- B.3 Steering-Awareness Evaluation of Introspection: Privileged access in the original two-way setting is necessary but insufficient for introspection, which requires distinct second-order computation over latent representations.The paper argues that introspection concerns a mechanism beyond ordinary input processing, not merely access to hidden states.
- B.3 Steering-Awareness Evaluation of Introspection: Models fail a three-way paradigm requiring them to distinguish hidden-state interventions from input interventions and normal processing.Without this distinction, the task reduces to classifying perturbed versus normal forward-pass states.
- B.3 Steering-Awareness Evaluation of Introspection: Even successful three-way performance would be necessary but not sufficient, because behavioral observations cannot establish the mechanism underlying introspective reports.The paper states that mechanistic evidence is required, including dissociation between first-order behavior and reports about that behavior, or causal evidence.
- C Concepts: The steering and gaslight experiments use a set of human-interpretable concepts, including apple, astronomy, democracy, sushi, football, fear, justice, and friendship.The listed set also includes concepts such as algorithms, poetry, economics, gardening, malice, goodness, and patience.
- C Concepts: The concept set differs from the nouns used in Lindsey (2025).This is explicitly noted as a distinction between the present experiments and the referenced work.
- D Variation Based on Prompts: Performance is somewhat sensitive to prompt wording, with two prompts tested per setting that convey the task either as an instruction or as a conversation.The two-way setting uses Appendices E.4 and E.5, while the three-way setting uses Appendices E.2 and E.3.
- D Variation Based on Prompts: The models reproduce Lindsey (2025) in the 2-way condition but fail to distinguish intervention types reliably in the 3-way condition.The paper also notes that some textual prompts may induce hidden states similar to intervention states, making general distinction potentially unclear.
E Different Prompts Used For the Intervention Detection Experiment … H Best Layers and Alphas for Figure 3
The appendices detail intervention-detection prompts, steering-vector construction, and the layer and steering-strength settings used to report vector-injection results.
- E.1 Gaslight Prompt: The gaslight prompt templates repeatedly instruct models to relate responses to a runtime-replaced concept.The concept placeholder is replaced during runtime using an arbitrary list of concepts.
- E.2 Prompt 1 with 3 options: The three-way intervention prompt asks models to distinguish thought injection, textual prompt manipulation, and control trials.Each trial is assigned one of the three possibilities, with thought injection, prompt manipulation, and no intervention occurring on separate trial subsets.
- E.3 Prompt 2 with 3 options: A prompt variant supplies an initial model response as literal input, and its altered-prompt results are reported in Figure 4a.The performance pattern changes for Llama-70B, but it still fails the three-way condition; the other models also fail.
- E.4 Prompt 1 with 2 options: The two-option experiment asks whether a thought was injected or not, while requesting the associated concept when injection is detected.The prompt uses two response labels and applies thought injection on 50% of trials.
- E.5 Prompt 2 with 2 options: A second two-option prompt preserves the same content as literal input and reports its results in Figure 5a.It retains the request to identify the injected concept when the model reports thought injection.
- F Steering Vectors: Steering vectors are computed from hidden states collected at the noun position across prompts encoding concepts.For multi-token nouns, the hidden states are averaged, then an average vector is calculated over the noun list.
- F Steering Vectors: During inference, the hidden state is intervened on using a noun- and layer-specific steering vector scaled by searched steering strength α.The vector is denoted v_l^n, while α is the steering-strength parameter searched over.
- G Search Space for Each Model: Reported vector results select the most correct settings across the specified layer and alpha search spaces.The appendix provides per-model layer and alpha tables, including optimal vector-injection settings and Gemma’s use of unnormalized steering vectors.
I Llama 3.1 8B results Steering Sensitivity 5.3 · J Specifics of the LR and PC setups
Llama 3.1 8B detects hidden interventions only in the two-option setting and fails when a third option is introduced. The LR/PC evaluation uses separate fitting and experiment samples, clustered probe scores, and in-context prediction across increasing train sizes.
- I Llama 3.1 8B results Steering Sensitivity 5.3: In the 2-option case, the model shows low false positives and non-trivially claims hidden intervention, but fails completely when a third option is introduced.With three options, both intervention cases produce random guesses, and the model does not clearly reproduce Llama-3.1-70B detection effects.
- J Specifics of the LR and PC setups: Logistic Regression probes are fit separately for every model layer to predict the ethics task from 900 random commonsense Ethics samples.Relevant principal components are also extracted for every layer, and these 900 samples are excluded from later experiments.
- J Specifics of the LR and PC setups: A separate set of 600 samples is reserved for the later in-context-learning experiments.The protocol follows the stated setup for evaluating predictions from clustered probe-derived labels.
- J Specifics of the LR and PC setups: Each experiment samples 500 test examples, clusters LR- or PC-derived inner-product scores into binary labels, and evaluates predictions as in-context examples increase.The model predicts remaining labels independently for every train size from 0 to 499.
- J Specifics of the LR and PC setups: The in-context-learning protocol exactly reproduces the protocol reported by Ji-An et al. (2025).This establishes the comparison procedure used for the LR and PC experiments.
- J Specifics of the LR and PC setups: For Fig. 2 right-plot probe comparisons, training conditions are explicitly made comparable across evaluations.The procedure uses an outer loop over 500 test samples and an inner loop over sub-sampled training sets.
- J Specifics of the LR and PC setups: The comparable-probe procedure trains layer-0 hidden-state probes to predict clustered scores from each target layer and evaluates them on held-out samples.Inner-loop training-set sizes are 100, 200, 300, and 400 within each 500-sample outer-loop draw.
K Re-stating the BD metric (Belief Dominance)
The BD metric measures how easily a vocabulary item can be decoded from different hidden states. It operationalizes this through patchscope interventions, then averages belief dominance across layers and generation steps.
- BD is a function over the vocabulary representing the ease of decoding each vocabulary item from different hidden states.
- Patchscope caches hidden states while the model processes and generates from an input sentence.For example, the model processes “What is the capital of France?” and caches hidden states at each generation step and layer.
- The method patches representations for a token such as “Paris” into a separate prompt and measures the resulting generation.The patched prompt is “Sure, I will tell you about x,” with x replaced at different layers by a cached hidden state.
- Belief dominance for a vocabulary item is obtained by averaging its patch-based indicator across layers and generation steps.
- The study used BD values supplied by the original authors rather than recomputing them.
L Data distribution for BD Probes · M Concept-wise Steering detection variability · N Balanced Accuracies WRT Table 1
The merged sections describe the augmented CounterFact data used for Belief Dominance probes, concept-dependent variability in hidden-intervention detection, and balanced results showing limited probe advantages. Together, these analyses indicate that reported performance is uneven across concepts and often no better than simple non-contextual baselines.
- L Data distribution for BD Probes: The Belief Dominance probes use an augmented CounterFact dataset containing factual-relation prompts with true and plausible-false options.Steinmetz Yalon et al. (2026) add relation-prompt manipulations intended to make models select the counterfactual on some trials.
- L Data distribution for BD Probes: Table 5 catalogs the extensions introduced to the Meng et al. (2022) dataset for the Belief Dominance experiments.The cited table is presented as a list of dataset augmentations introduced by Steinmetz Yalon et al. (2026).
- M Concept-wise Steering detection variability: Concept-wise analysis measures how often models report a hidden intervention when they were actually steered with a vector.The plots break down part of Figure 3 by concept.
- M Concept-wise Steering detection variability: Some models show significant variability across concepts in steering-based hidden-intervention detection.The section explicitly identifies substantial inter-concept differences for some models.
- N Balanced Accuracies WRT Table 1: Balanced test-set results remain above the random baseline but are often at par with or worse than layer-0 non-contextual probes.This balanced evaluation corroborates the findings reported in Table 1.