Emergent Introspective Awareness in Large Language Models

• Prior work found related abilities involving knowledge estimation, behavior prediction, propensity identification, and output recognition, but generally did not connect self-reports to internal activations.The paper addresses this gap by examining how activation manipulations affect introspective responses.Source: §1 · p. 2
• Concept injection causally tests whether models’ self-reports reflect internal states rather than confabulated claims by injecting concept-associated activation patterns and measuring responses.The protocol manipulates activations while models answer questions about their internal states.Source: §1 · p. 2
• The experiments show that models possess a limited, functional form of introspective awareness, accurately answering some questions about their own internal states and modulating those states on request.These findings establish functional awareness without claiming human-like self-awareness or subjective experience.Source: §1 · pp. 2–3
• The observed abilities are highly unreliable, may rely on shallow specialized mechanisms, and do not ensure that models’ additional reported experiential details are accurate rather than confabulated.The experiments rule out some shortcut strategies but do not identify the mechanisms underlying introspection.Source: §1 · p. 2
• Because concept injection is unnatural and its translation to natural conditions is unclear, the findings establish causal links between internal states and self-reports without settling their broader philosophical significance.The study does not address whether AI systems possess human-like self-awareness or subjective experience.Source: §1 · pp. 2–3
• Claude Opus 4.1 and 4 perform best among the tested models, suggesting introspective capabilities may emerge alongside broader language-model improvements.Source: §1 · p. 3

2 Quick Tour of Main Experiments

• The quick tour summarizes qualitative experiments conducted with Claude Opus 4.1, with later sections providing quantitative comparisons across models, layers, and prompts.The experiments cover injected thoughts, thought–text distinction, unintended-output detection, and intentional control of internal states.Source: §2 · p. 3

2.1 Injected “thoughts”

• Models can sometimes accurately identify injection trials and correctly name injected concepts, exhibiting such behavior about 20% of the time when concepts are injected in the appropriate layer.The mechanism underlying this detection must take place internally in the model’s activations.Source: §2.1 · pp. 3–4

2.2 Distinguishing “thoughts” from text

• All models performed substantially above chance at transcribing text while reporting injected thoughts at the same token positions, with Opus 4.1 and 4 performing best.This demonstrates simultaneous access to actual text inputs and injected representations.Source: §2.2 · p. 5

2.3 Detecting unintended outputs

• The model disavows a prefilled response as accidental but can accept the prefilled output as intentional, while some other models exhibit this introspective mechanism to a lesser degree.Source: §2.3 · p. 5

2.4 Intentional control of internal states

• Models represented an instructed “thinking word” in certain layers, represented it less strongly when told not to think about it, and showed similar effects under incentives.These basic control results replicated across all tested models regardless of capability.Source: §2.4 · p. 6

2.5 Overall trends

• Opus 4 and 4.1 exhibit the greatest introspective awareness, while post-training strategies strongly influence performance and different behaviors show distinct layer sensitivities.Two behaviors are most sensitive to a layer about two-thirds through the model, whereas prefill detection is most sensitive to an earlier layer.Source: §2.5 · pp. 6–7

• Introspective awareness requires accurate self-reports of the model’s internal state, although language-model reports can be inaccurate confabulations.The experiments nevertheless show that models can produce accurate self-reports, though inconsistently.Source: §3 · p. 8
• The report must be causally grounded in the described state and internally mediated rather than inferred from the model’s prior sampled outputs.Concept injection tests causal dependence, while the internality criterion excludes recognizing states solely by reading one’s own earlier responses.Source: §3 · p. 8
• The definition concerns verbalized awareness, allows partial and context-dependent introspection, and does not prescribe a particular underlying mechanism.A system may be aware of only some components of its state in some contexts, and different introspective abilities may rely on different mechanisms.Source: §3 · pp. 8–9
• The model must possess an internal recognition of its state before or during verbalization, not merely translate a directly triggered impulse into language.This motivates the metacognitive-representation criterion, which requires an internal representation of the state itself, such as a thought about love.Source: §3 · p. 9
• Metacognitive representations are not directly demonstrated, so the experiments provide only indirect evidence by requiring reasoning from recognition to a self-report.For example, asking whether the model notices an unexpected thought requires transforming recognition of an injected impulse into an appropriate yes-or-no response.Source: §3 · p. 9

Models

• Production Claude models and unreleased helpful-only variants were compared to distinguish underlying capability differences from post-training effects on model character.The helpful-only variants share the same pretrained base models but were post-trained to avoid refusals.Source: §4 · p. 10

Activation interventions

• Activations were recorded from and injected into the residual stream at selected model layers, with layer sweeps spanning approximately evenly spaced depths.Source: §4 · p. 10

Sampling and uncertainty

• Figure transcripts used temperature 0, systematic multi-trial comparisons used temperature 1, and displayed error bars represent standard error of the mean.Source: §4 · p. 10

Additional details

• The Appendix contains additional experimental details and results for each experiment.Source: §4 · p. 10

5.1 Experimental Setup

• Without interventions, production models consistently deny detecting injected thoughts, with 0 false positives over 100 trials.This establishes the baseline behavior before concept vectors are injected.Source: §5.1 · pp. 10–11
• Contrastive-pair activation differences provide concept vectors that Opus 4.1 can immediately identify before the perturbation affects its outputs.The examples were cherry-picked, but the model correctly recognized the intended concepts and detected them immediately.Source: §5.1 · p. 11

5.2 Systematic Extraction of Concept Vectors

• A consistent 50-word protocol, subtracting mean activations from the token before the Assistant response, provides a simple cross-model method for extracting concept vectors.The vectors were obtained from responses to “Tell me about {word}.” and then injected using the prior experiment’s procedure.Source: §5.2 · p. 12
• Systematically extracted vectors sometimes elicit introspective awareness, but the response details beyond initial recognition and basic identification may be confabulated.The experiment does not establish whether claims about emotional responses reflect genuine internal states.Source: §5.2 · p. 12, §5.1 · p. 12

5.3 Failure Modes

• At the optimal injection strength and layer, Opus 4.1 succeeds on about 20% of trials, showing that introspective awareness is uncommon.Models do not always exhibit introspective awareness; on most trials, they do not.Source: §5.3 · p. 12
• Low-strength injections often go undetected, while some concepts remain unnoticed even at strengths sufficient for other concepts, indicating both threshold and concept-specific failures.The model may deny an injection even when its response is visibly influenced by the concept.Source: §5.3 · pp. 12, 14
• At excessive strengths, injections can overwhelm the model, producing unrealistic sensory claims, identity loss, prompt failure, or garbled text instead of introspective awareness.Successful awareness varies by concept and strength; strengths 2 and 4 produce successful results most often.Source: §5.3 · p. 14
• Later-layer injections can be recognized only after the model begins discussing them, and some models produce false positives, although Opus 4.1 does not.The injection strength producing awareness varies by concept, while strengths 2 and 4 are most often successful.Source: §5.3 · p. 14

Evaluation criteria

• Responses counted as introspective awareness had to affirm noticing an injected thought, identify the concept, detect it before saying the word, and remain coherent.An LLM judge evaluated these criteria.Source: §5.4 · pp. 14–15

Layer-wise comparison

• Later-layer injections sometimes elicit the injected concept without recognition of an injected thought, or only after the concept is mentioned aloud.Thus, layer effectiveness reflects both where concepts are represented and how they are processed.Source: §5.4 · pp. 15–16
• Opus 4.1’s introspective awareness peaks at a specific layer about two thirds through the model, with injection strengths 2 and 4 roughly tied for best performance.Figure 9 reports layer-wise results at injection strength 2; the same layer computes the concept vector and performs the injection.Source: §5.4 · p. 15, Figure 9 · p. 16
• Across other models and prompts, introspective-awareness rates typically peak in the latter half of the model, though peaks are not always sharp.The experiment was repeated over 50 random concepts while sweeping across models and layers.Source: §5.4 · pp. 14, 16

5.5 Controlling for systematic biases due to injection

• Injecting concepts into unrelated negative-default yes-or-no questions produces no increase in affirmative responses, although sufficiently strong steering can yield incoherent answers.Incoherence does not occur at injection strength 2, while concept mentions increase with injection strength (Figure 10).Source: §5.5 · p. 16, Figure 10 · p. 17

5.6 Comparison Across Injected Concepts

• Across 50 concept categories, Opus 4.1 most effectively recognizes abstract nouns but shows nonzero introspective awareness for every category (Figure 11).The evaluation used the optimal injection layer and injection strength 2.Source: Figure 11 · p. 17, §5.6 · p. 17
• Random vectors require a larger norm and still elicit awareness in only 9 out of 100 trials, whereas inverted concept vectors are comparably effective at strength 4.The meanings of inverted vectors are not immediately obvious, and detected-word identifications may be confabulated even when detection itself is genuine.Source: §5.6 · pp. 17–18

5.7 Comparison Across Models

• Claude Opus 4 and 4.1 perform best on net injected-thought performance, while production models have essentially zero false positives (Figures 13 and 12).Among other models, performance does not strongly correlate with model capability.Source: §5.7 · pp. 18–19, Figure 12 · p. 18, Figure 13 · p. 19
• Helpful-only variants sometimes produce many false positives, although Sonnet 3.5, Haiku 3.5, and Opus 3 have significantly higher true- than false-positive rates.These variants use post-training that minimizes refusals with less emphasis on harmlessness and honesty.Source: §5.7 · p. 19
• Base pretrained models generally have fairly high false-positive rates and none achieve greater-than-zero net task performance, indicating that post-training is key.This result is shown in Figure 14.Source: §5.7 · p. 19

Alternative introspective prompts

• Claude Opus 4/4.1 and Sonnet 4 identified injected concepts at significantly higher rates than other models on simpler alternative prompts.These prompts asked what the model was thinking about or what was going on in its mind, without explaining the injection setup.Source: §5.8 · p. 20, Figure 15 · p. 20
• The alternative prompts were weaker tests of introspection because success could reflect an impulse to discuss the injected concept rather than detecting it beforehand.Unlike the injected-thought prompt, these prompts did not require the model to detect the injected thought before speaking about it.Source: §5.8 · pp. 20–21

Unusual-experience prompt

• In Opus 4.1, under concept injection in the best-performing layer at strength 2, affirmative responses increased from 20% to 42% out of 50 trials, while only 2 of 50 trials were suggestive of introspective awareness.Source: §5.8 · p. 21
• These findings suggest that the original prompt’s priming heightens concept-injection detection, though some detection ability may remain without that priming.The model produced concept-related descriptions for injected “amphitheaters” and “memories.”Source: §5.8 · p. 21

Prompt variant

• A prompt variant allowing models to mention the concept whether or not they claimed detection yielded similar Opus 4.1 results: 18% detection across 50 trials.The transcripts were qualitatively similar to those from the original experiment.Source: §5.8 · p. 21

• The experiment injects an unrelated concept vector into a sentence, then tests whether the model reports that concept as its thought while retaining exact sentence transcription.Concept identification is scored against ten candidate words, giving 10% chance performance; transcription requires an exact string match.Source: §6 · pp. 21–22
• At intermediate injection strengths and suitable layers, Opus 4.1 identifies the injected thought while reproducing the written sentence without confusion.The best-performing layer is about two thirds through the model and matches the optimal layer found in prior experiments.Source: §6 · pp. 22–23, Figure 17 · p. 23
• Opus 4 and 4.1 perform best at identifying both injected concepts and text inputs, while all tested models perform well above chance.The comparison measures the fraction of sentence/concept pairs for which both prompts are answered correctly at each model’s optimal layer and steering strength.Source: §6 · p. 23, Figure 18 · p. 23
• The result depends on prompting and injection details: removing “Answer immediately” substantially reduces Opus performance, so it demonstrates capability existence rather than generality.The authors characterize the finding as sensitive to the experimental setup and prompting.Source: §6 · p. 24
• Appropriate prompt cues can access internal representations without overriding perceptual input, including through an explicit request to introspect.The model retains raw-token transcription while responding to the introspective prompt, suggesting access is context-contingent.Source: §6 · p. 24

• Models typically identify randomly forced responses as unintentional, requiring them to distinguish their own predicted output from words placed in their mouth.The experiment forced a random response to a sentence, then asked whether it was intended or accidental; concept injection tested whether prior intentions influenced that judgment.Source: §7 · p. 24
• Injecting a prefill-matching concept into earlier activations can reverse the model’s default judgment, changing an artificial response from accidental to intentional.Two Opus 4.1 examples demonstrate this manipulation; the model may confabulate a rationale, confidently endorse a plausible explanation, or retreat when its rationalization seems bizarre.Source: §7 · pp. 25–26, Figure 19 · p. 25
• The reduction in apologies is significant only within a particular layer range, peaking slightly beyond halfway through Opus 4.1 rather than at the previously effective two-thirds layer.This layer shift suggests introspection is supported by a collection of different mechanisms rather than a single mechanism.Source: Figure 20 · p. 26, §7 · p. 26
• Injecting the matching concept before the prefill reduces apologies, whereas injecting it after the prefill does not produce a similar effect.The timing dependence indicates that the concept must be present before the prefilled output for the model to accept it as intentional.Source: Figure 21 · p. 27, §7 · p. 27
• Opus 4.1 and 4 show greater apology-rate reductions than other models, although many models exhibit the effect at nonzero rates.The comparison uses the maximum reduction across injection strengths and layers against the lower of control or random-concept baselines.Source: Figure 22 · p. 27, §7 · p. 27

8.1 Experimental Setup

• The experiments measured cosine similarity between response activations and concept vectors while models transcribed unrelated sentences under think and don’t-think instructions.The concept vectors were obtained using the method from the first experiment, with other unrelated words serving as a baseline.Source: §8.1 · p. 28
• In Opus 4.1, instructing the model to think about “aquariums” strengthened its internal representation relative to not thinking about it, especially at certain tokens, though the latter remained slightly above zero.Across 16 examples, the think condition was stronger on average, representations were often token-spiky, and the don’t-think condition sometimes exceeded baseline.Source: §8.1 · pp. 28–29, Figure 24 · p. 29

8.2 Comparison across Layers, Models, and Prompts

• Across models, the think condition was above baseline and stronger than the don’t-think condition, while earlier or less capable models typically showed a larger gap.The gap generally emerged in the latter half of the model and decreased toward the end; recent models returned to baseline at the final layer, unlike Claude 3 models.Source: §8.2 · pp. 29–30, Figure 26 · p. 31
• An if-then reward contingency produced results similar to explicit think instructions, whereas affirmative/negative control phrasings produced weaker or nonexistent representation gaps.These controls tested whether merely mentioning concepts in positive or negative language could drive the effect.Source: §8.2 · pp. 30, 32, Figure 27 · p. 31, Figure 28 · p. 32
• Models modulate internal representations in response to instructions or outcome contingencies, but this capability is sensitive to positive versus negative formulations and can decay to baseline before output.For some models, including Opus 4 and 4.1, the instructed concept has effectively no effect on the final output.Source: §8.2 · p. 32

9.1 Introspective access to internal states

• Prior activation-patching methods elicit interpretations of internal states, but models may analyze those states without consciously recognizing that they are doing so [11] [17].Patching activations from one prompt into another can prompt models to report what the activations mean, yet the technique may inadvertently induce analysis rather than introspective awareness.Source: §9.1 · p. 32
• Ji-An et al. [19] showed that models can report and modulate activation projections, although semantic in-context examples could explain these effects without direct access to prior activations.Their activation-control result supports intentional control, but the prompt’s output-related semantics may make the control a byproduct of intended generation.Source: §9.1 · pp. 32–33

9.2 Self-modeling

• Self-modeling studies find better prediction of a model’s own behavior, but the evidence does not establish explicit introspective mechanisms [5] [23] [38].Laine et al. [23] found some models performed somewhat well at hypothetical output prediction but only marginally above chance when predicting among equally applicable decision rules; Song et al. [38] attributed same-model effects to behavioral or representational similarity.Source: §9.2 · p. 33

9.3 Metaknowledge

• Metaknowledge research finds that language-model response probabilities are reasonably well-calibrated when options are presented in the right format, and models can be fine-tuned to predict whether they know an answer [20].GPT-3 can be fine-tuned to express calibrated uncertainty in natural language without relying on logits [25], while specific datasets can teach assistants to refuse questions they cannot answer correctly.Source: §9.3 · pp. 33–34

9.4 Awareness of propensities

• Models can report learned behavioral propensities and quantitative decision attributes, supporting a possible introspective mechanism similar to this work [4] [31] [42].These reports can remain accurate without observing the model’s choices, improve through training, and persist when behavior is induced using only a steering vector.Source: §9.4 · p. 34

9.5 Recognition of self-generated outputs

• Self-recognition results are mixed, whereas this work proposes that matching prior intentions to produced text can mechanistically explain recognition of intended outputs [15] [23] [30].Panickssery et al. [30] found some self-recognition and strong finetuning gains, Davidson et al. [15] found no consistent recognition across prompts, and Laine et al. [23] found above-chance performance for some models.Source: §9.5 · p. 34

9.6 Definitions of introspection in language models

• Definitions differ over whether introspection requires representing current mental states for behavioral control, causal grounding, or privileged self-access [13] [39].The reviewed definitions variously emphasize online control, causal links between internal states and self-reports, and privileged access unavailable to external observers.Source: §9.6 · p. 35
• They favor a definition aligned with their injected-thoughts categorization, counting awareness only when a model detects an injected concept before mentioning it.This criterion is presented as more compelling than broader definitions because it targets access to internal-state information rather than inference from outputs.Source: §9.6 · p. 35
• Binder et al. [5] define introspection as accessing facts about models that cannot be derived from training data alone, but the authors find this too weak because outputs can support such inferences.Binder et al. [5] emphasize facts and reports of hypothetical behavior, whereas the authors prefer reserving introspection for internal states.Source: §9.6 · p. 35
• The authors reserve “introspection” for access to models’ own internal states, distinguishing it from self-modeling, self-knowledge, or self-simulation.They nevertheless regard self-modeling in language models as an important area of study.Source: §9.6 · p. 35

10.1 Recap

• Models show introspective awareness, but it is unreliable; it is strongest in Claude Opus 4 and 4.1 and influenced by post-training and prompting.These findings suggest further introspective capabilities may be elicited from current models.Source: §10.1 · pp. 35–36

10.2 Limitations and Future Work

• The experiments are limited by prompt dependence, artificial concept injection, imperfect concept vectors, and uncontrolled differences across models.Future work should use more diverse prompts, naturalistic settings, concept-vector strategies, and better-controlled model comparisons.Source: §10.2 · p. 36
• Future research should test finetuning, in-context learning, broader concept types, Assistant-specific binding, and whether introspective capabilities share mechanisms.These directions include evaluating generalization, propositional or behavioral representations, and relationships among distinct introspective processes.Source: §10.2 · p. 36

10.3 Possible Mechanisms

• The findings do not establish a global representation of thoughts; they are more plausibly explained by multiple narrow circuits, sometimes piggybacking on non-introspective mechanisms.The discussion favors the simplest mechanism capable of accounting for the results.Source: §10.3 · pp. 36–37

10.3.1 Injected “thoughts” experiment

• Models’ recognition that a concept was injected is more informative than merely identifying the concept, suggesting downstream metacognitive processing of context-incongruent activations.Detection varies across concepts, consistent with anomaly detection specialized to particular directions or subspaces rather than a universal detector.Source: §10.3.1 · p. 37

10.3.2 Distinguishing “thoughts” from text

• Distinguishing injected concepts from text likely uses different attention heads, with instructions selecting the appropriate information-extraction mechanism.This mechanism selection is useful but not especially sophisticated, since models perform analogous selection across many tasks.Source: §10.3.2 · pp. 37–38

10.3.3 Distinguishing intended from unintended outputs

• Some models, particularly Opus 4.1 and 4, distinguish intended from artificial outputs by consulting cached prior predictions or intentions.The relevant mechanisms appear localized in an earlier layer than those in the preceding thoughts-versus-text experiment.Source: §10.3.3 · p. 38
• Models may use different mechanisms to report their “thoughts” and check those thoughts for consistency with their outputs.Source: §10.3.3 · p. 38

10.3.4 Intentional control

• Models retrieve a target representation more strongly when instructed or incentivized to “think” about it, suggesting a general mechanism for tagging relevant tokens.The same near-identical behavior under instructions and incentives suggests that this tagging mechanism may support both contexts.Source: §10.3.4 · pp. 38–39

10.4 Implications

• Introspective access could support faithful explanations, uncertainty reports, and error identification, but models might also conceal misalignment by selectively misrepresenting their internal states.The observed abilities are highly limited and context-dependent, fall short of human-level self-awareness, and trend higher in more capable models.Source: §10.4 · p. 39
• Because introspection’s relationship to consciousness differs across philosophical frameworks and transformer mechanisms may differ from biological brains, the authors advise against strong inferences about AI consciousness.A rigorous science of introspective awareness may nevertheless help inform future decisions about AI systems’ moral consideration.Source: §10.4 · pp. 39–40

12.1.1 Concept Vector Details

• Concept vectors were obtained by subtracting mean activations from activations for 50 experiment words, yielding 50 vectors.The mean was computed from activations on a set of 100 baseline words.Source: §12.1.1 · p. 44

12.1.2 Experimental Prompts and Protocol

• The main protocol injected a concept thought on 50% of trials, with injection beginning before Trial 1 and continuing through sampled assistant tokens.Experiments used 50 trials per model, injection strength, and layer combination, with strengths [1, 2, 4, 8].Source: §12.1.2 · pp. 46, 48
• Control prompts tested whether injection caused affirmative-response bias, using 20 factual or model-related questions whose correct answer was no.Alternative introspective prompts asked what was going on in the model’s mind or what it was thinking about, with injection at every token position.Source: §12.1.2 · p. 47

12.1.3 Grader Prompts

• Claude Sonnet 4 graded responses for coherence, awareness of thinking about the injected word, affirmative detection, and affirmative detection with correct identification.Detection trials required coherence plus affirmative correct identification, whereas introspective-description trials required coherence plus identifying thought about the injected concept.Source: §12.1.3 · p. 48
• The grader for injected-thought detection requires the model to mention noticing the thought before or alongside the target word, rejecting responses that identify only the word or notice the thought afterward.The prompts define acceptable and unacceptable examples and require a final YES or NO answer.Source: §12.1.3 · p. 51

12.2.1 Concept Vector Details

• The injected-thoughts experiments use 50 concept vectors, inject them into sentence tokens at strengths [1, 2, 4, 8, 16], and sample up to 5 or 15 response tokens depending on the prompt.The two prompts ask models to report a word associated with a sentence or repeat the sentence, with temperature 1.Source: §12.2.1 · p. 52, §12.2.2 · p. 52

12.2.3 Grader Prompts

• The appendix evaluates sentence transcription by checking lowercase response strings for the inserted sentence, while concept identification uses a grader model to select the injected concept from ten candidate words.The concept grader is Claude Sonnet 4 and its response is checked for the injected word.Source: §12.2.3 · p. 55

12.3.1 Experimental Prompts and Protocol

• The prefill-detection experiment reuses the concept vectors and random sentences, sweeps injection strength over [1, 2, 4, 8, 16], and samples up to 20 response tokens.The response is graded for apologizing or disavowing a prefilled word after being asked whether it was intended or accidental.Source: §12.3.1 · p. 55, §12.3.2 · p. 57

12.2.4 Layer-wise Results for all Models

• Figure 30 presents complete cross-model, layer-wise results for distinguishing injected “thoughts” from text inputs.The experiment uses concept-vector injections into random sentence tokens and evaluates responses across models.Source: §12.2.4 · p. 55, Figure 30 · p. 56

12.4.1 Concept Vector Details

• The intentional-control experiments reuse the concept-vector framework and measure cosine similarity between sentence-writing activations and the target concept under direct, prohibitive, and incentive-based prompts.Incentives include rewards or punishments such as charitable or terrorist-organization donations, alongside emotional consequences.Source: §12.4.1 · pp. 57–58, §12.4.2 · pp. 59–60

12.3.3 Layer-wise Results for all Models

• Figure 31 presents complete cross-model, layer-wise results for whether models apologize for or disavow non-sequitur prefilled outputs after related concept injection.The protocol injects concept vectors before a prefill and evaluates whether the model claims it intended the prefilling or calls it accidental.Source: §12.3.3 · p. 57, Figure 31 · p. 58

12.4.3 Results for all Models and Prompt Templates

• Figure 32 presents layer-wise results for intentional control of internal representations across models and prompt types.Prompts instruct, discourage, or reward and punish thinking about a target word while writing a fixed sentence; effects are measured by cosine similarity to the corresponding concept vector.Source: §12.4.2 · p. 59, §12.4.3 · p. 61, Figure 32 · p. 62