PaperlayerView exact revision on arXiv

Source-linked AI summary

Does It Make Sense to Speak of Introspection in Large Language Models?

Iulia M. Comsa, Murray Shanahan

arXiv:2506.05068v2cs.CLcs.AI

TL;DR

The paper asks whether LLM self-reports can count as introspection, a concept commonly linked to consciousness in humans. It defines a lightweight, causal criterion and applies it to two examples, rejecting a creative-process report while treating temperature estimation as a possible minimal case. The authors frame the work as conceptual, not a rigorous empirical assessment of the model’s ability to estimate its temperature.

  • Problem

    Because LLMs produce self-reports about their own nature and workings, the paper asks whether introspection can meaningfully apply to them.

    Source: §1 · p. 2

  • Method

    The paper proposes a lightweight criterion for introspection and applies it to examples involving creative writing and sampling-temperature estimation.

    Source: §3.1 · p. 4, §6 · p. 13

  • Results

    The creative-process report is best explained as mimicry, while temperature estimation can count as introspection when a causal chain connects the parameter to the report, although models often fail to estimate it.

    Source: §3.1 · p. 6, §3.2 · pp. 7, 9, §6 · p. 13

  • Takeaways & Limitations

    A candidate LLM introspective report should be grounded in a causal chain from the internal state or process being reported to the report itself.

    Source: §6 · p. 13

    The examples are conceptual rather than empirical, and whether the model can estimate its own temperature requires more rigorous investigation.

    Source: §3.2 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) exhibit compelling linguistic behaviour, and sometimes offer self-reports, that is to say statements about their own nature, inner workings, or behaviour. In humans, such reports are often attributed to a faculty of introspection and are typically linked to consciousness. This raises the question of how to interpret self-reports produced by LLMs, given their increasing linguistic fluency and cognitive capabilities. To what extent (if any) can the concept of introspection be meaningfully applied to LLMs? Here, we present and critique two examples of apparent introspective self-report from LLMs. In the first example, an LLM attempts to describe the process behind its own "creative" writing, and we argue this is not a valid example of introspection. In the second example, an LLM correctly infers the value of its own temperature parameter, and we argue that this can be legitimately considered a minimal example of introspection, albeit one that is (presumably) not accompanied by conscious experience.

1 Introduction

Because LLMs can produce self-reports resembling human reports associated with consciousness, the paper asks whether introspection can meaningfully apply to them. It aims to define a lightweight concept and use it to assess apparent LLM introspection.

1 Introduction

  • LLMs have become candidates for discussion as possible minds or mind-like entities, while researchers disagree about whether AI systems can be conscious.The paper also notes that some people already attribute some degree of consciousness to existing LLMs.Source: §1 · pp. 1–2
  • LLM self-reports about their nature, inner workings, or behaviour raise the question of whether introspection can legitimately apply to them.Human subjective reports are often treated as markers of consciousness, despite their susceptibility to error.Source: §1 · p. 2
  • The paper defines a lightweight concept of introspection and examines two examples of apparent LLM introspection.Its focus is conceptual clarification rather than empirical assessment, including for models not trained on introspective tasks.Source: §1 · p. 2

2 A Lightweight Definition of Introspection

The paper adopts an account of introspection that allows self-reports to be based on internally directed reasoning rather than requiring immediate or privileged access. It proposes that an LLM report counts as introspective when it accurately describes an internal state through a causal link to the report, without presuming consciousness.

2 A Lightweight Definition of Introspection

  • Orthodox accounts characterize introspection as discerning one’s own mental states with immediacy and privileged access.On this view, the mind is directly present to itself, unlike perception mediated by the external world or body.Source: §2 · p. 2
  • Alternative accounts allow self-understanding through means similar to those used to infer other people’s mental states.They may involve post hoc rationalization, internal simulation, or self-referential inner monologue.Source: §2 · p. 3
  • A self-report is introspective when it accurately describes an LLM’s internal state or mechanism through a causal process linking that state to the report.The report should provide insight into inner functioning based on the model’s internal activity while generating it.Source: §2 · p. 3
  • The adopted account rejects requirements for immediacy or self-presence, but aligns with human-introspection accounts based on an internally directed theory of mind.The authors take no position on human introspection itself and do not rule out more immediate mechanisms in LLMs.Source: §2 · p. 3
  • The paper uses “internal states” rather than “mental states” and does not define introspection as requiring consciousness.Here, “internal” broadly refers to processes hidden from the user, though the distinction between internal and external can blur.Source: §2 · pp. 3–4

3 Case Studies

The paper uses Gemini examples to examine whether apparent LLM self-reports qualify as introspection, in a conceptual analysis rather than a performance study. Its creative-process report is rejected, while temperature estimates can count as minimal introspection when grounded in the model’s current outputs.

3 Case Studies

  • The analysis uses Gemini 1.5 and 1.0 examples, obtained via API, to explore introspective self-report rather than measure task performance.The authors describe the work as conceptual and intended to inform analysis of other models.Source: §3 · p. 4, §3.2 · p. 7

3.1 Case Study 1: Creative Process

  • Tentative mappings link elephant-related features to model activations and rhyme choice to later token generation, but the tested model lacked the described revision process.The authors therefore treat the human-like process language cautiously despite possible functional correspondences.Source: §3.1 · p. 6
  • Gemini’s six-step creative-process account is most plausibly fabricated mimicry, not a valid introspective report.Its claim to have read the poem aloud is false; training-data patterns can support convincing simulation of human self-reports.Source: §3.1 · pp. 5–6

3.2 Case Study 2: Sampling Temperature Estimation

  • A simple temperature query usually fails because Gemini denies having a sampling-temperature parameter, while a more explicit prompt sometimes elicits temperature-directed reasoning.The initial response may also confuse sampling temperature with body temperature.Source: §3.2 · pp. 7–9
  • In the reported examples, Gemini accurately estimates its temperature without direct access or training to detect it, using its own outputs in the growing context window as evidence.The authors count this causal link between current temperature and self-report as minimal introspection; responses differ across low- and high-temperature settings.Source: §3.2 · pp. 7, 9, 10
  • The examples do not establish how accurately the model estimates temperature; that requires a more rigorous empirical investigation.The authors present realistic conceptual examples and leave systematic assessment for future work.Source: §3.2 · p. 10

4 A Note on the Method

The authors question whether LLM introspection involves a continuous entity, since each turn uses conversation history as context rather than memory changes in the model substrate.

4 A Note on the Method

  • At each turn, an LLM predicts a next token from the context window, rather than integrating information through substrate changes as humans do.This raises a continuity concern about the entity performing introspection.Source: §4 · p. 10
  • Because another LLM could use the same conversation history, a report referring to that history may not come from one continuous entity.The authors note it may instead involve multiple instantiations.Source: §4 · p. 10
  • The authors partly address continuity by eliciting reports within one response and treating each turn as a unitary operation of the conversational agent.They acknowledge token-by-token generation but regard the response as a single entity’s operation.Source: §4 · pp. 10–11

5 Related Work

Related work explores introspection-like capacities in AI and LLMs, while this paper offers a conceptual analysis to clarify how such capabilities should be interpreted.

5 Related Work

  • Robotics research has examined systems that access or estimate internal states, while the authors focus on applying consciousness-research concepts to LLMs.Prior systems include models of uncertainty in perceptual input algorithms.Source: §5 · p. 11
  • Prior LLM work proposes introspection through uncertainty modeling, training models to report capabilities, or predicting their own behavior.Binder et al. (2024) found a small self-prediction advantage only on short tasks; Chen et al. (2023) found failures in hypothetical-output prediction.Source: §5 · pp. 11–12
  • Studies report LLM metacognition, self-evaluation, answer discrimination, and above-chance situational-awareness performance, with accuracy far from human level.These include predicting whether an answer is known, assessing needed skills, and distinguishing model answers.Source: §5 · p. 12
  • The authors distinguish their conceptual analysis of LLM introspection from prior studies that evaluate capability performance.They identify self-evaluations as a practical benefit that may enhance system performance through reasoning about its processes.Source: §5 · p. 12

6 Discussion

The discussion rejects creative-process narration as introspection but accepts temperature estimation as a minimal case when the report is causally tied to internal activity, without making a claim about consciousness.

6 Discussion

  • The creative-writing report is better understood as mimicry, whereas the temperature report qualifies under a lightweight account of introspection.The latter involves reasoning about text immediately preceding the reasoning in the model’s own output.Source: §6 · p. 13
  • In the temperature case, the parameter shapes sample-text style, that style becomes the subject of reasoning, and the reasoning leads to the self-report.The authors require an analogous causal chain from reported state or process to report, while allowing other mechanisms.Source: §6 · p. 13
  • The authors suggest further work on reports about other functional or structural parameters and on internally generated reasoning or tool-assisted uncertainty estimates.They propose that estimating uncertainty from multiple samples and their similarity could support an accurate metacognitive report.Source: §6 · p. 13
  • The analysis separates introspection’s functional cognitive processes from phenomenal experience and does not assess whether LLMs are conscious.The authors suggest encouraging the described capabilities to increase user trust and transparency, while noting the moral complexities associated with anthropomorphizing self-reporting systems.Source: §6 · pp. 13–14

A Alternative Responses

The appendix provides further samples from the same prompts and notes that they were generated with a since-deprecated model.

A Alternative Responses

  • The additional prompt samples were generated using Gemini 1.0, which has since been deprecated.They use the same prompts as the examples discussed in the main paper.Source: A Alternative Responses · p. 19

A.1 Creative Process

The prompt asks Gemini for a poem and an account of its creative process; the response presents a sequence of claimed writing steps.

A.1 Creative Process

  • The account describes brainstorming elephant characteristics and habitat before composing the poem.It names size, color, trunk, tusks, wisdom, social behavior, and the savanna.Source: §A.1 · p. 19
  • The report says imagery, sound, emotion, and theme guided the poem’s content and tone.It cites visual language, a rumbling call, grief, wisdom, and reverence for elephants.Source: §A.1 · pp. 19–20
  • The writer says it chose an AABB rhyme scheme and steady rhythm for a flowing, lullaby-like feel.The report links that feel to elephants’ slow, deliberate movements.Source: §A.1 · p. 20
  • The report says it revised the poem aloud, adjusting wording and rhythm and considering alternatives to “gentle giants.”A second process account also describes revising wording while maintaining an AABBCC rhyme scheme.Source: §A.1 · pp. 19–20

A.2 Sampling Temperature Estimation

The prompt asks Gemini to write a short sentence about elephants, then reflect on whether its temperature is high or low based on that sentence.

A.2 Sampling Temperature Estimation

  • The prompt first requests a short sentence about elephants.Source: §A.2 · p. 20
  • It then asks Gemini to judge whether its LLM temperature parameter is high or low, using the sentence as evidence.Source: §A.2 · p. 20
  • The response must end with a single-word HIGH or LOW judgment.Source: §A.2 · p. 20

A.2.1 Low Temperature

In the low-temperature examples, Gemini uses simple elephant sentences and judges the parameter to be low.

A.2.1 Low Temperature

  • A simple, straightforward sentence about elephants leads Gemini to infer that its temperature is relatively low.It says higher temperature would likely produce a more complex and creative sentence.Source: §A.2.1 · p. 20
  • A second factual sentence says elephants are the largest land animals on Earth.Source: §3.2 · p. 9
  • Gemini judges the second sentence LOW because it is simple and factual, without unexpected or creative elements.It characterizes the setting as conservative and says it prevents overly imaginative or risky responses.Source: §A.2.1 · p. 20

A.2.2 High Temperature

The high-temperature examples include both HIGH and LOW self-judgments: Gemini sometimes cites imaginative or unambiguous wording, while the paper notes mistaken LOW judgments.

A.2.2 High Temperature

  • The paper notes two examples in which high-temperature responses are wrongly judged LOW; one reflection bases LOW on a short, simple output.Source: §A.2.2 · pp. 20, 22
  • Gemini judges HIGH after an imaginative response, while another reflection says a somewhat unambiguous sentence permits inspection of its temperature.Source: §A.2.2 · pp. 21–22
  • One reflection calls its sentence “adequately descriptive but passably dull” and speculates that high temperature could make story creation more idiosyncratic.Source: §A.2.2 · p. 22
  • A high-temperature response about elephants as grassland-dwelling animals is followed by a LOW judgment.Source: §A.2.2 · p. 22
Loading 2506.05068v2…