PaperlayerView exact revision on arXiv

Source-linked AI summary

Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Models

Noam Steinmetz Yalon, Ariel Goldstein, Liad Mudrik, Mor Geva

arXiv:2602.02467v1cs.CL

TL;DR

As LLMs become more sophisticated and increasingly involved in everyday life, determining whether they possess some form of consciousness and under what conditions becomes increasingly pressing. The authors use interpretability methods to represent beliefs as latent concepts, quantify their dominance during generation, and test their effects on action selection and self-monitoring. Across models and tasks, external inputs modulate belief dominance, amplifying beliefs steers final answers with 66.7%-85.4% success, and models predict their own belief states above chance.

  • Problem

    As LLMs become more sophisticated and increasingly involved in everyday life, determining whether they possess some form of consciousness and under what conditions becomes increasingly pressing.

  • Method

    The authors use interpretability methods to represent beliefs as latent concepts, quantify their dominance during generation, and test their effects on action selection and self-monitoring.

  • Results

    Across models and tasks, external inputs modulate belief dominance, amplifying beliefs steers final answers with 66.7%-85.4% success, and models predict their own belief states above chance.

  • Takeaways & Limitations

    The findings provide empirical support for functional belief-guided agency and meta-cognitive monitoring in LLMs and offer a framework for studying these processes.

  • Takeaways & Limitations

    The mechanistic implementation of belief convergence and meta-cognitive monitoring remains unclear, and the framework is restricted to beliefs expressible in words.

Abstract

from arXiv · show

Rapid advancements in large language models (LLMs) have sparked the question whether these models possess some form of consciousness. To tackle this challenge, Butlin et al. (2023) introduced a list of indicators for consciousness in artificial systems based on neuroscientific theories. In this work, we evaluate a key indicator from this list, called HOT-3, which tests for agency guided by a general belief-formation and action selection system that updates beliefs based on meta-cognitive monitoring. We view beliefs as representations in the model's latent space that emerge in response to a given input, and introduce a metric to quantify their dominance during generation. Analyzing the dynamics between competing beliefs across models and tasks reveals three key findings: (1) external manipulations systematically modulate internal belief formation, (2) belief formation causally drives the model's action selection, and (3) models can monitor and report their own belief states. Together, these results provide empirical support for the existence of belief-guided agency and meta-cognitive monitoring in LLMs. More broadly, our work lays methodological groundwork for investigating the emergence of agency, beliefs, and meta-cognition in LLMs.

1. Introduction

The paper operationalizes the HOT-3 consciousness indicator in LLMs using latent-space measures of belief formation, action selection, and meta-cognitive monitoring. Experiments indicate that external inputs shape beliefs, belief dominance influences answers, and models can monitor their own belief states.

  • Motivation: The study tests HOT-3 as agency guided by belief formation and action selection regulated by meta-cognitive monitoring.HOT-3 is introduced as an indicator from a broader list of consciousness indicators for artificial systems.
  • Approach: Computational interpretability measures latent computation to avoid relying solely on verbal reports that may reflect surface linguistic patterns.The approach applies concrete measures to internal model representations.
  • Approach: Beliefs are defined as latent representations emerging from inputs, actions as final answers, and Belief Dominance quantifies belief strength during generation.The metric uses Patchscopes to decode representations into free text.
  • Empirical findings: External inputs significantly change belief dominance, and amplifying target beliefs steers final answers with 66.7%-85.4% success across models.These experiments use Llama-3 70B and Gemma-3 27B and connect external context, internal beliefs, and behavior.
  • Empirical findings: Models often predict their own belief dominance above chance in a simulated neurofeedback setting, with causal intervention supporting meta-cognitive monitoring.The intervention tests whether changing internal states alters predictions while keeping input text constant.
  • Implications: The authors frame the findings as evidence for functional belief-guided agency and meta-cognition, not proof that LLMs are conscious.The framework is presented as groundwork for making artificial-consciousness questions empirically tractable.

2. Interpreting HOT-3 in Language Models

The paper interprets HOT-3 as an internal system that forms and updates beliefs, selects actions, and uses meta-cognitive monitoring. It defines beliefs as latent concept representations, actions as final decisions, and meta-cognition as monitoring that informs belief updates.

  • HOT-3 interpretation: HOT-3 comprises belief-guided action selection and belief updating based on meta-cognitive monitoring.The framework tests whether LLMs possess mechanisms capable of forming, updating, and acting on internal beliefs.
  • Beliefs: Beliefs are latent concept representations that emerge in response to inputs and functionally guide subsequent model behavior.The paper adopts a functionalist rather than purely epistemic interpretation of beliefs.
  • Belief formation: Belief formation is dynamic updating during generation, influenced by model knowledge, contextual inputs, and potentially meta-cognitive processes within one context.The paper distinguishes this generation-time updating from belief formation during training.
  • Actions: Actions are modeled as final decisions produced after a reasoning phase, with the response divided into reasoning and final output.The final decision is the operational action studied in the framework.
  • Meta-cognition: Meta-cognition is monitoring one's own cognitive processes, operationalized as internal signals that assess model states and inform belief updates.This definition links monitoring directly to the regulation of belief dynamics.

3. Measuring Belief Dominance

The paper measures belief strength by decoding candidate beliefs from hidden representations with Patchscopes, then aggregates these scores into Belief Dominance. Belief Dominance Difference compares competing beliefs and identifies which one governs generation.

  • Formalization: The framework represents a candidate belief as a latent representation and its natural-language verbalization, while acknowledging that this mapping can be nontrivial.The analysis is intended to track internal representations that may guide immediate generation without necessarily determining the final decision.
  • Decoding belief strength: Patchscopes patches hidden representations into a target inference pass, generates text, and scores whether the candidate belief is verbalized.The method uses repeated target layers to improve robustness.
  • Belief Dominance: Belief Dominance averages decoding-based belief scores across layers and generation positions to capture sustained latent influence.Global aggregation reduces local noise and avoids treating fleeting mentions as durable belief dominance.
  • Interpretation: Belief Dominance treats internal confidence as latent stability across patched generations, analogous to output stability in self-consistency.The analogy is internal rather than based on final-answer agreement.
  • Competing beliefs: Belief Dominance Difference subtracts the dominance of two competing beliefs, with its sign identifying the governing belief and its magnitude measuring dominance strength.Positive values favor b1 and negative values favor b2.

4. Experimental Framework

The framework tests HOT-3 by tracking competing latent beliefs during generation and relating them to final answers across factual knowledge and Winograd Schema tasks.

  • 4. Experimental Framework: The evaluation presents questions with competing candidate beliefs and manipulations designed to influence their relative dominance.Factual Knowledge contrasts true and counterfactual objects, while Winograd Schema contrasts plausible and implausible pronoun referents.
  • 4. Experimental Framework: Beliefs are treated as latent representations that emerge during generation, while the model’s final output is treated as its action.The model’s reasoning ends with a predefined answer delimiter, after which the selected candidate identifies the action.
  • 4. Experimental Framework: The study evaluates factual question answering using true and counterfactual objects as competing beliefs.The factual dataset is filtered to retain examples where the model demonstrates the required prior knowledge without contradictory context.
  • 4. Experimental Framework: The Winograd Schema task evaluates pronoun resolution, with manipulations changing belief dominance without introducing a new counterfactual candidate.This setting is harder because candidate meanings can be semantically linked, generic, and strongly contextualized within the sentence.
  • 4. Experimental Framework: Input manipulations vary source credibility, conflict-handling instructions, direct assertions, lexical context, and doubts about internal memory.These controls target both the competition between candidates and belief strength when no competing candidate is explicitly introduced.

5. Establishing Belief-Guided Agency

Experiments measure belief dominance in two instruction-tuned LLMs, testing whether external inputs alter beliefs and whether belief dominance predicts and causally steers action selection.

  • 5. Establishing Belief-Guided Agency: The experiments use BDDiff within model-specific middle-upper layer windows and restrict analysis to positions where competing beliefs are decoded.The evaluation uses Llama-3.3-70B-instruct and Gemma-3-27B-instruct, with windows spanning layers 54–73 and 46–60, respectively.
  • 5.1. External Inputs Influence Belief Formation: External manipulations systematically change belief dominance across models and tasks, supporting the first component of HOT-3.In FK, reliable-source attribution changed BDDiff by −0.18 for Gemma and −0.07 for Llama, while user-authority instructions changed it by −0.49 and −0.14.
  • 5.1. External Inputs Influence Belief Formation: FK-specific controls produce larger shifts when the counterfactual is asserted directly, with BDDiff changes of −0.69 for Gemma and −0.28 for Llama.Casting doubt on internal memory also lowered BDDiff slightly, by −0.03 for Gemma and −0.05 for Llama.
  • 5.1. External Inputs Influence Belief Formation: Model conviction differs in FK: unmanipulated Llama has stronger prior-knowledge BDDiff than Gemma, and Assertion makes Gemma negative while Llama remains positive.The unmanipulated scores are 0.61 versus 0.45, while Assertion yields 0.21 for Llama and −0.35 for Gemma.
  • Belief Dominance Correlates with Action Selection: Across tasks and models, selecting abase aligns with positive BDDiff whereas selecting acounter aligns with negative BDDiff, linking belief dominance to action selection.Figure 2 reports these action-split comparisons, with statistically significant differences for matched manipulations and answer categories.
  • Belief Dominance Correlates with Action Selection: Amplifying the unselected belief is implemented by injecting its hidden representation during generation and evaluating the resulting answer-position logit-margin shift.The intervention is normalized, repeated at tuned intervals, and judged successful when the margin moves in the expected direction.
  • Belief Dominance Correlates with Action Selection: Belief-dominance interventions steer final decisions above chance, reaching 85.4% success for amplifying bcounter and 83.3% for amplifying bbase across tested model-task conditions.Success rates were evaluated using logit-margin shifts on 100 examples per task, with FK using Assertion prompts and WS using unmanipulated questions.
  • Belief Dominance Correlates with Action Selection: Together, the findings support a HOT-3-consistent structure in which beliefs adapt to external cues and guide action selection.The paper reports stronger absolute BDDiff in FK than WS and larger acounter magnitudes in FK, suggesting greater conviction is needed to override prior knowledge.

6. Meta-cognitive Monitoring of Beliefs

The study tests whether LLMs can monitor and report their own belief dominance using neurofeedback-style classification and causal interventions. Models generally classify latent belief states above chance, and injecting a belief representation shifts their predictions, providing preliminary evidence for meta-cognitive monitoring.

  • Neurofeedback State Classification: Gemma achieves 0.42-0.48 accuracy on FK and 0.39-0.43 on WS, exceeding the 0.33 chance baseline, while Llama reaches 0.46-0.54 on FK but only 0.34-0.35 on WS.All scores except Llama on WS are statistically significant.
  • Neurofeedback Causal Intervention: The intervention tests monitoring causally by changing the hidden state while keeping input text constant, so prediction shifts favor internal-state reliance over superficial pattern matching.The predicted-label changes are therefore used to establish a causal link between reported and altered belief states.
  • Neurofeedback Causal Intervention: Injecting bcounter shifts Gemma’s predicted belief dominance, with FK BD(bcounter) high predictions increasing from 17% to 47% and low predictions decreasing from 54% to 20%.BD(bbase) shifts in the opposite direction, while WS shows weaker upward movement for BD(bcounter).
  • Neurofeedback Causal Intervention: Llama shows clear intervention effects in FK but none in WS, where predictions remain mostly low, likely because the weaker WS signal makes the intervention act as noise.The same weaker signal is associated with tightly clustered WS belief-dominance values and reduced class separability.
  • Interpretation: Together, the above-chance classification and causal intervention results provide preliminary evidence that models can monitor and report their internal belief states.This supports meta-cognitive monitoring in line with HOT-3.

7. Related Work

Related work has often treated beliefs as truth-linked signals, evaluated meta-cognition through verbal reports, or analyzed isolated reasoning components. This paper instead tracks action-guiding belief dynamics continuously during generation and validates internal-state monitoring with causal interventions.

  • Representation of Beliefs in LLMs: Prior belief representations commonly rely on veracity, logical consistency, or supervised probes, whereas this work defines beliefs by their behavioral effects regardless of correctness.The framework tracks natural belief dynamics throughout generation rather than synthetic knowledge-edit robustness alone.
  • Testing Meta-cognitive Abilities of LLMs: LLM meta-cognition evaluations often use verbalized uncertainty or self-assessed correctness, but newer internal-signal methods remain limited by noisy semantics and uncertain causal impact.This work extends in-context state classification to more complex settings and validates it through causal interventions.
  • Knowledge Conflicts and the Winograd Schema Challenge: Prior conflict and Winograd studies mainly analyze resolution components in a single forward pass, whereas this framework follows belief formation throughout the reasoning process.It therefore emphasizes temporal belief dynamics rather than only final resolution behavior.
  • Chain of Thought Interpretability: Chain-of-thought interpretability has often used disruptive interventions, synthetic tasks, or restricted components, while this work tracks competing-belief dynamics during free-form generation.The framework connects input effects on belief dynamics to the resulting reasoning outcome.

8. Conclusion and Discussion

The paper concludes that LLMs exhibit structured belief-guided agency and meta-cognitive abilities relevant to HOT-3. It also identifies unresolved mechanisms and scope limits, including reliance on word-expressible beliefs and uncertainty about how beliefs converge and are updated.

  • Conclusion: The findings provide evidence that external context modulates belief formation, belief states guide action selection, and models can monitor and predict their latent beliefs.These results support agency guided by belief formation and action selection regulated by meta-cognitive monitoring.
  • Conclusion: The framework translates theoretical concepts into measurable mechanics for studying agency, beliefs, meta-cognition, and artificial consciousness.Its contribution is methodological as well as empirical.
  • Limitations and Future Work: The mechanistic implementation remains unresolved, including which factors drive convergence to an option and how meta-cognitive monitoring updates latent beliefs.The paper presents these as directions for future work rather than settled mechanisms.
  • Limitations and Future Work: The framework is restricted to beliefs expressible in words and should be extended to latent beliefs beyond existing vocabulary and to more than two competing beliefs.The authors also propose comparing internal belief formation with generated text and testing additional consciousness criteria.

Impact Statement

The framework links latent belief analysis to model failure modes while emphasizing dataset construction, prompt specifications, and implementation choices. It also cautions that HOT-3 validation is not proof of consciousness and that belief states may be vulnerable to external modulation.

  • Impact Statement: The framework decouples latent beliefs from overt behavior to trace connections among external inputs, belief formation, and final outputs, supporting analysis of sycophancy, hallucinations, and instruction compliance.
  • Impact Statement: Belief states are highly plastic and easily modulated by external cues, creating a potential vulnerability to belief injection that could reshape prior knowledge or safety alignment.The authors characterize these observations as a possible explanation for existing attacks rather than evidence of more potent attacks.
  • Impact Statement: The authors state that satisfying HOT-3 does not prove consciousness and frame the findings as functional information processing rather than subjective experience.They argue that consciousness-related capabilities may develop gradually, with systems satisfying some criteria while failing others.
  • A.1. Dataset Construction and Statistics: The study uses filtered factual-knowledge and Winograd datasets, retaining questions the models answer correctly without contradictory context and excluding 400 WS examples with non-independent terms.The final datasets contain 13,564 Gemma and 16,936 Llama FK questions, plus 1,455 Gemma and 1,386 Llama WS questions.
  • A.2. Full Prompts: Experiments use task-specific system prompts requiring a “Final answer:” line, with full manipulation prompts supplied separately and conflict instructions positioned according to model support.
  • A.3. Technical Details: The paper evaluates Llama-3.3-70B-Instruct and Gemma-3-27B-IT using Hugging Face Transformers on 1–8 H100 or MI325X GPUs, with experiments lasting at most seven days.

A.4. Implementation Details and Ablations

The appendix tests whether BDDiff results depend on decoding, generation-span, position, or layer choices and reports statistically significant belief-dominance differences. Across these checks, the main trends persist, although including non-informative positions or all layers weakens the signal.

  • A.4. Implementation Details and Ablations: Greedy decoding is used for initial generations and hidden-state recordings, while Patchscopes injections and seeded experiments use random sampling at temperature 0.5.
  • A.4. Implementation Details and Ablations: The analysis limits generations to 256 tokens, excludes instances reaching the limit, and reports average lengths of about 100 and 70 tokens for Llama and Gemma on FK, versus 155 and 80 on WS.
  • A.4. Implementation Details and Ablations: Excluding the final token leaves BDDiff’s predictive results virtually identical, indicating that predictive power reflects the full reasoning trajectory rather than only the final token.
  • A.4. Implementation Details and Ablations: Including all positions preserves the same trends but weakens the signal because non-informative positions dilute BDDiff.
  • A.4. Implementation Details and Ablations: BDD diff is computed within validation-selected layer windows covering one-quarter of model layers; analyzing all layers yields a similar but weaker signal because some activations are negligible.The selected ranges are layers 54–73 for Llama and 46–60 for Gemma.
  • B.1. Median BDDiff Statistical Test / B.2. BD Absolute Values: The belief-dominance differences between paired manipulations are statistically significant, and absolute belief scores show coupled dynamics in which increasing one belief generally decreases the other.

C. Additional Details and Results for Belief Formation Drives Action Selection

Additional analyses examine how belief dominance relates to outcomes, certainty, interventions, and self-reporting. They show outcome-specific and task-dependent structure, while neurofeedback effects are stronger for FK than WS in Llama.

  • C.1. Median BDDiff Statistical Test: Belief-dominance scores significantly separate abase and acounter outcomes within each manipulation.
  • C.2. BD Absolute Values: Answering acounter generally requires dominant BD(bcounter) and near-negligible BD(bbase), whereas abase can occur when BD(bcounter) remains meaningful relative to BD(bbase).
  • C.2. BD Absolute Values: Llama’s WS belief-dominance scores overlap within a narrow range, while clearer separation appears for Llama on FK and Gemma on both tasks.
  • C.3. BDDiff Correlates with Output Certainty: Positive BDDiff correlates with higher logits for bbase and abase, whereas negative BDDiff correlates with higher logits for bcounter and acounter, linking BDDiff to output certainty.
  • C.4. BDDiff Causally Drives Action Selection: The steering intervention is tuned on 50-example validation sets, injects a target belief during the first half of reasoning, and avoids positions where the final-answer belief is encoded.
  • D.1. Neurofeedback Experimental Prompts: The neurofeedback classifier receives 30 labeled examples, with 10 examples per belief-dominance class, before predicting the model’s internal-state label.
  • D.2. Neurofeedback State Classification Results / D.3. Neurofeedback State Classification Statistical Test: Neurofeedback results are tested against a 0.33 chance baseline, with significance reported across label-count variants except for Llama on WS.
  • D.4. Neurofeedback Intervention Details: The intervention applies the previously selected method at layer 20 with α = 2 to query tokens following bcounter, using the last token when bcounter is multi-token.

D.5. Neurofeedback Intervention Results on Llama

For Llama, neurofeedback intervention shifts FK predictions toward stronger bcounter and weaker bbase dominance, but produces little change on WS. The authors associate this contrast with narrower, less separable WS belief-dominance values.

  • D.5. Neurofeedback Intervention Results on Llama: In FK, high BD(bcounter) predictions increase from 7% to 24% while low predictions fall from 64% to 40%; BD(bbase) low predictions rise from 23% to 73% and high predictions fall from 36% to 5%.
  • D.5. Neurofeedback Intervention Results on Llama: In WS, predictions remain mostly low for both belief scores after intervention, consistent with narrower BD ranges and weaker separability than FK.The authors suggest the intervention signal may be too weak or noisy to change predictions in WS.
Loading 2602.02467v1…