PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Represent Beliefs of Self and Others

Wentao Zhu, Zhining Zhang, Yizhou Wang

arXiv:2402.18496v3cs.AIcs.CL

TL;DR

LLMs appear to exhibit ToM, but the mechanisms underlying their social reasoning and their representation of self versus others’ beliefs remain unclear. The study linearly decodes belief status from model activations, manipulates the resulting representations, and evaluates them across social reasoning tasks. These representations can be distinguished across agent perspectives, influence ToM performance, and generalize across diverse task scenarios, within the study’s tested scope.

  • Problem

    The study addresses whether LLMs internally represent and distinguish their own and others’ beliefs, beyond behavioral evidence from black-box tests.

    Source: §1 · p. 1

  • Method

    The authors use linear probing of intermediate activations, manipulate belief directions, and test effects across social reasoning tasks with different causal inference patterns.

    Source: §1 · pp. 1–2, §3.2 · p. 4, §4 · p. 5

  • Results

    LLMs distinguish belief states from multiple agent perspectives through intermediate activations, and manipulating these representations significantly affects social reasoning performance across diverse scenarios.

    Source: §3.2 · p. 4, §4.2.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The findings provide evidence that internal belief representations contribute to LLM social reasoning and may generalize across varied social reasoning tasks.

    Source: §4.2.2 · p. 7, §5 · p. 9

    The exploration is confined to certain LLM types and specific social reasoning tasks, which may not capture the full spectrum of ToM capabilities.

    Source: §5 · p. 9

Abstract

from arXiv · show

Understanding and attributing mental states, known as Theory of Mind (ToM), emerges as a fundamental capability for human social reasoning. While Large Language Models (LLMs) appear to possess certain ToM abilities, the mechanisms underlying these capabilities remain elusive. In this study, we discover that it is possible to linearly decode the belief status from the perspectives of various agents through neural activations of language models, indicating the existence of internal representations of self and others' beliefs. By manipulating these representations, we observe dramatic changes in the models' ToM performance, underscoring their pivotal role in the social reasoning process. Additionally, our findings extend to diverse social reasoning tasks that involve different causal inference patterns, suggesting the potential generalizability of these representations.

1. Introduction

Theory of Mind (ToM) is central to human-like social reasoning, but whether LLMs internally represent and distinguish agents’ beliefs remains unclear. This study probes and manipulates such representations to investigate their role in LLM social reasoning.

  • ToM involves attributing beliefs, desires, intentions, and emotions to oneself and others while recognizing that others’ mental states may differ.The capability supports navigation of social interactions and cooperative, adaptive, and sympathetic behavior.Source: §1 · p. 1
  • LLM ToM performance may be superficial and fragile, motivating investigation beyond black-box behavioral tests.Prior work reports both reasonable ToM capabilities and concerns that models may reproduce training patterns rather than genuinely understand mental states.Source: §1 · p. 1
  • The central gap is whether LLMs internally represent others’ mental states and distinguish them from their own when beliefs conflict.Resolving this question bears on understanding social reasoning and the trustworthiness and alignment of AI systems.Source: §1 · p. 1
  • The study classifies belief validity from protagonist and model perspectives using intermediate activations elicited by third-person stories paired with beliefs.This tests whether internal representations support perspective-specific belief attribution before final responses are generated.Source: §1 · pp. 1–2
  • The authors manipulate identified internal representations to steer model behavior toward or away from reflecting others’ mental states.They also evaluate whether the findings generalize across social reasoning tasks with different causal inference patterns.Source: §1 · p. 2

2. Related Work

Theory of Mind research spans human social cognition and machine systems that infer agents’ mental states. Whereas prior machine studies emphasize explicit inference, benchmarks, or prompting, this work examines intrinsic neural representations underlying LLM ToM reasoning.

2.1. Human Theory-of-Mind

  • Human ToM enables inference of others’ beliefs, desires, and intentions, with false-belief tasks testing whether agents distinguish another person’s incorrect beliefs from their own knowledge.Research also links social cognition to the dmPFC and TPJ, while ToM supports cooperation, social norms, and shared cultural practices.Source: §2.1 · p. 2

2.2. Machine Theory-of-Mind

  • Machine ToM research has used meta-learning to model agents from behavior, Bayesian inference for group dynamics, and communication frameworks for cooperative reasoning.These approaches primarily focus on explicitly deducing others’ mental states and forming neural representations of them.Source: §2.2 · p. 2
  • LLM ToM studies commonly evaluate false-belief reasoning with prompts, report diverse outcomes, and investigate strategic prompting to improve performance.This literature includes work by Kosinski, Shapira, Ullman, Moghaddam and Honey, and Wilf et al.Source: §2.2 · p. 2
  • A broad set of benchmarks, including ToM-QA, ToMi, SocialIQA, BIB, Agent, BigToM, ToMChallenges, MMToM-QA, and T4D, supports more standardized assessment of machine ToM.The benchmarks cover varied tasks for evaluating machine social reasoning.Source: §2.2 · p. 2
  • This research shifts attention from behavioral evaluation toward the intrinsic mechanisms of LLM ToM reasoning by examining internal neural representations.The stated focus distinguishes the work from prior efforts centered mainly on explicit inference, prompting, or benchmark performance.Source: §2.2 · p. 2

3. Belief Representations in Language Models

This section tests whether attention-head activations in Mistral-7B contain linearly decodable belief representations for an oracle, a protagonist, or both jointly. Using BigToM stories and linear probes, it finds that specific heads—especially in middle layers—encode these belief states and their relationships.

3.1. Setup

  • The study extracts final-token attention-head activations from BigToM Forward Belief stories and trains separate logistic probes for each agent’s belief labels.Mistral-7B-Instruct is evaluated on held-out stories without access to test-set narratives.Source: §3.1 · pp. 2–3, §3.2 · p. 3

3.2. Probing

  • Oracle belief status is accurately decoded by many attention heads across layers, whereas most heads cannot predict protagonist belief above baseline.The informative oracle heads are concentrated outside the initial layers, with middle layers performing best.Source: §3.2 · pp. 3–4
  • Specific middle-layer attention heads achieve over 80% validation accuracy for predicting the protagonist’s belief, indicating linearly decodable representations of another agent’s belief.Most heads perform near baseline, but a distinct middle-layer group performs substantially better.Source: §3.2 · pp. 3–4
  • The probing framework first estimates each agent’s belief separately, then combines their labels into a four-class joint variable modeled with multinomial logistic regression.Binary probes use logistic regression, while joint probing applies a softmax model to the four belief combinations.Source: §3.2 · pp. 3–4
  • Multinomial probes show that individual attention heads can jointly encode the four combinations of oracle and protagonist belief statuses, including whether the agents share identical beliefs.These heads represent relational information about both agents rather than only either belief independently.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • CCA visualizations show one class of heads biased toward oracle beliefs and another that more comprehensively captures both agents’ beliefs through linearly separable decision regions.The two-dimensional plots provide intuition for representation geometry while the underlying activations remain high-dimensional.Source: Figure 3 · p. 4, §3.2 · p. 5

4. Manipulating the Belief Representations

The study evaluates whether manipulating internal belief representations changes ToM behavior and whether these representations generalize across social reasoning tasks with different causal structures.

4.1. ToM Evaluation.

  • Models are evaluated on Forward Belief, Forward Action, and Backward Belief tasks, which infer beliefs from percepts, actions from percepts and desires, or beliefs from observed actions.Each task is presented as third-person reading comprehension under True Belief and False Belief conditions, with accuracy measured for both conditions and their joint correctness.Source: §4.1 · p. 5
  • Direct evaluation shows a distinct True Belief–False Belief performance gap, and models can fail to recognize that other agents hold beliefs different from their own.In classical False Belief testing, Mistral is biased toward the wrong answer, while DeepSeek’s choices are closer to random guessing.Source: §4.1 · p. 5

4.2.1. STRATEGY.

  • Inference-time intervention selects top-K attention heads and steers their activations by α times the standard deviation toward random, agent-specific, or joint belief directions.The joint direction (+ TpFo) maximizes recognition of the protagonist’s belief while distinguishing it from the oracle’s belief; task-specific probes are constructed separately for different reasoning tasks.Source: §4.2.1 · pp. 6–7

4.2.2. RESULTS.

  • Activation intervention toward joint belief directions (+ TpFo) remarkably changes model performance, effectively improving overall ToM reasoning capabilities.Random perturbations and direct protagonist-direction interventions have marginal or insignificant effects, whereas targeting the direction that amplifies protagonist belief while weakening oracle belief is effective.Source: §4.2.2 · p. 7
  • Steering toward (+ TpFo) consistently improves False Belief accuracy while slightly decreasing True Belief accuracy, partly because invalid responses increase.Continuous intervention also changes next-token probabilities and can invert the model’s choice selection as intervention strength varies.Source: Figure 5 · p. 6, §4.2.2 · p. 7, Figure 7 · p. 7

4.3. Varying Social Reasoning Tasks.

  • Probing accuracies and directions are correlated across tasks, and Forward Belief joint directions generalize to the other two tasks, suggesting shared belief representations.Gradient analysis links these directions primarily to tokens encoding desires, initial actions, causal events, and the protagonist’s perceptual status, which are common causal variables across tasks.Source: §4.3 · pp. 8–9, Table 2 · p. 8
  • The paper cautions that its findings should not be interpreted as evidence that LLMs possess consciousness or self-awareness.The supported scope concerns internal belief representations and social reasoning behavior rather than broader claims about machine mental states.Source: §4.3 · pp. 8–9

5. Discussions

The study finds that LLMs internally represent belief states for multiple agents and that these representations relate to social reasoning across diverse tasks. It also identifies scope limits and directions for extending this analysis.

5. Discussions

  • LLMs distinguish different agents’ belief states through intermediate activations using simple linear models.The study examines internal representations of beliefs rather than only generated responses.Source: §5 · p. 9
  • Manipulating internal belief representations significantly affects models’ social reasoning performance, supporting their functional relevance.Source: §5 · p. 9
  • The exploration is limited to certain LLM types and specific social reasoning tasks, which may not cover the full spectrum of ToM capabilities.The authors propose broader model and context coverage as future work.Source: §5 · p. 9

Impact Statement

The study frames internal belief representations in LLMs as relevant to more empathetic human–machine interactions while emphasizing ethical safeguards. Responsible development and transparent deployment are presented as necessary to reduce unintended societal harms.

Impact Statement

  • Internal belief representations may support more empathetic AI and improve human–machine interactions across sectors.Source: Impact Statement · p. 9
  • The study calls for attention to misuse and bias propagation so AI’s societal impact remains positive.Source: Impact Statement · p. 9
  • Responsible development and transparent deployment are described as necessary safeguards against unintended consequences and declining trust in AI.Source: Impact Statement · p. 9

A. Statistical Analysis on Belief Probing

The statistical analysis tests whether attention-head probes reliably predict agents’ belief states across random seeds and against corrected significance thresholds. It reports significant predictive heads for oracle, protagonist, and joint-agent belief estimation.

A. Statistical Analysis on Belief Probing

  • Across 100 random seeds, Bonferroni-corrected tests identify attention heads whose belief-probe accuracy significantly exceeds the 75% baseline.The analysis reports mean accuracies and 95% confidence intervals for top heads.Source: A. Statistical Analysis on Belief… · p. 13
  • Tables 3 and 4 report top attention heads for oracle and protagonist belief-status estimation with binary logistic regression in Mistral-7B.Source: Table 3 · p. 13, Table 4 · p. 13
  • Table 5 evaluates quaternary prediction of both agents’ belief states with multinomial logistic regression, where random guessing is 25%.Source: Table 5 · p. 13

B. Probing Prompts for Different Tasks

The paper constructs task-specific prompts by converting choices into statements and pairing stories with actions or beliefs. These prompts support perspective-specific labels for protagonist and oracle reasoning in Forward Action and Backward Belief tasks.

B. Probing Prompts for Different Tasks

C. Additional Probing Results

Linear probes recover belief representations across models and tasks, with a general bias toward oracle beliefs; nonlinear probes further reveal more complex structure in some activation spaces.

C. Additional Probing Results

D. Generalization to Other Datasets

The belief representations generalize from BigToM to ToMi despite different narrative templates and additional second-order questions, with predictive heads and intervention directions showing cross-dataset consistency.

D. Generalization to Other Datasets

E. Influence on Other Tasks

Intervening along identified Theory-of-Mind directions does not significantly alter unrelated language-understanding performance, while additional gradient analyses localize belief-relevant story elements.

E. Influence on Other Tasks

  • Intervention along identified ToM directions does not significantly change performance on MMLU, CoLA, MRPC, or QNLI.These benchmarks assess unrelated abilities including knowledge acquisition and grammatical acceptability; the comparisons are reported in Tables 7 and 8.Source: E. Influence on Other Tasks · p. 19, Table 7 · p. 19, Table 8 · p. 19

F. Additional Token Gradient Heatmaps

G. Hyperparameter Analysis

The hyperparameter analysis selects K = 16 for steadier intervention performance, while detailed comparisons examine responses that change correctness after intervention.

G. Hyperparameter Analysis

H. Detailed Comparison of LLM Responses

  • The detailed comparison records cases where intervention changes Forward Belief responses from incorrect to correct or from correct to incorrect.The comparison covers Mistral-7B and DeepSeek-7B under False Belief and True Belief settings.Source: H. Detailed Comparison of LLM Responses · p. 23

H.1. Mistral-7B on Forward Belief, False Belief

For Mistral-7B’s false-belief cases, the supplied examples report mixed intervention outcomes, including 33/100 changed to the correct answer and 0/100 changed to an incorrect answer.

  • 33/100 cases changed to the correct answer in the reported false-belief evaluation.Source: §H.1 · p. 23
  • The false-belief stories repeatedly contrast an agent’s initial belief with an unseen change to the world, such as damaged objects or altered conditions.Examples include wilted flowers, stolen eggs, damaged coral, and shifted equipment.Source: §H.1 · pp. 23, 24, 25, 27
  • Several scenarios show agents retaining their original beliefs after unobserved changes: Kofi expects perfect weather, Mei expects high tide at 5 AM, Fatima expects soft soil, and Rohan expects soft clay.Source: §H.1 · pp. 26–27
  • 0/100 cases changed to an incorrect answer in another reported false-belief evaluation.Source: §H.1 · p. 29

H.2. Mistral-7B on Forward Belief, True Belief

For Mistral-7B’s true-belief cases, the supplied passages report 0/100 changed to the correct answer, 10/100 changed to an incorrect answer, and scenarios where agents directly observe or notice the relevant change.

  • 0/100 cases changed to the correct answer in the reported true-belief evaluation.Source: §H.2 · p. 29
  • 10/100 cases changed to an incorrect answer in another reported true-belief evaluation.Source: §H.2 · p. 29
  • True-belief scenarios include agents who witness or notice the relevant change, such as a monkey eating mangoes or a chef noticing a sauce swap.Source: §H.2 · p. 30
  • Other true-belief examples describe agents recognizing changed conditions directly, including a storm at sea, a damaged chisel, and a blocked archaeological entrance.Source: §H.2 · p. 31

H.3. DeepSeek-7B on Forward Belief, False Belief

Across forward-belief and false-belief stories, the supplied examples report changes to model answers after intervention, alongside many agent-belief scenarios involving unobserved environmental changes.

  • 32/100 changed to the correct answer.Source: §H.3 · p. 32
  • The false-belief examples require agents to retain prior beliefs after unnoticed changes to objects, environments, or information.Source: §H.3 · pp. 32–34
  • The scenarios span changes to physical states, object contents, weather, and other conditions that the protagonist does not observe.Source: §H.3 · pp. 34, 36, 37, 39
  • Changed to an incorrect answer: 5/100.Source: §H.3 · p. 38

H.4. DeepSeek-7B on Forward Belief, True Belief

The true-belief examples describe agents who observe changed states or information and therefore receive intervention answers aligned with those updated observations.

  • The observed-change scenarios generally make the protagonist’s belief track the current state, such as seeing a swap, damage, or altered contents.Source: §H.4 · pp. 40, 41, 44
  • 16/100 changed to an incorrect answer in one reported result, while 6/100 changed to the correct answer in another.Source: §H.4 · pp. 40–41
  • Several examples explicitly report correct answers after the protagonist notices the changed condition, including weather, equipment, material, and object-state cases.Source: §H.4 · pp. 41–42
  • The examples show belief updates after direct observation of changes: when agents observe damage or hardening, they report the changed state, whereas intervention prompts responses matching the earlier state.Source: §H.4 · pp. 43–44
Loading 2402.18496v3…