Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning

1 Introduction

• Social information integration is presented as important for AI trust and safety and for reasoning about multi-agent situations.The paper calls this capability mentalizing, or Theory of Mind (ToM).Source: §1 · p. 1
• ToM involves both choosing whether and how deeply to mentalize and inferring agents’ mental states at that depth.The authors describe these as two steps; failure at either can produce an incorrect conclusion.Source: §1 · p. 1
• Whether LLMs invoke ToM receives little attention, although it is theoretically central, can be confused with poor logic, and carries resource and performance costs.The paper focuses on how ToM is used, not the evolutionary pressures behind social representation.Source: §1 · p. 2
• The paper surveys AI approaches, contrasts biological and synthetic definitions of ToM, and proposes evaluation improvements inspired by biological agents.It also reports experiments on two common ToM benchmarks to support its analysis of current research.Source: §1 · p. 2
• The authors identify a gap: LLM work tends to overlook the act of mentalizing and instead emphasizes logic problems.Their contributions include a cognitive-science-inspired two-step formulation and a roadmap for evaluating ToM invocation.Source: §1 · p. 2

2 ToM in AI research

• The section analyzes approaches and challenges for modeling and assessing ToM in AI communities, first reviewing existing AI methods and then discussing LLM-focused approaches.Source: §2 · p. 2

2.1 Formal Architectures of ToM

• Formal ToM models treat mentalizing as inference about other agents, with Depth of Mentalizing (DoM) describing recursive belief depth; higher levels can be modeled with interactive POMDPs.Observational tasks commonly use DoM(0), which lacks recursive beliefs; recursive formulations in LLMs are difficult because models are black boxes.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2 ToM Benchmarking

• LLM benchmarks range from Sally-Anne false-belief tests to conversational, higher-order, perceptual, epistemic, and multimodal tasks.ToMi and OpenToM use less predictable generation protocols and more natural settings than earlier benchmarks.Source: §2.2 · pp. 2–3
• Benchmark results remain difficult to interpret because task alterations affect performance, training-set transparency is limited, and most tests are static observations with a single answer.These tests do not capture moment-to-moment interaction or the range of possible beliefs and behaviors.Source: §2.2 · p. 3

2.3 ToM Add-ons

• ToM add-ons include task-specific prompts, updateable symbolic modules, post-model decoding, and symbolic planners, but typically assume ToM is required.The illustrated hierarchy distinguishes an agent’s own policy at DoM(-1), beliefs about a partner’s policy at DoM(0), and recursive beliefs at DoM(1).Source: §2.3 · p. 3, Figure 2 · p. 3

3 What Can We Learn From Biological Agents?

• Cognitive science highlights adaptability in DoM and intentionality: changing mentalizing depth with feedback and invoking ToM when needed while avoiding it when unnecessary.The paper describes the first inference step as theorizing about agents’ belief structures, which can be reassessed as information changes.Source: §3 · p. 3
• The paper formalizes ToM as a meta-decision to choose DoM given a prompt, followed by inference about an outcome given that DoM.The first step is represented as do(DoM(k)) given X; the second as P(Y | do(DoM(k)), X).Source: §3 · p. 3

3.1 Invoking DoM

• Cooperation can permit shallow mentalizing, whereas competition can favor hierarchical mentalizing, which requires more time and energy.The paper cautions that failing to capture shifts toward competitive mentalizing may hinder human-AI alignment.Source: §3.1 · p. 4
• A fixed or mismatched DoM can miss malicious intent or create false beliefs of harm, and the appropriate level may be unclear when partners differ.The paper therefore frames DoM as a distribution to estimate rather than a single answer in question-answering settings.Source: §3.1 · p. 4

What do the ToM benchmarks benchmark?

• Vignette benchmarks make ToM necessity and the required depth obvious, so they largely test step-II logic rather than whether to distinguish self from other.On HiToM and FANToM, GPT-4o-mini performs well and GPT-4o is nearly perfect at predicting required ToM order; Chain-of-thought prompting gives some improvement.Source: What do the ToM benchmarks benchmark? · p. 4
• The authors recommend testing whether DoM is correctly invoked, using interactive active-agent settings, and assessing DoM distributions across social contexts.They argue that existing AI tasks are non-interactive logical problems and that static benchmarks only distantly resemble real-world ToM use.Source: What do the ToM benchmarks benchmark? · p. 4
• Testing only the predicted outcome cannot distinguish undermentalization from reasoning failure, or establish that successful answers used ToM when it was unnecessary.Even formal models commonly assume a fixed DoM and invoked ToM, while add-ons typically address inference without directly addressing the decision to invoke it.Source: What do the ToM benchmarks benchmark? · p. 4

4 Discussion and Conclusion

• Current NLP benchmarks and AI work are largely non-interactive logical problems and therefore miss interaction and social adaptation.The paper notes that these benchmarks can still provide valuable reasoning tests and insights into LLM performance.Source: §4 · p. 5
• Evaluation should test whether models correctly invoke Depth of Mentalizing, use ToM interactively as active agents, and assess its dynamics across social contexts.The paper distinguishes cooperative from competitive contexts.Source: §4 · p. 5
• Cognitive science distinguishes social from non-social outcomes, motivating benchmarks that represent social dynamics.The paper connects this distinction to how models should be benchmarked for ToM.Source: §4 · p. 5
• The paper extends Wang et al. (2025)'s call for dynamic, user-centric ToM benchmarks by emphasizing a commonly overlooked phase of ToM reasoning.The supplied passage does not identify that phase by name.Source: §4 · p. 5
• Interactive ToM tasks should extend beyond simplified fixed worlds to diverse domains, contexts, incomplete information, cooperation levels, and heterogeneous agent DoM.The paper argues that multiple diverse scenarios should provide better coverage than a single scenario.Source: §4 · p. 5

Limitations

• The work evaluates attributed beliefs, not the full range of ToM factors such as emotions and desires.The authors still regard belief attribution as a critical component of ToM evaluation.Source: Limitations · p. 5
• The recommendations are qualified by the breadth of cognitive-science ToM literature and active debate over its components.The authors describe that literature as vast and often conflicting.Source: Limitations · p. 5

A A Case Study on Existing Benchmarks

• The authors argue that HiToM and FANToM challenge complex reasoning rather than ToM invocation, which they associate with Type C errors.They describe both benchmarks as challenging even for strong models.Source: A A Case Study on Existing Benchmarks · p. 9
• The test predicts each question's required ToM order; correct order prediction alongside an incorrect answer indicates failure in answering, not invocation.The benchmark samples are labeled with the order required to answer each question.Source: A A Case Study on Existing Benchmarks · p. 9

A.1 Experimental Setup

• HiToM uses Sally-Anne-like stories with higher-order beliefs, questions from order 0 through fourth-order beliefs, and cases involving communication and deception.Characters may intentionally report false statements.Source: §A.1 · p. 9
• FANToM uses conversational stories in which characters enter and leave, with factual questions and first- or second-order belief questions.The dataset includes question-type labels.Source: §A.1 · p. 9
• The evaluation uses 100 random examples from each dataset, while a small number of manual experiments are excluded.Source: §A.1 · p. 9
• Models predict question-level ToM order with the question alone or with its story, both with and without Chain of Thought.Source: §A.1 · p. 9
• The three tested models are GPT-3.5-turbo, GPT-4o-mini, and GPT-4o.Source: §A.1 · p. 9
• ToM order is the number of mentalization levels needed to answer a specific question, counting nested beliefs and excluding implicit beliefs.Factual questions require order 0; asking what someone thinks requires order 1.Source: §A.1 · p. 9

• GPT-4o-mini achieves good results, GPT-4o is nearly perfect, and GPT-3.5-turbo struggles.Source: §A.2 · p. 10
• CoT gives some improvement.Source: §A.2 · p. 10
• Scores are identical with and without the story as input.Source: §A.2 · p. 10

• FANToM can leave implicit ToM in actions uncounted as additional levels, as when “dealing” implies an internal intention.Source: §A.3 · p. 10
• FANToM questions can suggest that a character has a belief even when the story's intended answer is that they have none.Source: §A.3 · p. 10
• FANToM assumes reports are true and does not count that assumption as ToM; HiToM includes deception but makes the truth explicit.Source: §A.3 · p. 10
• The HiToM error-type analysis concerns model explanations, whereas this experiment tests depth independently of the question itself.Source: §A.3 · p. 10
• Predicting ToM order is much easier than the original task, even for GPT-3.5-turbo, and can be done from the question alone.Source: §A.3 · p. 10
• The challenging part of the benchmarks is inference given the ToM order, not predicting the order itself.The benchmarks also frame ToM level as a prediction task with a single correct solution.Source: §A.3 · p. 10