Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities

1 Introduction

• ToM is the ability to attribute mental states to oneself and others, including recognizing that others may have different beliefs, desires, and intentions.Although Chain-of-Thought prompting can improve reasoning on some tasks, simple prompting solutions have not resolved ToM difficulties (Wei et al., 2022; Gandhi et al., 2023).Source: §1 · p. 1
• SIMTOM first filters context to what the person in question knows, then answers ToM questions using that filtered context, without additional training and with minimal prompt-tuning.The framework is inspired by Simulation Theory and improves over off-the-shelf models using zero-shot multiple-choice and Chain-of-Thought probing.Source: §1 · p. 2
• The authors find that prompted LLMs can perform perspective-taking and that stronger perspective-taking is closely tied to improved ToM capabilities.They propose SIMTOM as a simple baseline for future ToM research; the code is publicly available.Source: §1 · p. 2

2.1 “Simulation” Theory of Mind

• Simulation Theory describes ToM as a two-step process: taking another person’s perspective, then answering a question from that perspective.The theory has philosophical and empirical support but remains debated.Source: §2.1 · p. 2
• Perspective-taking involves simulating another person’s beliefs and goals, after which one reasons as if in that person’s shoes to answer a ToM question.Simulation Theory describes this as imagining what the person believes or desires; some theorists characterize the subsequent reasoning as reusing a shared cognitive mechanism.Source: §2.1 · p. 2

2.2 Are LLMs Capable of ToM?

• Most modern LLMs struggle with simple zero-shot ToM tasks, while reports of emerging capabilities are contested by failures on dataset alterations and concerns about benchmark limitations or leakage.The paper therefore considers unsupervised zero-shot evaluation; prior work reports that finetuned supervised models can be brittle and fail to generalize out of domain.Source: §2.2 · p. 2
• GPT-3.5-Turbo scores 12.5% on BigTOM’s false-belief action subset, while GPT-4 substantially trails human performance on ToMI and its strong BigTOM performance may reflect dataset creation by GPT-4.The authors conclude that LLMs do not yet reliably display zero-shot ToM capabilities.Source: §2.2 · p. 2

3 Benchmarking Theory-of-Mind Capabilities

• Sally–Anne tests assess beliefs and belief-driven actions, including false versus true beliefs and first- versus second-order questions.A character’s belief may differ from the story’s actual state when another agent changes the world while the character is absent.Source: §3 · p. 3

3.1 ToMI

• ToMI provides Sally–Anne stories, questions, and answer choices; this study uses an updated version that relabels second-order questions and disambiguates container locations.The updated version is attributed to Arodi and Cheung (2021) and Sap et al. (2022).Source: §3.1 · p. 3

3.2 BigTOM

• BigTOM (Gandhi et al., 2023) broadens Sally–Anne-style evaluation beyond object-location changes with more natural language and social settings.Its causal templates specify an agent’s desire, percept, and initial belief, then generate an environmental change and the agent’s resulting belief or action; GPT-4 generates the outputs.Source: §3.2 · p. 3

4 SIMTOM: SIMULATED Theory of Mind

• SIMTOM is a two-stage prompting framework intended to enhance zero-shot ToM capabilities by first modeling an agent’s perspective, then answering from it.Figure 2 depicts perspective-taking as identifying what the agent knows and wants before querying the model for an answer.Source: §4 · p. 3, Figure 2 · p. 4

4.1 Motivation · CoT Prompting

• In the motivating latte example, chain-of-thought can answer with the drink actually served rather than the drink the woman believes she received.The paper hypothesizes that ToM difficulty arises from combining perspective-taking and question-answering in one inference pass, motivating two passes.Source: §4.1 · p. 3, CoT Prompting · p. 3

4.2 Perspective-Taking · SimToM Step #1: Perspective-Taking

• SIMTOM implements perspective-taking by filtering the story to events the target character knows about before answering.The first-stage prompt asks which events the character knows; this operationalizes perspective-taking as “switching roles” to understand relevant beliefs and goals.Source: CoT Prompting · p. 3, §4.2 · p. 4, SimToM Step #1: Perspective-Taking · p. 4

4.3 Question-Answering · SimToM Step #2: Question-Answering

• The second stage asks the question using the perspective-filtered story, hiding information the character does not know.For the latte example, the woman’s version retains filling the cup with whole milk, adding cinnamon, and serving it, but omits the unseen milk swap.Source: §4.3 · p. 4, SimToM Step #2: Question-Answering · p. 4

5.1 Prompting

• Models answer multiple-choice probes by choosing from story-specific options, though they can abstain, which may lower performance below 50%.SIMTOM prompts vary somewhat by dataset and minimally between Llama and GPT models.Source: §5.1 · pp. 4–5

5.2 Evaluation

• Results are reported both across false-belief questions and across all question types for each benchmark.False-belief averages cover Forward Action and Belief on BigTOM, and first- and second-order questions on ToMI.Source: §5.2 · p. 5
• Evaluation uses Llama2-7b and 13-b, GPT-3.5-Turbo, and GPT-4 at temperature 0.0 for reproducibility.Open models run on a single A100 GPU; closed models are queried by API.Source: §5.2 · p. 5
• BigTOM evaluation uses Forward Action and Forward Belief, excludes Backward Belief, and retains its balanced 200 questions per type as a binary task.The character name is parsed as the story’s first word; random accuracy is 50%.Source: §5.2 · p. 5
• ToMI evaluation uses a balanced 1,000-example sample, with 100 examples from each of ten question types.The character name is parsed as the third word in the question.Source: §5.2 · p. 5
• Adding both template-derived answer choices to ToMI questions makes evaluation binary multiple choice and avoids ambiguous free-form baseline answers.Without choices, LLMs may produce answers that do not match either valid option; random accuracy is 50%.Source: §5.2 · pp. 5–6

• SIMTOM substantially improves over zero-shot and zero-shot CoT prompting across both BigTOM and ToMI.Source: §6 · p. 6

BigTOM Results

• On BigTOM false-belief questions, SIMTOM gains 29.5% and 14.2% absolute accuracy over GPT-3.5-Turbo zero-shot and zero-shot CoT, respectively.Across all BigTOM question types, False Belief gains over zero-shot CoT are 8.6% for Llama2-7b-chat and 5.7% for GPT-3.5-Turbo. GPT-4 saw a slight decrease, potentially confounded by BigTOM's GPT-4-generated data; GPT-4 had saturated human performance zero-shot (Gandhi et al., 2023).Source: §6 · p. 6

ToMI Results

• On ToMI false-belief questions, SIMTOM gains 16%, 19%, and 47% absolute accuracy over zero-shot CoT for Llama2-7b-chat, Llama2-13b-chat, and GPT-3.5-Turbo, respectively.Across all ToMI question types, gains over zero-shot CoT are 13.4% for GPT-4 and 16.1% for Llama2-13b-chat; control conditions such as True Belief are less challenging for modern models.Source: §6 · p. 6
• On GPT-3.5-Turbo ToMI false-belief questions, SC-CoT scores 33.50% versus SIMTOM's 81.00%, while Tree-of-Thoughts also underperforms SIMTOM.SC-CoT performs similarly to CoT across question categories, and the authors suggest Tree-of-Thoughts may be mismatched because its voting was designed for creative-writing generations rather than rationales. Second-order questions are harder for most models than first-order questions.Source: §6 · p. 6

SimToM-Single

• Single-prompt perspective-taking is less effective than using two prompts, and single-inference approaches often answer from the world state rather than the character’s mental state.A multi-step CoT variant replacing perspective-taking with reasoning fell to 58.25% on BigTOM total and 46.8% on ToMI, indicating that SimToM’s gains are not explained by multi-step prompting alone.Source: SimToM-Single · pp. 6–7, Table 2 · p. 7

7.2 Oracle Perspective-Taking

• 96% accuracy with oracle perspectives on both ToMI and BigTOM False Belief tests approaches solving these tasks, though performance lags on all ToMI question types.Four human annotators produced perspectives from stories without seeing questions or answer choices; models using these perspectives improved from 81.0% to 96% on ToMI False Belief and from 70.5% to 96.0% on BigTOM False Belief.Source: §7.2 · pp. 7–8

7.3 Domain-Specific Perspective-Taking Strategies

• Domain-specific perspective-taking raises BigTOM False Belief accuracy from 70.5% to 90.5% and improves accuracy across all ToMI questions from 72.8 to 79.3.The method adds three examples of perspective-taking from in-domain ToMI stories; it still trails oracle perspective-taking by over 10% on ToMI False Belief and 5.5% on BigTOM False Belief.Source: SimToM-Domain · p. 8, §7.3 · p. 8

7.4 Extension to Complex Theory-of-Mind Scenarios

• On 30 hand-selected CosmosQA ToM questions, perspective-taking sometimes inferred useful information from tangential clues, but could also produce hallucinated or unsupported perspectives.The analysis highlights the relationship between inference and hallucination as important for real-world settings with limited information.Source: §7.4 · p. 8

Evaluating LLM Capabilities on ToM

• Recent zero-shot ToM evaluations use LM, multiple-choice, and Chain-of-Thought probing, while this study minimally changes prompts to respect reasonable evaluation standards (Shapira et al., 2023).Prompt styles can elicit different performance capabilities.Source: §8 · p. 8

Enhancing LLM Reasoning Capabilities

• Reasoning methods address distraction by selecting relevant facts (Shi et al., 2023; Creswell et al., 2022), whereas perspective-taking also excludes facts the simulated agent should not know.The passage notes that how to define relevance for selection in ToM is unclear.Source: §8 · p. 9

Symbolic Reasoning

• Symbolic approaches separate neural and symbolic reasoning or use belief-state graphs, while this work studies ToM without external memory or human-written inference-time algorithms.The passage says it is unclear how to apply these symbolic methods to naturalistic ToM datasets such as BigTOM.Source: §8 · p. 9

9 Conclusion

• SIMTOM implements perspective-taking in a separate prompt before question-answering, rather than combining both steps in one prompt.The first prompt filters context to what the character knows.Source: §9 · p. 9
• With oracle perspective-taking, today’s LLMs could close-to-solve ToM benchmarks.The conclusion also suggests that LLMs with improved perspective-taking could perform even more strongly on ToM tasks.Source: §9 · p. 9
• LLMs are substantially stronger at ToM reasoning than a single inference pass suggests.The conclusion frames perspective-taking as the reason their capabilities appear underestimated when probed with one pass.Source: §9 · p. 9

10 Future Work

• Future evaluations should test perspective-taking in complex, lifelike ToM settings beyond Sally-Anne false-belief tests, reflecting its broader scope in cognitive science.Such evaluations may uncover new challenges in modeling perspective-taking.Source: §10 · p. 9

Limitations

• The method implements perspective-taking by hiding story information, but perspective-taking may instead require imputing unseen information, as in imagining oneself in another’s shoes.The authors attribute this limitation partly to the datasets available for ToM reading comprehension.Source: Limitations · p. 9
• Current ToM reading-comprehension datasets, to the authors’ knowledge, use Sally-Anne belief scenarios involving changed world states that characters do not know about.These scenarios test basic anticipation of beliefs and actions from the perspective of an omniscient viewer.Source: Limitations · p. 9
• The study tested no models smaller than 7B parameters, leaving open whether only LLMs can achieve this level of ToM performance with SimToM.The authors suggest improving LLM performance may make this limitation less disruptive to future ToM research.Source: Limitations · p. 9

Ethics

• Real-world ToM could enable inferring mental states without consent and deceptive political or financial persuasion.The authors flag these as potential unethical use cases of advanced ToM capabilities.Source: Ethics · p. 10
• ToM inferences could also produce inaccurate, harmful, or unfair interactions that amplify biases in LLMs.The authors recommend careful assessment of outcome distributions for research on direct-interaction use cases, particularly potentially harmful domains such as unconscious belief change.Source: Ethics · p. 10
• Advanced ToM could also support more empathetic and socially aware technology, with growing impact as AI enters more interactive products.The authors recommend particular care for research on direct interactions with people rather than offline prediction.Source: Ethics · p. 10

A A Further Discussion of Cognitive Science Perspectives on Simulation Theory

• Simulation Theory is supported by arguments that understanding others’ beliefs involves simulating their perspective (Gordon, 1986, 1995; Gordon, 2007), and by mirror-neuron evidence.The philosophical argument draws on Evans (1982), who describes self-directed ToM as observing external circumstances rather than introspection.Source: A A Further Discussion of Cognitive… · p. 12
• Heal (1994) suggests that artificial cognitive processes such as those in LLMs may achieve Simulation Theory capabilities if they produce similar mental states.The authors use this possibility to motivate their approach.Source: A A Further Discussion of Cognitive… · p. 12
• Theory-Theory explains ToM through commonsense rules, but the authors do not adjudicate between it and Simulation Theory, instead implementing a simulation-based approach on established ToM tasks.The passage gives the “Law of Sight” as an example of a Theory-Theory rule.Source: A A Further Discussion of Cognitive… · p. 12

B SIMTOM Prompts

• SimToM prompts are reproduced in full and vary slightly by model architecture, while retaining essentially the same core content and instructions.For Llama-2, extra directions discourage refusing non-factual answers and require a single-word response.Source: B SIMTOM Prompts · p. 12

GPT/Llama-2-chat

• The GPT/Llama-2-chat prompt filters a story to events the specified character knows, using rules about shared locations and leaving or re-entering them.It instructs the model to output only those events, without explanation.Source: GPT/Llama-2-chat · p. 12

B.1.2 Simulation

• The simulation prompt asks the model to answer as the named character based on the information provided, choosing one of the listed choices concisely.The prompt says one sentence is enough.Source: §B.1.2 · p. 12

Llama-2-chat

• The Llama-2-chat prompt asks the model to answer as the named character and requires one listed choice, forbidding an insufficient-information response.It requires a single-word answer with no additional output.Source: Llama-2-chat · p. 13

B.2.1 Perspective Taking

• The perspective-taking prompt asks the model to imagine being the named character and consider a story with an unexpected event.Source: §B.2.1 · p. 13
• If the character notices the unexpected event, the prompt retains the original story; otherwise, it keeps only events before the event and requires a yes/no awareness label.The output format is “Sees/Notices/Realizes: (Yes/No) Story:” followed by the original or edited story.Source: §B.2.1 · p. 13

Llama-2-chat

• The Llama-2-chat perspective prompt presents a story containing an unexpected event.Source: Llama-2-chat · p. 13
• For Llama-2-chat, the prompt retains the story if the named character notices the event, but otherwise keeps only events before it that the character knows.Source: Llama-2-chat · p. 13

B.2.2 Simulation

• The simulation prompt tells the model to answer the question as the named character based on the information provided above.Source: §B.2.2 · p. 13
• The simulation answer should be concise, at most one sentence, and use the specified “Answer:<option>)<answer>” format.Source: §B.2.2 · p. 13

Llama-2-chat

• The Llama-2-chat answering prompt places the perspective information alongside instructions to answer concisely in the specified option-and-answer format.Source: Llama-2-chat · p. 13
• It then asks the model to answer the question as the named character and choose the most straightforward answer.Source: Llama-2-chat · p. 13

ToMI

• ToMI prompts first select events the character knows, then answer the mental-state question using the rewritten sequence as the new story.The prompt asks for both steps in one response.Source: ToMI · p. 14
• For ToMI, characters know events they perform and events occurring in a location while they are there, including object locations and movements.After leaving, they no longer know events in that location, but may re-enter.Source: ToMI · p. 14

BigTOM

• BigTOM prompts ask the model to imagine being the named character and determine whether that character notices an unexpected event.If the character notices it, retain the original story; otherwise, keep only events before the change, then answer from the rewritten story.Source: BigTOM · p. 14
• The BigTOM format requires a yes/no awareness judgment and, for an unaware character, truncates the story before the unexpected event.It then asks for a choice between the supplied answer options based on the retained story.Source: BigTOM · p. 14

ToMI

• A ToMI example applies these rules: William knows events in the dining room while present, but not Abigail’s unrelated cellar events.The example also shows that William knows a later object move after re-entering the dining room.Source: ToMI · p. 15

BigTOM

• A second BigTOM prompt explicitly identifies the sentence describing the change, checks whether the character notices it, and edits the excerpt only if they do not.When the character notices the change, it outputs the original excerpt.Source: BigTOM · p. 15

0-shot

• The 0-shot prompt requests a concise answer of at most one sentence in the format ’Answer:<option>)<answer>’.Source: 0-shot · p. 15

0-shot CoT

• The 0-shot CoT prompt asks for step-by-step reasoning before the final answer, requires an option to be selected, and disallows “none of the above” or insufficient-information responses.Source: 0-shot CoT · p. 16

F Non-Instruction Tuned Models

• Non-instruction-tuned models are typically prompted through natural continuations or few-shot examples in the desired format.Source: F Non-Instruction Tuned Models · p. 16
• For Llama-2-13b and Llama-2-7b, the tested continuation prompt continued the question rather than enabling perspective-taking, while few-shot examples no longer provided zero-shot evaluation.Source: F Non-Instruction Tuned Models · p. 16
• The investigation was therefore limited to fine-tuned instruction-following models.Source: F Non-Instruction Tuned Models · p. 16

G Breakdown of Results Across Question Types

• Table 3 reports BigTOM results and identifies SIMTOM ablations, including domain-specific perspective-taking and the single-prompt variant, alongside SC-CoT and ToT.Source: Table 3 · p. 17
• Table 4 reports ToMI results by question type; its nt column covers control questions about people who did not witness a change, and Mem-real averages memory and reality scores.The caption also reports that adding ToMI-specific rules to zero-shot baselines did not change their performance, indicating that the rules did not artificially inflate SIMTOM's performance relative to the baselines.Source: Table 4 · p. 17