Source-linked AI summary
LLMs achieve adult human performance on higher-order theory of mind tasks
Winnie Street, John Oliver Siy, Geoff Keeling, Adrien Baranes, Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Blaise Aguera y Arcas, Robin I. M. Dunbar
TL;DR
Research on LLM theory of mind has largely focused on second-order reasoning, despite higher-order reasoning’s relevance to multi-party interaction. The paper introduces MoToMQA and compares five LLMs with a newly gathered adult benchmark. GPT-4 and Flan-PaLM perform at or near adult levels overall, with GPT-4 exceeding the adult comparison on sixth-order tasks.
Problem
Most LLM theory-of-mind research has focused on second-order reasoning, while multi-party interactions call for higher-order reasoning.
Source: §1 · p. 1Method
The authors introduce MoToMQA, a short-story true/false benchmark spanning ToM orders 2–6, and compare LLM performance with a newly gathered adult benchmark.
Source: §3 · p. 3, §2.2 · p. 3Results
Across MoToMQA, GPT-4 and Flan-PaLM performed at or near adult levels overall; GPT-4 reached 93% accuracy on sixth-order tasks versus humans’ 82%.
Source: §5 · pp. 7–8Takeaways & Limitations
The authors interpret the findings as evidence that GPT-4 and Flan-PaLM developed ToM reasoning beyond manipulation of superficial statistical relationships.
Source: §8 · p. 9The benchmark comprises 140 English statements, stops at six orders, and does not control for the types of cognitive or affective states involved.
Source: §6 · p. 9
Abstract
from arXiv · showhide
This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (e.g. I think that you believe that she knows). This paper builds on prior work by introducing a handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to compare the performance of five LLMs to a newly gathered adult human benchmark. We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level performance on ToM tasks overall, and that GPT-4 exceeds adult performance on 6th order inferences. Our results suggest that there is an interplay between model size and finetuning for the realisation of ToM abilities, and that the best-performing LLMs have developed a generalised capacity for ToM. Given the role that higher-order ToM plays in a wide range of cooperative and competitive human behaviours, these findings have significant implications for user-facing LLM applications.
1 Introduction
Theory of Mind (ToM) supports reasoning about others’ mental states, while prior LLM research has focused mainly on second-order tasks despite the relevance of higher-order reasoning to multi-party interactions. This paper introduces MoToMQA and reports adult-level or near-adult-level performance by GPT-4 and Flan-PaLM.
1 Introduction
- ToM is the ability to infer and reason about one’s own and others’ mental states, supporting prediction and influence of behaviour.The paper describes ToM as central to human social intelligence.Source: §1 · p. 1
- Earlier LLM ToM research focused mostly on second-order reasoning, although multi-party social interactions call for higher-order reasoning.The paper defines order as the number of mental states involved in a ToM reasoning process.Source: §1 · p. 1
- GPT-4 and Flan-PaLM reached at-human or near-human performance, respectively, on the paper’s ToM tasks.Figure 1 compares humans and five LLMs on tasks up to order 6.Source: §1 · p. 1, Figure 1 · p. 2
- MoToMQA tests LLM theory of mind at orders 2–6 using true/false questions about characters in short stories.The benchmark is based on an adult ToM test.Source: §1 · p. 1
2 Related work
Prior work establishes higher-order ToM’s relevance to social complexity while reporting mixed evidence about LLM capabilities. This study extends that work with a new handwritten test, log-probability scoring, and an adult human benchmark.
2.1 Higher-order ToM
- Higher-order ToM is associated with social complexity and supports group negotiation, bonding, and cultural practices; adults generally infer up to five orders.Reported examples include humour, religion, and storytelling.Source: §2.1 · p. 2
2.2 LLM ToM
- Prior LLM ToM findings were mixed, and only two cited studies examined higher orders; this study adds new handwritten stories and tests one order beyond He et al. [2023].The passage describes He et al.’s range as orders 0–4, equivalent to this paper’s orders 2–5.Source: §2.2 · p. 2
- The paper follows an IMT convention in which the assessed subject’s mental state is first-order and an inference about another’s state is second-order.Some scholars instead begin the count at zero-order.Source: §2.2 · p. 2
- The Sally-Anne task tests false-belief understanding by asking where Sally will search after Anne moves an object while Sally is away.The task contrasts Sally’s belief about the object’s location with the location known to the observer.Source: §2.2 · p. 2
- The study uses log probabilities for candidate responses, which account for multiple ways a model could provide the correct answer.The authors present this choice as improving data robustness.Source: §2.2 · p. 3
- The authors compare LLMs with a newly gathered adult benchmark, while explicitly not assuming that human and LLM performance relies on the same cognitive processes.They call the adult comparison relevant because LLMs primarily interact with adults and human higher-order ToM continues developing into early adulthood.Source: §2.2 · p. 3
3 Materials and method
The study introduces MoToMQA to assess ToM and factual reasoning across increasing orders, and compares five LLMs with a newly gathered adult benchmark. Its procedures use controlled story, prompt, and response-order conditions and derive model answers from candidate-token probabilities.
3 Materials and method
- MoToMQA adapts the Imposing Memory Task into seven short social-interaction stories with true/false ToM statements at orders 2–6 and matched factual statements.The benchmark includes 20 statements per story; factual statements span levels 2–6 and provide a recall control for ToM statements, which require recall plus inference.Source: §3 · p. 3
3.1.1 Human procedure
- The adult survey gathered 29,259 UK-based first-language English responses, balanced across age and gender groups; each participant saw one statement after reading a story twice.Participants were randomly assigned a story and statement, and each saw only one statement to prevent learning across trials.Source: §3.1.1 · p. 4
3.1.2 LLM procedure
- Five LLMs were tested under human and simplified prompt conditions, with true/false response order also varied to assess anchoring.The human prompt reproduces the human-study wording, while the simplified prompt uses Question/Answer tags; response options appeared in both true-first and false-first orders.Source: §3 · p. 3, §3.1.2 · p. 4
3.2 Dataset creation
- Model responses were estimated by aggregating probabilities for equivalent true and false tokens, then classifying statements as true when affirmative probability exceeded 50%.Human responses were reduced to one binary answer per statement; the human and LLM treatment conditions were mapped together despite slight differences, and analyses omitting 16 mismatched statements retained the same conclusions.Source: §3.2 · pp. 5–6
4 Results
GPT-4 and Flan-PaLM were the strongest models on ToM overall, with GPT-4 not significantly different from adults; performance comparisons varied by order and task. Factual statements were easier than ToM statements for several groups, and response-option order affected PaLM and GPT-3.5.
4.1 ToM task performance
- GPT-4 did not differ significantly from adults overall (p = .522), while Flan-PaLM performed below adults (p = .017); both outperformed GPT-3.5, PaLM, and LaMDA.GPT-4 and Flan-PaLM did not significantly differ from each other (p = .105).Source: §4.1 · p. 6, Table 2 · p. 7
- By order, humans did not differ significantly from GPT-4 at orders 2, 3, 4, or 6, but outperformed both GPT-4 and Flan-PaLM at order 5.Humans did not significantly differ from Flan-PaLM at the other tested orders.Source: §4.1 · p. 6
- GPT-4 scored 94.6% at order 3 and 92.9% at order 6, both significantly above its 73.2% at order 4; humans reached 98.2% at order 5.Flan-PaLM also scored significantly higher at order 3 (94.6%) than order 4 (78.6%).Source: §4.1 · p. 6
4.2 Factual task performance
- On factual tasks, GPT-4 and Flan-PaLM performed best among models with no significant difference, and GPT-4 did not differ significantly from adults.Adults performed significantly better than Flan-PaLM on factual tasks.Source: §4.2 · pp. 6–7
4.3 Comparing performance on ToM and factual tasks
- Factual accuracy exceeded ToM accuracy for adults, Flan-PaLM, GPT-4, and GPT-3.5, while the two measures did not significantly differ for PaLM or LaMDA.The factual statements require recall, whereas ToM statements require recall plus inference.Source: §4.3 · p. 7
4.4 Anchoring effect
- True-first ordering increased the proportion of true responses from PaLM (73.2% vs. 47.1%) and GPT-3.5 (43.9% vs. 22.9%), but not from Flan-PaLM, GPT-4, or humans.LaMDA responded true to every statement in either order.Source: §4.4 · p. 7
5 Discussion
GPT-4 and Flan-PaLM performed at or near adult levels on most MoToMQA orders, with GPT-4 exceeding adults on sixth-order tasks. The discussion relates performance to model size and finetuning, considers differences across ToM orders and task types, and highlights potential benefits and risks.
5 Discussion
- On MoToMQA, GPT-4 and Flan-PaLM performed strongly compared with humans; at every level except order five, neither model differed significantly from human performance, and GPT-4 exceeded humans on sixth-order ToM.Source: §5 · p. 7
- Performance showed a positive relationship with model size, while PaLM and Flan-PaLM’s contrast suggests finetuning may be needed to realize ToM capacity.The authors say further testing across more models is needed to assess this hypothesis.Source: §5 · p. 8
- Instruction finetuning, language mastery, and multimodality are proposed as possible contributors to ToM performance, but the discussion presents these as explanations to investigate.The authors suggest multimodality might help GPT-4 use visual behavioral signals in the stories.Source: §5 · p. 8
- GPT-4 and Flan-PaLM declined slightly from ToM orders two to four, but GPT-4 rose at orders five and six while Flan-PaLM continued declining.The authors suggest GPT-4 may have learned a pattern of human performance, while differences in pretraining corpora may explain why Flan-PaLM lacks it.Source: §5 · p. 8
- Humans and LLMs performed better on factual recall than ToM tasks, and GPT-3.5 and PaLM were the only subjects showing an anchoring effect.The authors suggest these models’ anchoring effect indicates they lack generalized ToM capacity and are not robust to prompt perturbations.Source: §5 · pp. 8–9
- Higher-order ToM could support interpreting interlocutors and mediating multi-party conflicts, but may also increase risks of persuasion, manipulation, and exploitation.The authors call for research on real-world interactions and guardrails that mitigate risks without suppressing potential benefits.Source: §5 · p. 9
6 Limitations
The benchmark’s limited size, language, and order range constrain the scope of its findings. It also leaves cognitive and affective state types uncontrolled.
6 Limitations
- The benchmark contains 140 English-language test statements and covers ToM only through order six.The authors state that its size limits the generalizability of the findings.Source: §6 · p. 9
- An English-only test cannot capture potential linguistic and cultural variation in human ToM or assess model ToM in other languages.The models are able to produce other languages, but this benchmark does not assess their ToM in them.Source: §6 · p. 9
- The study did not control whether statements involved cognitive or affective states, and the six-order ceiling may not exhaust human or LLM capacity.Source: §6 · p. 9
7 Future research
The authors propose broader benchmarks, testing beyond sixth-order ToM, and more embodied multimodal evaluation as directions for future work.
7 Future research
- Develop culturally diverse, multilingual benchmarks that parameterize cognitive and affective states to capture potential differences in reasoning about them.Source: §7 · p. 9
- Extend the test suite beyond sixth-order ToM to investigate the limits of human and LLM performance.Source: §7 · p. 9
- Use multimodal paradigms with signals such as facial expressions, gaze, and tone of voice to reflect embodied human ToM.Source: §7 · p. 9
8 Conclusion
GPT-4 and Flan-PaLM showed adult-level or near-adult higher-order ToM, while smaller and non-finetuned models showed limited or no capacity; GPT-4 also surpassed adults on sixth-order tasks. The authors interpret the results as evidence of ToM reasoning beyond superficial statistical relationships, but stop short of concluding that this demonstrates the cognitive ability called Theory of Mind.
8 Conclusion
- GPT-4 and Flan-PaLM showed adult-level or near-adult higher-order ToM, while smaller and non-finetuned models showed limited or no capacity.GPT-4 performed better than humans on sixth-order tasks.Source: §8 · p. 9
- The authors interpret the results as evidence that GPT-4 and Flan-PaLM developed ToM reasoning beyond superficial statistical relationships, citing their novel test suite and resistance to prompt perturbations.Source: §8 · p. 9
- The authors refrain from concluding that LLM performance demonstrates human-like Theory of Mind, noting that human and LLM developmental processes differ.They suggest functionally equivalent behavior may warrant recognition as a different kind of understanding.Source: §8 · p. 9
A.1 English language screener
The screener required participants to have English as a first language and to use English most commonly. The criteria aimed to focus on linguistic proficiency and reduce grammatical complexity as a performance confound.
A.1 English language screener
- Participants qualified only if English was a first language and their most commonly used language.English as a first language meant a language, or one of the languages, first learned as a child.Source: §A.1 · p. 13
- The screener targeted participants who use English day to day, excluding those who had shifted primarily to another language.This was intended to screen out people who might be less fluent in English despite learning it first.Source: §A.1 · p. 13
- The authors avoided the term “native speaker” to distinguish linguistic proficiency from socio-cultural identity and other irrelevant factors.They also sought to reduce the chance that grammatical complexity would confound performance.Source: §A.1 · p. 13
A.2 Human pilot study
The pilot used participant responses and explanations to identify problems in the stimuli, then revised ambiguous wording before the final survey. A timer was added after early responses suggested participants were rushing.
A.2 Human pilot study
- The Qualtrics pilot included 1,440 participants (10 responses per statement); after the first 50 had a one-minute median response time, researchers disabled “Next” for 60 seconds for the remaining 1,390.They retained the timer for the final survey.Source: §A.2 · p. 13
- Reviewers found ambiguity in 15 of 17 poorly performing statements and revised 14 statements and one story.Participant explanations helped identify wording problems that could account for low performance.Source: §A.2 · p. 13
- The two remaining low-performing items, an order 4 statement and an order 2 statement, were left unchanged because responses suggested inattention.The researchers reviewed both items and made no wording changes.Source: §A.2 · p. 13
A.3 LLM prompt conditions
The appendix documents LLM prompt conditions, model details, API procedures, and supplementary condition checks. It also describes the tables used to report ToM performance by order and compare ToM with factual performance.
A.3 LLM prompt conditions
- Table 3 records the exact LLM input text for each of the four tested conditions.The conditions varied prompt and question wording as described in the study.Source: §A.3 · p. 13
A.4 Details of the LLMs tested
- LaMDA was evaluated as a 35B-parameter model without fine-tuning, pretrained on 1.56T words and optimized for safety and factual grounding.The training mixture included public and web text, with dialogs making up half the described dataset.Source: §A.4 · p. 14
- The study used 540B-parameter PaLM and Flan-PaLM, with Flan-PaLM instruction-finetuned on more than 1.8K natural-language tasks.The described PaLM training corpus contained 780B words; instruction finetuning was intended to improve performance when models receive instructions.Source: §A.4 · p. 14
- The evaluated GPT-3.5 Turbo Instruct model was fine-tuned for instruction following, while GPT-4 underwent RLHF after pretraining.The GPT-3.5 model's parameter size was undisclosed; GPT-4's size was also unpublished in the paper's account.Source: §A.4 · p. 15
A.5 LLM procedure
- Google APIs accepted candidate tokens for log-probability scoring, whereas GPT models required tokenization and a logit bias of 100 to ensure candidates appeared.The authors considered these API differences unavoidable but judged that they did not prohibit fair comparison using relative probabilities of semantically equivalent true and false tokens.Source: §A.5 · p. 15
A.6.1 Story and prompt conditions
- Prompt-condition changes did not significantly affect ToM or factual accuracy for any tested LLM, and story condition did not significantly affect human accuracy.The reported comparisons include human versus simplified prompts for each model and human story-present versus story-absent conditions.Source: §A.6.1 · p. 16
- Table 6 reports human and model performance on ToM tasks through order 6, using Wilson intervals instead of traditional confidence intervals.The caption notes Wilson intervals are bounded between 0 and 1 and can be asymmetric.Source: Table 6 · p. 17
- Table 7 reports the comparison of LLM and human performance on ToM versus factual tasks using an independent-samples test of proportions..Source: Table 7 · p. 18