Source-linked AI summary
Probing the Robustness of Theory of Mind in Large Language Models
Christian Nickel, Laura Schrewe, Lucie Flek
TL;DR
Whether LLMs’ apparent Theory of Mind on simple tasks reflects robust social reasoning remains an open question. This paper introduces a complexity-grouped benchmark and evaluates four open-source models; low goal accuracy indicates only limited ToM capabilities.
Problem
LLMs have shown ToM-like performance on simple psychology-style tasks, but it remains unclear whether they learned agents’ mental states or patterns in familiar task wording.
Source: §1 · p. 1Method
The authors create a manually crafted benchmark of unexpected-content and unexpected-transfer tasks across 10 complexity classes and evaluate four state-of-the-art open-source LLMs.
Source: §1 · p. 1, §2.2 · p. 4Results
Takeaways & Limitations
The complexity-grouped dataset provides a basis for further investigation of ToM capabilities and their weak spots.
Source: §5 · p. 8, §6 · p. 9The tested models may be too small to reveal more informative patterns, and further insights may require higher goal accuracy.
Source: §5 · p. 8
Abstract
from arXiv · showhide
With the success of ChatGPT and other similarly sized SotA LLMs, claims of emergent human like social reasoning capabilities, especially Theory of Mind (ToM), in these models have appeared in the scientific literature. On the one hand those ToM-capabilities have been successfully tested using tasks styled similar to those used in psychology (Kosinski, 2023). On the other hand, follow up studies showed that those capabilities vanished when the tasks were slightly altered (Ullman, 2023). In this work we introduce a novel dataset of 68 tasks for probing ToM in LLMs, including potentially challenging variations which are assigned to 10 complexity classes. This way it is providing novel insights into the challenges LLMs face with those task variations. We evaluate the ToM performance of four SotA open source LLMs on our dataset and the dataset introduced by (Kosinski, 2023). The overall low goal accuracy across all evaluated models indicates only a limited degree of ToM capabilities. The LLMs' performance on simple complexity class tasks from both datasets are similar. Whereas we find a consistent tendency in all tested LLMs to perform poorly on tasks that require the realization that an agent has knowledge of automatic state changes in its environment, even when those are spelled out to the model. For task complications that change the relationship between objects by replacing prepositions, we notice a performance drop in all models, with the strongest impact on the mixture-of-experts model. With our dataset of tasks grouped by complexity we offer directions for further research on how to stabilize and advance ToM capabilities in LLM.
1 Introduction
Prior studies report ToM-like performance on simple tasks but raise questions about whether it survives task variations. This work introduces a benchmark to systematically probe those variations in LLMs.
1 Introduction
- Simple psychology-style tasks have suggested ToM in LLMs, but performance reportedly vanishes under slight task variations, and a large public dataset for systematic study was lacking.The open question is whether models learned how agents’ minds work or merely task-associated linguistic patterns.Source: §1 · p. 1
- The paper creates a manually crafted benchmark of unexpected-content and unexpected-transfer tasks across 10 complexity classes and evaluates it on four state-of-the-art LLMs.The authors argue that identifying challenging complications may help characterize model ToM behavior and guide further research.Source: §1 · pp. 1–2
1.1 Related Work
- Related work includes FANToM’s dynamic social interactions, ToMBench’s multiple-choice cognition tasks, and Shapira et al.’s six tasks probing different aspects of ToM.The paper frames its dataset as enabling systematic study of complications discussed by Ullman (2023), beyond the few examples publicly available.Source: §1 · p. 1, §1.1 · p. 2
2 Methodology
The methodology builds a manually varied ToM dataset, prompts four open-source models, and evaluates their answers at both scene and whole-scenery levels. Its task variations span ten complexity classes, including five added by this work.
2.1 Overview
- The evaluation prompts four state-of-the-art open-source models with machine-evaluable instructions, extracts their final answers, and reports overall and complexity-class performance.The dataset also records the protagonist’s belief after each sentence, which the authors identify as potentially useful for future CoT reasoning and faithfulness research.Source: §2.1 · p. 2
2.1.1 Dataset Creation and Outline
- Seven stage-play settings yield 68 sceneries and 16 scenes per scenery, producing 1,088 scenes with correct solutions.Sceneries are task variations; some complexity classes do not apply to every stage, and the construction adapts object swaps and belief or world-state questions.Source: §2.1.1 · p. 3
2.1.2 Complexity Classes
- The authors add five complexity classes to Ullman (2023), including automatic change knowledge, unrelated information, induction from baseline, untrustworthy testimony, and conclusion from sentiment.Automatic change knowledge requires inferring a world-state change independent of the protagonist’s actions, even though the process is described in the task.Source: §2.1.2 · pp. 3–4
2.2 Models and Inference
- The evaluation uses Llama-2-70b, Vicuna-33b, Yi-34B-Chat, and Mixtral-8x7B-Instruct, with temperature 1.0 and output limits scaled to input length.Mixtral is described as a mixture-of-experts model comprising eight 7B-parameter models.Source: §2.2 · p. 4
2.3 Evaluation Metrics
- Turn accuracy counts correctly answered scenes, while goal accuracy requires every scene in a scenery to be answered correctly.Answers are extracted, standardized to accommodate spelling and spacing variation, then checked using string inclusion while excluding the wrong solution.Source: §2.3 · p. 4
3 Results
Across the four models, turn accuracy is better than coin toss, but most models usually fail to answer every question about a scene correctly. Performance varies across complexity classes and between the two evaluated datasets.
3 Results
- Turn accuracy is better than coin toss for all four models, but most models usually fail to answer every question about a scene correctly.The reported overall goal accuracy is above coin-toss level.Source: §3 · pp. 4–5, Figure 2 · p. 5, Figure 3 · p. 6
3.1 Llama-2-70-b-chat-hf
- Llama 2 achieves approximately 70% turn accuracy overall, while its goal accuracy is zero in most complexity classes.Automatic change knowledge has 53.75% turn accuracy; automatic state change has the highest goal accuracy at 20%.Source: Table 1 · p. 5, §3.1 · p. 6, Table 2 · p. 6
3.2 Vicuna-33b-v1.3
- Vicuna has 58.00% overall turn accuracy and the lowest goal accuracy among the evaluated models.Its turn accuracy ranges from 38.75% for automatic state change to 69.79% for induction from baseline.Source: Table 1 · p. 5, Table 2 · p. 6, §3.2 · p. 7
3.3 Mixtral-8x7B-Instruct-v0.1
- Mixtral records 68.47% overall turn accuracy and 1.47% overall goal accuracy.Automatic change knowledge is its weakest turn-accuracy class at 48.75%.Source: Table 1 · p. 5, Table 2 · p. 6, §3.3 · p. 7
3.4 Yi-34B-Chat
- Yi records 72.89% overall turn accuracy and 4.41% overall goal accuracy.It is second-best in overall turn accuracy, and its goal accuracy is on the same level as Llama 2's.Source: Table 1 · p. 5, Table 2 · p. 6, §3.4 · p. 7
3.5 Dataset Baseline
- All four models perform slightly better on this paper’s dataset than on the dataset used in Kosinski (2023).Yi has the largest reported performance difference, almost 10%; only Yi records goal-accuracy success on this dataset, at 14%.Source: §3.5 · p. 7
4 Analysis
Across the tested models, turn accuracy exceeded chance, but goal accuracy remained consistently low, indicating no robust Theory of Mind. Automatic-change knowledge and preposition replacement were especially challenging, while model size alone did not explain performance.
4 Analysis
- Goal accuracy was consistently low across complexity classes, suggesting models learned linguistic patterns but lacked robust Theory of Mind.Llama2-70B, Mixtral-8x7B, and Yi-34B exceeded the 1.53 · 10−5 coin-toss baseline with overall goal accuracy between 1.47% and 4.41%.Source: §4 · pp. 7–8
- All models showed an impressive performance drop on automatic-change knowledge, a task requiring several steps to apply an agent’s mental model of environmental dynamics.The proposed explanation is that models must recognize the agent’s assumption, apply it, and predict the resulting state.Source: §4 · p. 8
- Vicuna and Mixtral sharply declined on preposition replacement, while Mixtral appeared better than Vicuna on transparent-container tasks.The authors suggest weaker spatial reasoning in smaller models as a possible explanation for Mixtral’s preposition-replacement result.Source: §4 · p. 8
- Yi-34B performed very close to Llama2-70B on turn and goal accuracy, while similarly sized Vicuna-33B performed significantly worse.The authors conclude that model size alone does not tell much about ToM capabilities.Source: §4 · p. 8
5 Discussion and Future Work
The authors regard the models’ ToM capabilities as limited, but low goal accuracy constrains comparisons among complexity classes. They point to larger-model evaluations, contamination-aware benchmarking, and alternative prompting as directions for further work.
5 Discussion and Future Work
- Low goal accuracy across tested models limits meaningful conclusions, although automatic state change appears especially challenging.The authors suggest Chain-of-Thought prompting as a promising approach for this class.Source: §5 · p. 8
- The tested models may be too small to show whether ToM capabilities emerge, so the authors recommend evaluating larger models, including GPT-family models.They also note that higher goal accuracy could enable finer-grained comparisons across complexity classes.Source: §5 · pp. 8–9
- Internet-scraped training data may contaminate benchmarks and overestimate model capabilities; the handwritten dataset may help assess models trained after Kosinski (2023) became public.Source: §5 · p. 9
- The authors identify experimenting with other prompting approaches as another way to gain insight into LLM ToM capabilities.Source: §5 · p. 9
6 Conclusion
The paper introduces an extensible benchmark for testing ToM task complications and evaluates four state-of-the-art LLMs. None showed robust ToM; low goal accuracy limits conclusions about complication difficulty, while automatic-change knowledge remains a notable challenge.
6 Conclusion
- The benchmark contains 1,088 scenes in ten complexity classes plus a non-complexity class addressing data draught, and is designed to systematically evaluate complications proposed by Ullman (2023).It extracts answers from elaborate LLM outputs and can be extended to longer chain-of-thought outputs.Source: §6 · p. 9
- None of the four evaluated models showed robust ToM, although they answered some subtasks correctly, aligning with results from Shapira et al. (2023) and FANToM.Source: §6 · p. 9
- Low overall goal accuracy prevents meaningful conclusions about the impact of complications on goal accuracy, and the tested models may have fewer parameters than those in Kosinski (2023).The authors say explanatory power about complication difficulty remains limited.Source: §6 · p. 9
- Automatic-change knowledge had low performance across models, and the authors identify chain-of-thought prompting and evaluation on larger models as promising next steps.Source: §6 · p. 9
A.1 Complexity Class Examples
The examples illustrate five complexity classes by introducing complications to agents’ beliefs about objects, testimony, and environmental changes.
A.1 Complexity Class Examples
- Automatic-change knowledge: Charlie’s green mangos ripen into red ones, but he does not know about the change.Source: §A.1 · p. 11
- Unrelated information: Aya encounters a bottle amid distracting details, while its contents remain unknown to her.Source: §A.1 · p. 11
- Induction from baseline: Esther infers a paper bag’s contents from its resemblance to bags used for sweets at a supermarket.Source: §A.1 · p. 11
- Untrustworthy testimony: a sister says a bottle contains sweet rather than spicy sauce, despite usually saying the opposite as a trick.Source: §A.1 · p. 11
- Conclusion from sentiment: a gift card says the present is from parents who know the recipient likes licorice and no other sweets or cookies.Source: §A.1 · p. 11
A.2 Full Prompt
The prompt asks the model to report agents’ final beliefs about object properties after a described scene. It specifies a JSON response and warns that agents do not automatically perceive objects or labels.
A.2 Full Prompt
- For each scene, report what each agent believes about an object’s location or contents after all events, using “unknown” when the agent does not know its location.The response should contain only the final information about the agents’ beliefs.Source: §A.2 · p. 11
- Return a JSON object with a top-level “final” entry and fields for the requested objects, containing each agent’s belief about the object’s property.The examples ask what Sam thinks is inside the box after each sentence.Source: §A.2 · pp. 11–12
- Unless explicitly stated, agents do not automatically see objects or read their labels.The example scene says Sam can see the box and reads its “bananas” label.Source: §A.2 · pp. 11–12