Source-linked AI summary
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, Maarten Sap
TL;DR
Existing ToM evaluations largely use passive narratives, while interactive conversations better test information-asymmetric mental-state tracking. FANTOM introduces a question-answer benchmark with multiple question types designed to expose illusory ToM. The benchmark is challenging for current LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.
Problem
Existing language-model ToM evaluations primarily use passive narratives, although ToM is important for understanding dynamic social interactions and narratives can contain reporting biases.
Method
FANTOM evaluates ToM through question answering over multiparty conversations with information asymmetry and multiple question types requiring the same underlying reasoning.
Results
Current LLMs perform significantly worse than humans on coherent ToM reasoning, including when using chain-of-thought reasoning or fine-tuning.
Takeaways & Limitations
The results support developing interaction-oriented ToM benchmarks that better reflect conversational use cases and test scenarios not encountered during training.
Takeaways & Limitations
FANTOM is limited to small talk on specific topics and a single relationship type in which participants have no prior knowledge of one another.
Abstract
from arXiv · showhide
Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering. Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that demand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANToM is challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.
1 Introduction
FANTOM introduces an interactive, conversation-based benchmark for evaluating whether language models track multiple characters’ beliefs under information asymmetry. Its questions test coherent ToM across formats and reveal substantial gaps between models and humans.
- FANTOM evaluates ToM in conversations, which expose interactions more directly than passive narratives and reduce susceptibility to reporting biases.The benchmark targets dynamic social interactions where effective communication requires understanding others’ mental states.
- 10K questions cover 256 multiparty conversations in which characters join and leave, creating distinct mental states through information asymmetry.Information shared while a character is absent remains inaccessible to that character after returning.
- FANTOM converts factual question-answer pairs into multiple belief questions that probe characters’ mental states at different reasoning levels.The design specifically targets coherent understanding rather than isolated answers.
- Illusory ToM refers to answering some questions correctly while failing others that require the same underlying ToM reasoning.Using multiple question types helps expose this inconsistent performance pattern.
- Models score more than 70% lower than humans on average across individual questions and the full question set.Chain-of-thought improves most models but does not substantially close the human-performance gap.
- FANTOM is presented as the first benchmark for conversation-based ToM evaluation in language-based models.The benchmark is released to support further discussion of LLM ToM evaluation.
FANTOM
FANTOM constructs conversational ToM evaluations around information asymmetry, psychological criteria, and repeated question formats. It uses factual answers to generate belief and answerability questions that test whether models maintain distinct mental states and avoid shortcut matching.
- The benchmark design targets natural social interactions, psychological prerequisites for ToM validation, and empirical considerations for evaluating LLMs.These design goals organize the construction of FANTOM.
- Conversations minimize reporting bias and align evaluation with real-world scenarios by presenting interactions without explicit hints about mental states.The authors contrast this with narratives that may include artifacts models can exploit.
- Non-merging: FANTOM uses information asymmetry to satisfy the non-merging criterion by keeping specific conversation information inaccessible to selected characters.The model receives the full conversation as an omniscient observer, while the character lacks information from their absence.
- Mentalizing: The benchmark discourages shortcut pattern matching by creating false answers with high word correlation to the input.This tests whether models reason about mental representations rather than rely on salient lexical overlap.
- FANTOM varies question formats, including free-form, multiple-choice, and yes-or-no questions, to reveal format-dependent inconsistencies in ToM performance.Reiterative questions apply different formats to the same input context.
- The dataset contains small-talk conversations where multiple characters join or leave as topics and subtopics change, producing natural information asymmetry.Explicit joining and leaving utterances mark when participants gain or miss information.
- Factual QA pairs: Factual question-answer pairs target inaccessible information, and each factual question has FULL FACT A and LIMITED FACT A answers based on different information access.The full answer uses preceding conversation during PersonX’s absence, while the limited answer uses only PersonX’s accessible conversation.
- ToM QA construction: ANSWERABILITY Q[LIST] asks which characters know a factual answer, requiring models first to determine the answer and then identify who can access it.This is explicitly defined as a two-step reasoning task.
4 Experiments
FANTOM evaluates conversational theory of mind across information-asymmetric interactions, revealing substantial gaps between model and human performance and exposing failures of coherent mental-state reasoning. Across question formats and contexts, models struggle especially when reasoning must remain consistent across multiple related questions.
- Experimental Setup: Thirteen instruction-tuned language models were evaluated against graduate-student human performance on FANTOM questions.The evaluation included GPT-4, ChatGPT, InstructGPT, Flan models, Falcon, Mistral, Zephyr, and Llama-2 Chat.
- Overall Results: Models scored significantly worse than humans on individual questions and the full question set, by more than 70% on average.Fine-tuning could produce scores comparable to humans on individual question types, but this did not extend to consistent multi-question performance.
- Chain-of-thought and Error Analysis: Chain-of-thought improved scores but did not close the human gap, and its error reduction came with increased exclusion of characters who were aware of the information.For list questions, models more often included unaware characters; for binary questions, false negatives were more frequent than in list questions.
- Comprehending Facts vs. Distinct Beliefs: Models often reproduced FACTQ-like answers for BELIEFQ[DIST.], indicating difficulty representing characters’ distinct perspectives despite substantial word overlap between question types.BELIEFQ[DIST.] uses incorrect answers with greater context overlap, testing whether models can avoid shortcut pattern matching and reason about inaccessible information.
- Overall Results: Models performed worse on free-response than choice questions, with many binary-choice scores remaining below or near the 50-point random baseline.The same free-response disadvantage also appeared for ANSWERABILITY and INFOACCESS questions across list and yes/no formats.
- Reasoning Complexity: Models performed consistently worse on ANSWERABILITY Q[Y/N] than INFOACCESS Q[Y/N], while list-question results showed no consistent corresponding pattern.ANSWERABILITY requires determining the factual answer and then identifying which characters can access it, adding a reasoning step beyond INFOACCESS.
- Context and Coherence: Performance decreased on full conversations compared with short relevant contexts because models had to identify relevant information before reasoning about mental states.Models also performed much better on BELIEFQ[CHOICE] than on its ANSWERABILITY and INFOACCESS prerequisites, then dropped sharply when evaluated across all question types.
5 Related Work
Prior ToM benchmarks mainly test models with narrative-based false-belief, emotion, and mental-state reasoning, while quantitative studies find that LLMs remain below human-level ToM across benchmarks.
- Existing ToM benchmarks primarily evaluate models using narratives describing situations and false beliefs.
- Other benchmarks assess social understanding through faux pas, emotions, and mental states presented in narratives.
- Quantitative investigations report that LLMs have not achieved human-level ToM across various benchmarks.
- Few-shot examples, chain-of-thought prompting, and specific inference methods can improve LLM performance on ToM tasks.
6 Conclusion & Discussion
FANTOM introduces conversation-based, information-asymmetric ToM evaluation with varied belief questions and finds that current LLMs struggle with coherent reasoning and consistency. The benchmark also exposes reporting biases and motivates interaction-oriented evaluation.
- FANTOM introduces a benchmark for testing language models’ ToM in conversational interactions through question answering.
- Its conversations use information asymmetry and participant entry or exit to create distinct mental states, while multiple belief-question types probe illusory ToM.
- Current LLMs perform significantly worse than humans on FANTOM even with chain-of-thought reasoning or fine-tuning.
- Models often fail to give consistent answers to questions requiring the same underlying ToM reasoning.
- Models tend to rely on conditioned information and context overlap instead of distinguishing information accessible or inaccessible to each agent.
- The benchmark is publicly available to encourage further research on interaction-oriented ToM evaluation.
7 Limitations
FANTOM’s scope is limited to small-talk conversations on specific topics, unfamiliar participants, language-based models, and automatically generated content that may contain offensive material.
- FANTOM covers small-talk conversations on specific topics and only one relationship type in which participants lack prior knowledge of one another.
- The benchmark does not cover the broader dynamics introduced by relationships such as family, friendship, or co-worker ties.
- The evaluation focuses solely on language-based models, leaving multimodal ToM reasoning for future work.
- Because conversations are generated with LLMs, they may contain offensive content; casual topics and manual crowdworker validation are used to reduce this risk.
8 Societal and Ethical Considerations
The paper uses ToM as a social-reasoning construct without attributing consciousness or genuine mental-state understanding to AI models. Its experiments instead find no coherent ToM reasoning and describe reliance on word correlations.
- The authors clarify that ToM is used to study AI limitations in social reasoning rather than to promote anthropomorphism.
- The paper states that AI models do not possess subjective consciousness or true understanding of intentions, beliefs, or desires.
- Experiments demonstrate that current LLMs do not exhibit coherent ToM reasoning and primarily rely on word correlations.
FANTOM Construction
FANTOM’s full question-set examples are provided in Tables 5 and 6.
- Tables 5 and 6 contain full examples of FANTOM question sets.
- The benchmark’s question-set examples are documented in Table 5 and Table 6.
- Readers can find complete FANTOM question-set examples in Tables 5 and 6.
A.1 Generating Conversations with Information Asymmetry
FANTOM generates multiparty conversations from predefined topics and subtopics, with characters joining or leaving to create information asymmetry. GPT-4 then extracts inaccessible information while the conversation history is continued across participation changes.
- Generating Conversations with Information Asymmetry: Conversations are generated from predefined main-topic subtopics, shuffled before each subtopic-specific conversation is produced.
- Generating Conversations with Information Asymmetry: Scripts begin with two or three characters and can include up to five simultaneous participants.
- Generating Conversations with Information Asymmetry: Characters are randomly selected to join or leave, using predefined leaving reasons and prior conversation history to continue the dialogue.
- Generating Conversations with Information Asymmetry: When a character rejoins, GPT-4 extracts information shared while that character was absent and produces a summary of the preceding context.
- Generating Conversations with Information Asymmetry: A sample conversation illustrates how participants discuss workouts through exchanges between Sabrina and Anna.
- Generating Conversations with Information Asymmetry: Table 5 provides a sample from the conversation-generation material.
A.2 Generating Factual QA Pairs
FANTOM constructs factual question-answer pairs from information inaccessible to a participant, then converts them into belief questions with omniscient and participant-centered answers.
- Generating Factual QA Pairs: The pipeline begins by constructing factual question-answer pairs related to inaccessible information.
- Generating Factual QA Pairs: GPT-4 generates three non-yes-or-no FACTQs from each inaccessible-information summary.
- Generating Factual QA Pairs: Each FACTQ receives a FULL FACT A from the absent participant’s preceding conversation and a LIMITED FACT A from the participant’s accessible conversation.
- Generating Factual QA Pairs: FACTQs are converted into first- or second-order theory-of-mind questions about a character’s beliefs.
- Generating Factual QA Pairs: The full and limited factual answers become OMNISCIENT-VIEW BELIEF A and PERSONX-CENTRIC BELIEF A, respectively.
A.4 Evaluation for ANSWERABILITY Q[Y/N] and INFOACCESS Q[Y/N]
FANTOM evaluates yes-or-no answerability and information access by parsing standardized affirmative and negative expressions, alongside comparative benchmark statistics.
- Evaluation for ANSWERABILITY Q[Y/N] and INFOACCESS Q[Y/N]: Pattern matching extracts yes-or-no answers from model responses.
- Evaluation for ANSWERABILITY Q[Y/N] and INFOACCESS Q[Y/N]: Responses such as “yes,” “knows,” and “true” are mapped to affirmative answers.
- Evaluation for ANSWERABILITY Q[Y/N] and INFOACCESS Q[Y/N]: Responses such as “no,” “does not know,” and “false” are mapped to negative answers.
- Evaluation for ANSWERABILITY Q[Y/N] and INFOACCESS Q[Y/N]: Table 8 compares the basic statistics of FANTOM and ToMi.
B Experiments
The experiments compare human performance with diverse language-model baselines on FANToM, including zero-shot, chain-of-thought, and fine-tuned settings. Results indicate that most evaluated models perform poorly relative to the benchmark’s random baselines.
- Human performance evaluation: Eleven student volunteers solved 32 question sets without tutorials, examples, or extra instructions beyond those given to models.
- Baseline models: The evaluated baselines include proprietary GPT models and open-source instruction-tuned encoder-decoder and decoder-only models.
- Results of other models: Most other large language models score low on FANToM, with random-baseline scores of 50 for BELIEFQ[CHOICE], ANSWERABILITY Q[Y/N], and INFOACCESS Q[Y/N].
- Fine-tuning details: Fine-tuning Flan-T5-XL used early stopping after validation loss increased following the first epoch, while special completion tokens limited over-generation.
- Benchmark materials: The benchmark uses multi-character conversations and predefined reasons for characters leaving, as illustrated by conversational samples and supporting tables.
- Benchmark comparisons: FANToM and ToMi are compared through reported dataset statistics, while Table 9 reports zero-shot results for humans and language models under matched instructions.