Source-linked AI summary
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, Vered Shwartz
TL;DR
The paper asks whether LLMs possess robust Neural Theory of Mind, addressing conflicting prior findings and limited evaluation scope. It evaluates models across six ToM tasks and probing methods, finding certain abilities but poor robustness, especially under adversarial variation. The authors therefore caution against inferring ToM from anecdotes, narrow benchmarks, or human-designed psychological tests.
Problem
Conflicting findings and limited prior testing leave unresolved whether LLMs possess robust Neural Theory of Mind.
Method
The paper evaluates LLMs across six ToM tasks, compares probing methods, and adds adversarial and true-belief examples to test robustness.
Results
LLMs exhibit certain N-ToM abilities, but performance varies across tasks and decreases on adversarial examples, consistent with reliance on shallow heuristics.
Takeaways & Limitations
ToM claims should be based on broad, robust evaluations rather than anecdotes, a few benchmarks, or human-designed psychological tests.
Takeaways & Limitations
The study’s datasets are limited in scope and size, and some examples are ambiguous or cause formatting and answerability problems for models.
Abstract
from arXiv · showhide
The escalating debate on AI's capabilities warrants developing reliable metrics to assess machine "intelligence". Recently, many anecdotal examples were used to suggest that newer large language models (LLMs) like ChatGPT and GPT-4 exhibit Neural Theory-of-Mind (N-ToM); however, prior work reached conflicting conclusions regarding those abilities. We investigate the extent of LLMs' N-ToM through an extensive evaluation on 6 tasks and find that while LLMs exhibit certain N-ToM abilities, this behavior is far from being robust. We further examine the factors impacting performance on N-ToM tasks and discover that LLMs struggle with adversarial examples, indicating reliance on shallow heuristics rather than robust ToM abilities. We caution against drawing conclusions from anecdotal examples, limited benchmark testing, and using human-designed psychological tests to evaluate models.
1 Introduction
The paper addresses conflicting claims about LLMs’ Neural Theory of Mind by evaluating them across diverse tasks and probing methods. It finds some N-ToM abilities, but they are not robust and may reflect shallow heuristics.
- ToM concerns understanding that others’ thoughts, beliefs, and emotions can differ from one’s own.
- Robust N-ToM metrics are needed because apparently illusory machine abilities raise the stakes of claims about machine-like intelligence.
- Prior studies reached opposing conclusions about LLM ToM, while simple changes to questions were reported to break their performance.
- LLMs show certain N-ToM abilities, but their performance is not robust and declines on datasets designed to prevent annotation artifacts and on adversarial examples.
- The authors caution against relying on anecdotes, a few benchmarks, or psychological tests designed for humans when assessing model ToM.
2 Background: ToM and Clinical Tests
This section defines Theory of Mind and introduces clinical tasks used to assess false-belief understanding and faux pas recognition. It also situates these tests within broader NLP work on modeling mental states and social reasoning.
- Theory of Mind involves understanding mental states, beliefs, desires, intentions, and emotions of oneself and others.
- False-belief tests present a character with partial information, causing a mistaken belief that differs from reality known to the listener.
- False Belief Test: The Sally–Anne Test asks where Sally will look for a relocated marble, distinguishing her false belief from the marble’s actual location.
- False Belief Test: Second-order belief questions ask what one character thinks another character will believe or do.
- False Belief Test: The Smarties Test examines unexpected contents when a protagonist is unaware of a container’s actual contents because of false labeling.
- Faux Pas Test: A faux pas occurs when someone says something inappropriate without considering the listener’s knowledge or feelings, often producing unintended negative consequences.
- Faux Pas Test: The original faux pas test measures detection, identification, comprehension, and false-belief understanding across 10 stories.
- NLP research has modeled mental-state and emotion detection, humorous responses, and datasets testing whether models possess ToM.
3 Data
The paper evaluates six ToM-related tasks and introduces dataset variants designed to test robustness against artifacts and spurious correlations.
- The evaluation uses all datasets listed in Table 1, including existing ToM benchmarks and newly constructed variants.
- Existing Benchmarks & Variants: ToMi contains over 1,000 synthetic stories testing memory, reality, and first- and second-order false belief.
- Existing Benchmarks & Variants: ToMi’ reformulates 30 ToMi stories and 180 questions as sentence-completion tasks while preserving their meaning.
- Creation of Adv-CSFB: Adv-CSFB contains 110 unexpected-contents examples and 73 unexpected-transfer examples concerning reality and protagonists’ beliefs.
- Creation of Adv-CSFB: Adv-CSFB combines false-belief, true-belief, and adversarial examples to evaluate whether models rely on shallow patterns.
4 Experiments & Results
The experiments evaluate 15 LLMs across ToM datasets and probing methods. Performance varies substantially by dataset and probing technique, and no model robustly exhibits ToM across all datasets.
- Meta-Evaluation: The study evaluates 15 LLMs in a zero-shot setting across all considered ToM-related datasets against a most-frequent-class baseline.
- Probing Techniques: 7% is the GPT-4 accuracy difference between MC-probing and CoT-probing on SocialIQa, showing that probing method affects performance.
- Meta-Evaluation: 96% accuracy is achieved by flan-t5-xxl on TriangleCOPA, while the best model on FauxPasEAI performs below the most-frequent-class baseline.
- Meta-Evaluation: Best-model performance appears correlated with dataset age, with older datasets receiving higher performance.
- Meta-Evaluation: No model robustly exhibits ToM on all datasets, although some models show strong abilities on individual datasets.
- Probing Techniques: CoT-probing generally improves performance, but reasoning can occasionally produce erroneous conclusions.
5 Clever Hans vs. Generalized Reasoning
Across N-ToM tasks, newer LLMs perform strongly on selected familiar categories but fail sharply on adversarial or altered versions, indicating non-robust social reasoning.
- Evaluation focus: The evaluation tests whether apparent N-ToM performance reflects general reasoning or memorization and shallow heuristics.The study compares varied question types and probing setups across ToM datasets.
- Spurious correlations: The discrepancy between ToMi’ and ToM-k suggests models recognize Sally-Anne patterns rather than generalize to exact mental-state reasoning.ToMi includes omissions, duplications, and second-order questions, while ToM-k contains simple positive examples.
- Adversarial robustness: LLMs do not exhibit robust performance across Adv-CSFB categories, with later models excelling in some categories while failing on others.The analysis uses MC-probing on belief-focused questions and reports reproducible results at temperature 0.
- Adversarial robustness: 0% is GPT-4’s accuracy on both in→on and transparent access, down from 71.4% and 66.7% by davinci-002.Other-person accuracy also declined from 93.8% by davinci-002 to 68.8% by GPT4.
- Spurious correlations: GPT-3.5 reaches 81% on ToMi but only 46% on false-belief questions, close to random performance.ToMi includes questions about facts versus beliefs, including true and false beliefs.
- Spurious correlations: GPT-4’s SocialIQa accuracy is 79% overall and 74.5% on questions about others, showing a smaller but measurable question-type gap.The comparison concerns questions focused on the main character versus others.
6 Summary of Findings and Insights
The paper finds that N-ToM performance varies substantially across benchmarks and that adversarial changes expose reliance on shortcuts. It therefore cautions against inferring robust mental-state understanding from narrow success.
- Summary of findings: Adv-CSFB was created to test whether models solve ToM questions for the right reasons or rely on surface cues and shallow heuristics.The benchmark targets adversarial examples in unexpected-content and unexpected-transfer tasks.
- Summary of findings: The results show that models lack robust N-ToM abilities despite successfully solving some datasets and performing poorly on others.The paper reports failures on small variations of original tasks, including for GPT-4.
- Interpretation: Performance gaps across question types suggest reliance on shortcuts, heuristics, and spurious correlations that can lead models astray.The paper discusses reporting bias and a tendency toward surprising rather than unsurprising continuations as possible contributors.
- Interpretation: Instruction-following training might make newer models treat all details as important, causing errors when labels are irrelevant under transparent or unreadable conditions.The paper gives unexpected-content examples where the protagonist cannot read the label or the container is transparent.
- Interpretation: The reassessment of main-character versus others questions revisits whether the observed difference reflects centering theory.Sap et al. suggested that texts tend to focus on describing a single protagonist.
ELIZA Effect & Anecdotal Generative vs. Automatic Large-Scale Multiple-Choice Testing
Generative demonstrations can trigger anthropomorphic judgments, so the paper favors cautious interpretation and broader, more direct evaluation of LLM social reasoning.
- ELIZA Effect: High-quality generated text can produce the ELIZA effect, leading people to infer human-like intelligence from fluent model outputs.The paper links this risk to models being designed to generate text that appears high-quality to human observers.
- Anecdotal Generative Testing: Epistemic humility can yield multiple rationales without a committed answer, allowing confirmation bias to make partial responses appear correct.The paper warns that observers may notice a correct answer and rationale even when the model does not clearly answer.
- Human Psychological Tests: Success on a human-designed psychological instrument does not support the same conclusions for LLMs as it does for humans.Alternative explanations, including heuristic reliance, must be considered when models succeed on such tests.
- Machine Intelligence and Anthropomorphism: The paper cautions that “neural ToM” should not be treated as evidence of AI cognition or as an endorsement of human-like social intelligence.This caution reflects the risk of anthropomorphizing non-human animals and computers.
- Benchmark Interpretation: A model’s success on one ToM example or task is not sound proof that it possesses ToM in general.The paper argues that future benchmarks should test varied ToM aspects and assess LLMs directly rather than relying on clinical tests for humans.
- Automatic Multiple-Choice Testing: Aggregated benchmark scores can obscure performance differences across questions with different types and complexities.The paper discusses pairing difficult and easy questions through a joint score as one way to expose these differences.
- Interpreting Model Abilities: Claims that ToM emerged spontaneously cannot be evaluated reliably without access to training data because potential data leakage cannot be quantified.Related training exposure, online test descriptions, and user interactions are possible alternative explanations.
7 Conclusion
The study concludes that contemporary LLMs show enhanced but limited ToM abilities, which are often non-robust and sometimes rely on simple heuristics rather than generalized reasoning.
- 7 Conclusion: Contemporary LLMs demonstrate an enhanced yet limited degree of Theory of Mind abilities.Their performance does not establish broad or unrestricted ToM competence.
- 7 Conclusion: LLM ToM abilities are often not robust, with some evidence of over-reliance on simple heuristics rather than robust generalized reasoning.This conclusion is based on the paper’s research and replication studies.
Limitations
The study’s evidence is bounded by limited dataset scope, ambiguity, possible test-data leakage, and evaluation of only a subset of available LLMs.
- Benchmark Scope and Human Ambiguity: The datasets were limited in scope and size, although Theory of Mind is required across most human interaction.Some dataset items could also be ambiguous because of missing context or inherent ambiguity.
- Data Leakage: OpenAI acknowledged that GPT-4 was trained on test data from BIGBench, creating a documented data-leakage concern.The paper notes that this issue complicates interpretation of benchmark performance.
- Benchmark Administration: Some LLMs refused ambiguous questions or failed to produce the required multiple-choice format, and probability distributions were unavailable for all models.These issues affected MC-probing and may require better instructions or answer-mapping methods.
- Limited Text-Only LLMs: The experiments covered only a limited number of accessible LLMs and did not explore the full range of currently available models.Future work could examine additional LLMs and multimodal models.
Ethical Statement
The ethical statement addresses dataset content and model-output risks by using non-offensive evaluation materials and not recording generated responses.
- Data: The publicly available narratives were reviewed by the authors to ensure that they did not contain offensive content.This review applied to all existing and newly created datasets used in the study.
- Models: Although LLMs may generate offensive content for some inputs, the study used non-offensive prompts and did not record model responses.The models were used for evaluation only.
8 Appendices
The appendices document task examples, prompting, decoding settings, and model-specific implementation details used in the evaluation. They also illustrate an unexpected-transfer failure in which ChatGPT handles a false-belief question but fails a trivial true-belief question.
- Examples: ChatGPT solves a more complex false-belief unexpected-transfer question while failing a trivial true-belief question in a ToMi example.The passage attributes this pattern likely to exposure to the Sally–Anne task.
- Examples: The appendix illustrates Ullman’s variations for the unexpected contents and unexpected transfer tasks.Figures 6 and 7 provide the corresponding illustrations for examples in ToM-k.
- Prompting: The evaluation generally uses MC-probing by concatenating each test with all possible answers and instructing the model to choose one option.Table 6 exemplifies the prompt format for each task.
- Implementation: The appendix reports implementation details for FlanT5, FlanUl2, GPT, ChatGPT, and AI21 models using transformer or completion APIs.The listed configurations include model names, generation functions, token limits, and temperature settings.
8.4 Complete Results
The complete results show discrepancies in reported scores, differences between reading-comprehension and Theory-of-Mind performance, and inconclusive evidence about test-set contamination. Joint scoring further exposes failures on complex questions even when models answer story questions correctly.
- Complete Results: 85% or 90% was obtained for GPT-3.5 on Unexpected Transfer, yielding an 85% or 87.5% two-task average versus the reported 93%.The discrepancy arose from two clear mispredictions and one borderline-format prediction.
- Complete Results: Comparisons across 11 random 100-question subsets found no significant performance differences between standard and secret SocialIQa test sets for GPT-3.5 and GPT-4.The result is inconclusive about whether the models were trained on the normal test set.
- Complete Results: Table 7 reports LLM accuracy against a most-frequent-class baseline, using question-level and story-level metrics for Adv-CSFB and FauxPas-EAI.Table 8 provides zero-shot subset comparisons for ToMi’, with percentages reported throughout.
- Complete Results: GPT-3.5 answers simple reading-comprehension questions more accurately than first- and second-order Theory-of-Mind questions in ToMi’.The complete ToMi’ results support a gap between comprehension and questions requiring ToM skill.
- Complete Results: The joint score counts a story as correct only when all questions from that story are answered correctly, unlike the simple average score.This scoring setup reveals failures on complex questions despite correct answers about the story.