PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs

Maarten Sap, Ronan LeBras, Daniel Fried, Yejin Choi

arXiv:2210.13312v2cs.CLcs.AI

TL;DR

The paper asks whether large language models possess social intelligence and Theory of Mind, and evaluates GPT-3 on social reasoning and mental-state tasks. The models struggle on both benchmarks, leading the authors to examine pragmatic, architectural, and training-related explanations and alternatives to scale alone.

  • Problem

    Whether modern large language models exhibit social intelligence and Theory of Mind remains an open question despite their strong performance on many language tasks.

  • Method

    The paper evaluates large pretrained language models with SOCIALIQA and TOMI, then interprets the results through theories of language pragmatics and discusses person-centric alternatives.

  • Results

    GPT-3 struggles across both Theory-of-Mind tasks, reaching 55% on SOCIALIQA and 55–60% on TOMI mental-state questions versus 90–100% on factual questions.

  • Takeaways & Limitations

    The findings suggest that social intelligence and Theory of Mind do not emerge automatically in large pretrained language models and that increasing scale alone may not be most effective.

  • Takeaways & Limitations

    The study evaluates GPT-3 through an API and uses TOMI’s automatically constructed stories, motivating confirmation with larger models and more naturalistic settings.

Abstract

from arXiv · show

Social intelligence and Theory of Mind (ToM), i.e., the ability to reason about the different mental states, intents, and reactions of all people involved, allow humans to effectively navigate and understand everyday social interactions. As NLP systems are used in increasingly complex social situations, their ability to grasp social dynamics becomes crucial. In this work, we examine the open question of social intelligence and Theory of Mind in modern NLP systems from an empirical and theory-based perspective. We show that one of today's largest language models (GPT-3; Brown et al., 2020) lacks this kind of social intelligence out-of-the box, using two tasks: SocialIQa (Sap et al., 2019), which measures models' ability to understand intents and reactions of participants of social interactions, and ToMi (Le et al., 2019), which measures whether models can infer mental states and realities of participants of situations. Our results show that models struggle substantially at these Theory of Mind tasks, with well-below-human accuracies of 55% and 60% on SocialIQa and ToMi, respectively. To conclude, we draw on theories from pragmatics to contextualize this shortcoming of large language models, by examining the limitations stemming from their data, neural architecture, and training paradigms. Challenging the prevalent narrative that only scale is needed, we posit that person-centric NLP approaches might be more effective towards neural Theory of Mind. In our updated version, we also analyze newer instruction tuned and RLFH models for neural ToM. We find that even ChatGPT and GPT-4 do not display emergent Theory of Mind; strikingly even GPT-4 performs only 60% accuracy on the ToMi questions related to mental states and realities.

1 Introduction

The paper investigates whether large language models exhibit social intelligence and Theory of Mind, evaluating social reasoning and mental-state inference with benchmark tasks. GPT-3 struggles substantially on both abilities, and the authors argue that scaling alone may not suffice.

  • Motivation: Theory of Mind requires reasoning about other people’s intents, reactions, and mental states in social situations.The paper illustrates this through emotional reactions and beliefs about objects’ locations.
  • Approach: The study evaluates GPT-3 on social commonsense, emotional intelligence, and reasoning about people’s mental states and realities.These abilities are assessed with SOCIALIQA and TOMI question-answering tasks.
  • Results: 55% accuracy was achieved by the best few-shot GPT-3 setup on SOCIALIQA, more than 30% below human performance.The model performed better when reasoning about protagonists than secondary participants.
  • Results: 60% accuracy was GPT-3’s peak on TOMI questions about participants’ mental states, versus 90–100% on factual questions.This contrast indicates a substantial gap between mental-state and factual reasoning in the evaluated task.
  • Implications: The authors attribute these shortcomings partly to training-text pragmatics and propose person-centric and interaction-based directions beyond simply increasing model scale.They specifically discuss interactional data selection, person-centric inductive biases, and interaction-based language learning.

2 Theory of Mind & Large LMs

Theory of Mind is framed as a human ability to infer others’ mental states and predict their actions, closely related to social intelligence and language use. The paper motivates testing whether large language models possess these abilities and highlights limitations in interpreting social perspectives.

  • Evaluation: Figure 2 organizes SOCIALIQA accuracy by language-model type, model size, and the number of few-shot examples k.The figure is intended to compare performance across these three conditions.
  • Theory of Mind: Theory of Mind involves ascribing and inferring others’ mental states and predicting their likely actions.It supports navigating social situations from everyday interactions to complex negotiations.
  • Theory of Mind: Social intelligence enables people to navigate and understand social situations, while pragmatics links effective communication to reasoning about interlocutors’ mental states.The paper presents this relationship as a theoretical framing rather than a settled causal direction.
  • Theory of Mind: The paper notes that the direction of the relationship between language development and Theory of Mind remains debated.Some researchers argue language enables Theory-of-Mind-like abilities, while others propose that Theory of Mind can precede language.
  • Evaluation: Figure 3 compares GPT-3-DAVINCI’s SOCIALIQA accuracy when questions concern the situation’s main agent versus other participants.This isolates whether the participant receiving the reasoning focus affects performance.

3 SOCIALIQA: Do LLMs have Social Intelligence and Social Commonsense?

SOCIALIQA evaluates whether language models can reason about participants’ intents, reactions, and other social consequences. GPT-3 performs substantially below humans, with only marginal gains from larger models or additional examples.

  • Benchmark and probing: SOCIALIQA tests nine social reasoning dimensions across agent-focused and other-participant questions in 1,954 three-way QA tuples.Each instance contains a context, question, and three answer choices.
  • Results: GPT-3 performs more than 20% below models fine-tuned on SOCIALIQA, while larger GPT-3 models provide only small gains.The comparison uses the SOCIALIQA fine-tuned-model result reported by Lourie et al. (2021).
  • Results: Increasing examples beyond k=10 yields only a 1% improvement for GPT-3-DAVINCI, suggesting performance plateaus or grows logarithmically.
  • Results: GPT-3-DAVINCI performs better on agentcentric than other-oriented questions and often confuses which participant the question targets.The paper illustrates this error with predictions that attribute one participant’s likely action to another.

4 TOMI: Can LLMs Reason about Mental States and Realities?

The TOMI benchmark tests whether language models distinguish factual object locations from participants’ mental states and realities, including true- and false-belief reasoning. GPT-3 models perform substantially worse on MIND questions than FACT questions, and scaling or additional examples provide limited gains.

  • Benchmark and setup: TOMI uses English stories about moved objects, asking either factual-location questions or where participants think objects are located.The benchmark includes distractor actions and two possible answers: the original or final object location.
  • Benchmark and setup: MIND questions cover first- and second-order beliefs, while MIND-TB and MIND-FB distinguish participants present or absent during the object’s movement.FACT questions do not require Theory of Mind, whereas MIND questions require reasoning about participants’ mental states and realities.
  • Results: 60% accuracy is the best GPT-3 result on TOMI MIND questions, compared with 90–100% on factual questions.GPT-3-DAVINCI achieves the strongest performance among the tested GPT-3 models, while smaller models remain at or below 55% accuracy.
  • Results: Increasing the number of few-shot examples beyond k = 4 does not substantially improve TOMI performance, consistent with the findings on SOCIALIQA.The reported gains from scaling up GPT-3 are also very small.
  • Results: GPT-3-DAVINCI’s MIND-TB accuracy peaks at k = 4, then declines, while its performance on MIND-TB and MIND-FB is comparably poor for k > 8.The peak is attributed to defaulting to the most recently mentioned object location, which is correct for MIND-TB in those cases.

5 Discussion: Towards NLP with Neural Theory of Mind

The paper argues that large-pretrained language models do not automatically acquire social intelligence or Theory of Mind, and examines data, architectural, and training-related reasons for this limitation. It proposes socially grounded data, person-centric inductive biases, interactive learning, and improved evaluation as directions toward neural Theory of Mind.

  • Large-pretrained language models do not automatically develop social intelligence or Theory of Mind despite successes on tasks that may appear socially demanding.The paper contrasts these limitations with GPT-3’s ability to generate emotionally coherent stories and social commonsense knowledge.
  • The Pragmatics of “Static” Text: Static text may limit social learning because it omits shared knowledge, communicative intent, alternative word choices, and relationships among multiple characters.The authors connect these omissions to reporting bias, pragmatic inference, and coreference difficulties.
  • Beyond static text as training data?: Socially grounded text could support more efficient Theory of Mind learning, but such data are difficult to find naturally and costly to create.A proposed compromise is joint or sequential training on static text plus commonsense, socially grounded, or interactional data.
  • Person-centric neural inductive biases?: Person-centric architectures could improve entity awareness, but Theory of Mind requires richer social grounding than coreference information and remains difficult to obtain at LLM scale.The paper points to unsupervised inductive biases and neuroscience-inspired representations as possible directions.
  • Interactive and experiential grounding?: Interactive learning and multi-agent communication can enable some Theory of Mind capabilities, yet simulation-trained models often struggle to generalize beyond their environments.The authors therefore suggest hybrid approaches combining static-text pretraining with interactive learning.
  • Probing and evaluating TOM: Better benchmarks should measure specific social abilities while minimizing annotation artifacts and spurious correlations.The paper presents evaluation development as a parallel priority to improving models.

6 Conclusion

The conclusion reports that large language models struggle substantially with social intelligence and Theory of Mind abilities, even though they perform impressively on other language tasks. It frames these results as motivation to investigate data, architecture, and training changes for socially intelligent NLP systems.

  • GPT-3’s SOCIALIQA performance lags human performance by >30%, while TOMI mental-state accuracy reaches only 55–60% versus 90–100% on factual questions.
  • The paper argues that social-situation and false-belief reasoning are necessary but not sufficient aspects of Theory of Mind.
  • The authors examine pretraining through pragmatics and discuss person-centric architectures, data selection, and training objectives as possible directions.

7 Limitations

The paper’s evidence is bounded by its model access, automatically constructed evaluation data, benchmark scope, and focus on English-language systems. It also notes that the two benchmarks operationalize only limited aspects of the broader Theory of Mind construct.

  • The study evaluates GPT-3 through an API rather than larger or open-source models, so its findings require confirmation on those systems.
  • TOMI’s automatically generated stories may contain plausible but atypical object-location combinations that influence answering.
  • The authors acknowledge that models could exploit surface patterns or spurious correlations in the two datasets.
  • SOCIALIQA and TOMI measure only specific aspects of broader social intelligence and Theory of Mind; solving them would not establish that a model has Theory of Mind.
  • The analysis focuses on English models and benchmarks, although false-belief abilities can be influenced by language structure and grammar.

A.1 Data Preprocessing

The preprocessing procedure uses publicly available SOCIALIQA training and development data and normalizes casing for consistency across contexts, questions, and answers.

  • The researchers downloaded the SOCIALIQA training and development datasets from the publicly available website.
  • The dataset version retains ATOMIC dimensions and identifies whether each question focuses on the agent or another character.
  • They normalize casing so contexts, questions, and answers begin with a capital letter when needed.

A.2 Further SOCIALIQA results

Further SOCIALIQA analyses show that GPT-3’s performance varies substantially across reasoning dimensions and remains below human performance even for larger proprietary models.

  • The appendix reports further SOCIALIQA results and presents comparisons with BIG-G and PaLM.
  • GPT-3-DAVINCI struggles most with Need questions but performs relatively better on Intent and Effect questions.
  • Additional analyses compare GPT-3-DAVINCI across nine reasoning dimensions using a 35-shot setup.
  • 73% peak accuracy leaves PaLM below humans, while BIG-G reaches only 45% peak accuracy on SOCIALIQA.

B.1 Data Preprocessing

The TOMI data were generated, manually filtered, balanced across reasoning types, and released as a preprocessed development dataset.

  • The TOMI generator initially produced 5994 training and 5994 development stories.
  • After removing questions answered from an omniscient perspective, 5190 training and 5170 development stories remained.
  • The final development set used stratified sampling to balance six FACT and MIND question types.
  • The preprocessed TOMI development dataset was released publicly.

B.2 Further TOMI results

Further TOMI analyses separate factual from mental-state reasoning, examine question order and recency effects, and describe the few-shot probing procedure.

  • The appendix includes additional TOMI figures comparing model sizes, few-shot examples, question order, and reasoning types.
  • GPT-3-CURIE and GPT-3-DAVINCI approach near-perfect accuracy on FACT questions but struggle substantially on MIND questions.
  • GPT-3-DAVINCI performs better on MIND-1st than MIND-2nd questions for 2 < k < 16, with the gap persisting somewhat at k=24.
  • GPT-3-DAVINCI’s tendency to select the most recently mentioned location helps explain concave MIND-TB and convex MIND-FB accuracy patterns.
  • The probing method selects the answer with the highest conditional likelihood given the context, question, and k training examples.

D What About ChatGPT or GPT-4? Effect of Instruction-tuning & RLFH

The paper evaluates newer instruction-tuned and RLHF OpenAI models on SOCIALIQA and TOMI using zero-shot prompting. These models improve social-question performance, but mental-state reasoning remains weak, so neural Theory of Mind is not established.

  • D What About ChatGPT or GPT-4? Effect of Instruction-tuning & RLFH: The study extends its analysis from language-modeling-only systems to instruction-finetuned and RLHF models, including ChatGPT and GPT-4.
  • D What About ChatGPT or GPT-4? Effect of Instruction-tuning & RLFH: Newer OpenAI models are evaluated zero-shot on randomly selected 400-example subsets of SOCIALIQA and TOMI.
  • D.1 SOCIALIQA: Social Intelligence and Social Commonsense: 60%, 67%, and 79% are reported for RLHF models under multiple-choice SOCIALIQA probing, exceeding the 33% random-chance level.
  • D.1 SOCIALIQA: Social Intelligence and Social Commonsense: All models perform worse on non-main-character SOCIALIQA questions, with GPT-4 showing a 7% accuracy decrease.
  • D.3 Discussion: Interpretation is limited by opaque training data and possible benchmark contamination, including SOCIALIQA data reportedly present in GPT-4’s training data.
  • D.2 TOMI: Reasoning about Mental States and Realities: On TOMI, GPT-3.5-IFT and GPT-3.5-RLHF improve over essentially random GPT-3-DAVINCI performance by 17% and 19%, respectively.
  • D.2 TOMI: Reasoning about Mental States and Realities: GPT-3.5-Turbo reaches 60% on TOMI MIND questions, slightly exceeding GPT-4’s 59%, while improvements occur mainly on factual questions.
  • D.3 Discussion: The authors conclude that newer models have not clearly achieved neural Theory of Mind, with mental-state performance only 10% above random chance.
Loading 2210.13312v2…