PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Theory-of-Mind Performance in Large Language Models via Prompting

Shima Rahimi Moghaddam, Christopher J. Honey

arXiv:2304.11490v3cs.AIcs.CL

TL;DR

The paper asks whether LLMs can reason about agents’ mental states and whether prompting improves their ToM performance. It evaluates four GPT models with ToM scenarios and in-context prompts, finding that prompted RLHF-trained models exceed 80% accuracy, with GPT-4 reaching 100%.

  • Problem

    The paper examines whether LLMs can reliably reason about agents’ mental states, a capacity described as important for social understanding.

    Source: Introduction · p. 1

  • Method

    The study evaluates four GPT models on ToM comprehension scenarios and tests whether in-context prompting improves their performance.

    Source: Models · p. 3, Experimental Design · p. 3, General Discussion · p. 11

  • Results

    With appropriate prompting, all RLHF-trained models exceeded 80% ToM accuracy, and GPT-4 reached 100%, compared with near 80% for GPT-4 in zero-shot settings.

    Source: Conclusion · p. 13

  • Takeaways & Limitations

    The results indicate that LLMs may possess ToM reasoning capacity without expressing it unless given appropriate context or prompting.

    Source: General Discussion · p. 11

    The main analyses used only two chain-of-thought examples and 16 ToM questions, mostly probing agents’ beliefs.

    Source: General Discussion · p. 12

Abstract

from arXiv · show

Large language models (LLMs) excel in many tasks in 2023, but they still face challenges in complex reasoning. Theory-of-mind (ToM) tasks, which require understanding agents' beliefs, goals, and mental states, are essential for common-sense reasoning involving humans, making it crucial to enhance LLM performance in this area. This study measures the ToM performance of GPT-4 and three GPT-3.5 variants (Davinci-2, Davinci-3, GPT-3.5-Turbo), and investigates the effectiveness of in-context learning in improving their ToM comprehension. We evaluated prompts featuring two-shot chain of thought reasoning and step-by-step thinking instructions. We found that LLMs trained with Reinforcement Learning from Human Feedback (RLHF) (all models excluding Davinci-2) improved their ToM accuracy via in-context learning. GPT-4 performed best in zero-shot settings, reaching nearly 80% ToM accuracy, but still fell short of the 87% human accuracy on the test set. However, when supplied with prompts for in-context learning, all RLHF-trained LLMs exceeded 80% ToM accuracy, with GPT-4 reaching 100%. These results demonstrate that appropriate prompting enhances LLM ToM reasoning, and they underscore the context-dependent nature of LLM cognitive capacities.

Introduction

The introduction frames theory-of-mind (ToM) as reasoning about agents’ mental states and asks whether suitable prompting can improve LLM comprehension. It then outlines the models, scenarios, scoring procedure, and prompt conditions used to test that question.

Introduction

  • Prior ToM evaluations often used single-word or multiple-choice outputs and zero-shot prompts or examples without step-by-step reasoning, motivating tests with suitable prompts.The paper asks whether newer LLMs might perform better when prompt context changes.Source: Introduction · p. 2

Models

  • The study tests GPT-4, Davinci-2, Davinci-3, and GPT-3.5-Turbo, which differ in their training methods.Davinci-3 and GPT-3.5-Turbo were trained with RLHF; the passage describes GPT-4's training details as limited.Source: Models · p. 3

Experimental Design

  • The design compares 16 ToM scenarios, which describe people’s mental states, with 16 Photo control scenarios, which describe scenes without agents.The scenarios were adapted from stimulus sets used in human fMRI research and matched in general difficulty.Source: Experimental Design · p. 3

Dependent Variable

  • Accuracy is the proportion of correct responses across 320 questions, based on 20 runs per model and scenario; responses were manually labeled.Labels allowed correct reasoning among multiple reasonable possibilities but counted incorrect conclusions and inconclusive responses as incorrect.Source: Dependent Variable · p. 4

Experimental Conditions

  • Four prompt conditions compare zero-shot, zero-shot with step-by-step thinking, two-shot chain-of-thought, and two-shot chain-of-thought with step-by-step thinking.The figure’s example includes all text after “Prompt:”, including the final line beginning “A:”.Source: Experimental Conditions · p. 5, Figure 1 · p. 6

Results

Zero-shot ToM accuracy did not consistently improve across newer models, but prompting raised performance for RLHF-trained models. With appropriate prompts, all RLHF-trained models exceeded 80% accuracy, and GPT-4 reached 100%.

Zero-shot Performance

  • Zero-shot Photo accuracy increased with model recency, but ToM accuracy fell from Davinci-2 through GPT-3.5-Turbo; GPT-4 still led both tasks.GPT-4's ToM accuracy exceeded Davinci-2's by 0.11 (p < 0.001).Source: Zero-shot Performance · p. 7, Figure 2 · p. 7

Performance when Supported by Prompting

Conclusion

Interim Discussion

  • On scenarios missed at zero-shot, Davinci-3 and GPT-4 improved with ToM, non-ToM inferential, and Photo CoT examples, suggesting gains were not simple mimicry.The examples used distinct reasoning patterns, yet each class boosted performance on the selected scenarios.Source: Interim Discussion · pp. 9–10, Figure 4 · p. 10

General Discussion

  • Davinci-2 was the only tested GPT-3.5 variant not fine-tuned with RLHF and the only model not boosted by prompting; the authors suggest RLHF may explain the difference.The models were similar in size, at approximately 175 billion parameters.Source: General Discussion · p. 11
  • Preliminary tests also found prompting gains on non-ToM inferential scenarios, leaving open whether the gains are specific to ToM or reflect broader inferential reasoning.The authors identify this generality question as an avenue for further testing.Source: General Discussion · p. 12
  • The main CoT analysis used only two examples and 16 ToM questions, mostly about beliefs; responses were manually scored and failure modes were not quantitatively analyzed.The authors call for more varied ToM tasks and further study of model-specific failure modes.Source: General Discussion · pp. 12–13
  • The authors caution that these results do not settle whether LLMs possess a mental faculty for representing mental states; performance varies with prompt context.They argue against reducing LLM ToM to a simple yes-or-no judgment.Source: General Discussion · pp. 11, 13

16 Photo scenarios

The 16 Photo scenarios use records and depictions to test whether answers reflect what was true when an image or record was made, rather than what changed later.

16 Photo scenarios

  • Several scenarios ask whether an earlier image or map reflects a later change, such as a car repaint, a moved door, a fallen apple, or rising water.The scenarios distinguish the depicted state from subsequent events.Source: §16 · pp. 17–18
  • The set also asks whether records show past conditions despite subsequent changes to a house, a blouse, or an island.Examples include renovations, a blouse changing color, and an island altered by an eruption.Source: §16 · pp. 17–18
  • Other questions contrast historical depictions with later changes to a landscape or landmark, including a tree replaced by a fountain, petroglyphs that crumbled, and land flooded by a dam.They ask what the earlier depiction shows, not what is visible now.Source: §16 · pp. 17–18

Supplement B

The 16 ToM scenarios ask what people believe, expect, or know when they have incomplete information about events or changes around them.

Supplement B

  • The set also tests expectations based on mistaken perceptions, incomplete news, or routines, such as colorblindness, an unread update, or a usual drink order.Other examples involve borrowed shoes, a car moved overnight, and a game’s final score.Source: Supplement B · p. 19
  • Other questions ask what one person believes about another person’s state, including illness, sleep, or a presumed location.The scenarios distinguish what an observer knows from what actually happened.Source: Supplement B · pp. 19–20
  • Several scenarios test whether an agent updates a belief after missing an event, such as a changed location, a moved object, or a changed situation.Examples include Jenny’s relocated chocolate, John’s keys, and campers returning to a cooler.Source: Supplement B · p. 20
  • Some scenarios ask whether an agent knows about an unseen change or mistaken appearance, including an altered dish, melted ice cream, or a fish that is actually a tire.A further example asks whether Laura assumes her horse’s mane was braided while she was away.Source: Supplement B · p. 20

Supplement C

Human participants performed similarly on the Photo and ToM scenarios, supporting Photo as a matched control while showing that the questions were not at ceiling for humans.

ToM accuracy of human participants

  • Photo accuracy was 86% ±4% and ToM accuracy was 87% ±4% among human participants.The authors conclude that the two scenario types were comparable in difficulty for participants.Source: ToM accuracy of human participants · p. 21
  • The authors treat Photo scenarios as a useful ToM control because human performance was matched, while describing both scenario types as relatively easy but below ceiling.They note modest incentives for accuracy among online participants.Source: ToM accuracy of human participants · p. 21
  • The authors caution that differences in testing conditions rule out a direct quantitative comparison between human and LLM performance.Human participants nevertheless made errors on the questions more than 10% of the time.Source: ToM accuracy of human participants · p. 21

Supplement D

Supplement D describes examples used to test prompting approaches and examines selected zero-shot errors in GPT-4 and Davinci-3. Its supplied figure description identifies the accuracy comparisons but does not provide their outcomes.

Effects of Various In-context Examples on ToM Performance of LLMs

GPT-4

  • The GPT-4 analysis assesses four scenarios it answered incorrectly in zero-shot testing.The examples involve an inference about illness, another person’s belief, a tulip change, and a drink order.Source: GPT-4 · p. 23

Davinci-3

  • The Davinci-3 analysis assesses six scenarios it answered incorrectly in zero-shot testing.Examples include a moved car, a colorblind agent’s belief, and a person’s mistaken expectation about an object’s location.Source: Davinci-3 · pp. 23–24
  • Figure S.1 reports per-scenario accuracy for GPT-4 and Davinci-3 across four prompting conditions, based on 20 repetitions of a single prompt.The mean accuracy for the Davinci-3 analysis is shown in Figure 4, but its values are not provided here.Source: Davinci-3 · p. 24, Figure S.1 · p. 25

Supplement E

The supplement tests whether prompting effects extend to inferential scenarios that do not involve reasoning about agents’ beliefs. Prompting boosted accuracy for Davinci-3 and GPT-3.5-Turbo, while Davinci-2 did not improve and GPT-4 was already at ceiling.

Generalizability of ToM Results to Inferential Reasoning in Non-ToM Scenarios

  • The new scenarios required inferential reasoning but were not false-belief scenarios, so the ToM-example reasoning steps could not be directly transferred.The evaluation compared zero-shot performance with the same Two-shot chain-of-thought ToM examples plus step-by-step thinking prompt.Source: Generalizability of ToM Results to… · pp. 26–27
  • Davinci-3 and GPT-3.5-Turbo improved significantly with prompting, Davinci-2 did not, and GPT-4 was at ceiling in both settings.Zero-shot accuracy declined from Davinci-2 to Davinci-3 to GPT-3.5-Turbo, while GPT-4 reached ceiling performance.Source: Generalizability of ToM Results to… · p. 27, Figure S.2 · p. 27
  • The authors interpret the gains as evidence that prompting effects may extend beyond ToM, since the questions required no agent-mind reasoning and used different reasoning steps.They therefore suggest possible generalization to a broader class of inferential reasoning tasks.Source: Generalizability of ToM Results to… · p. 27
  • GPT-3.5-Turbo used its default temperature, likely 0.7, while the other three models used 0.4 for this analysis.The API did not yet allow control of GPT-3.5-Turbo’s inference-time temperature when these data were collected.Source: Generalizability of ToM Results to… · p. 27

Related papers

Loading 2304.11490v3…