PaperlayerView exact revision on arXiv

Source-linked AI summary

Infusing Theory of Mind into Socially Intelligent LLM Agents

EunJeong Hwang, Yuwei Yin, Giuseppe Carenini, Peter West, Vered Shwartz

arXiv:2509.22887v2cs.CL

TL;DR

LLM social agents typically do not explicitly model others’ mental states, and prior ToM evaluations have often focused on QA rather than social interactions. TOMA uses dialogue lookahead to select useful mental-state and utterance pairs for training. On Sotopia, it improves over base-model variants and shows more strategic behavior, long-horizon adaptation, and better partner relationships.

  • Problem

    LLM ToM research has typically evaluated ability through QA, while interactive social agents have largely overlooked explicit mental-state modeling despite the importance of understanding conversation partners.

    Source: §1 · p. 1

  • Method

    TOMA generates mental-state hypotheses and associated utterances, uses dialogue simulation to select pairs with high goal achievement, and fine-tunes the model on those pairs.

    Source: §1 · p. 1, §2.2 · p. 3, §2.3 · p. 3

  • Results

    TOMA improves Sotopia scores by up to 18.9% for Qwen2.5-3B and 6.9% for Qwen2.5-7B versus the best base-model variant, while showing strategic, goal-oriented behavior, long-horizon adaptation, and better partner relationships.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings support explicit modeling of social reasoning and internal agent mechanisms as a step toward socially intelligent LLM agents.

    Source: §7 · p. 9

    LLM-based judges may still be limited in capturing subtle aspects of multi-turn social interactions.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Theory of Mind (ToM)-an understanding of the mental states of others-is a key aspect of human social intelligence, yet, chatbots and LLM-based social agents do not typically integrate it. In this work, we demonstrate that LLMs that explicitly use ToM get better at dialogue, achieving goals more effectively. After showing that simply prompting models to generate mental states between dialogue turns already provides significant benefit, we further introduce ToMAgent (ToMA), a ToM-focused dialogue agent. ToMA is trained by pairing ToM with dialogue lookahead to produce mental states that are maximally useful for achieving dialogue goals. Experiments on the Sotopia interactive social evaluation benchmark demonstrate the effectiveness of our method over a range of baselines. Comprehensive analysis shows that ToMA exhibits more strategic, goal-oriented reasoning behaviors, which enable long-horizon adaptation, while maintaining better relationships with their partners. Our results suggest a step forward in integrating ToM for building socially intelligent LLM agents.

1 Introduction

The paper asks whether explicit Theory of Mind (ToM) can improve LLM social reasoning, where prior work often evaluates ToM in QA rather than interactive social situations. It introduces ToMAgent, which selects mental-state and utterance pairs using dialogue simulations, and reports stronger Sotopia performance and more strategic, goal-oriented behavior.

1 Introduction

  • Prior ToM work for LLMs typically evaluates ability through QA, while interactive social research has largely overlooked explicit mental-state modeling.This gap matters because social interaction success depends on understanding conversation partners as well as expressing one’s own intentions and beliefs.Source: §1 · p. 1
  • ToMAgent combines partner mental-state hypotheses with candidate utterances, simulates dialogue outcomes, and fine-tunes on the most successful conversations.The model generates partner mental states and strategic utterances likely to support goal achievement.Source: §1 · pp. 1–2
  • Simply prompting LLMs to generate mental states between dialogue turns can already contribute significantly to goal achievement.The paper evaluates TOMA in Sotopia scenarios spanning collaboration, negotiation, persuasion, and competition.Source: §1 · pp. 1–2
  • TOMA improves Sotopia scores by up to 18.9% for Qwen2.5-3B and 6.9% for Qwen2.5-7B versus the best base-model variant.The analysis also reports more strategic, goal-oriented, long-horizon behavior and better partner relationships.Source: §1 · p. 2

2 Methodology

TOMA trains an LLM to generate mental states and corresponding utterances that support social goals. It samples dialogue contexts, evaluates candidate pairs through short-horizon simulation, and fine-tunes on selected pairs.

2 Methodology

  • TOMA is a look-ahead training framework for achieving agents’ social goals through dialogue, such as finding a mutually agreeable solution to a conflict.The method combines context sampling, candidate generation, simulated outcome scoring, and fine-tuning.Source: §2 · p. 2

2.1 Sampling Conversations to Seed Scenarios and Agents

  • Training contexts come from sampled Sotopia-Pi episodes and include a scenario, agents’ goals, and a partial dialogue history.The histories are truncated to at most four turns so goals have not yet been achieved.Source: §2.1 · pp. 2–3

2.2 Generating and Scoring ToM Hypotheses and Utterances

  • For each context, the target model generates K mental-state hypotheses covering at least three ToM dimensions and J utterances per hypothesis, then scores candidate pairs through up to four future dialogue turns.Pairs with an average goal score of at least 9 are retained; if none meet that threshold, the top-scoring pair is kept.Source: §2.2 · p. 3

2.3 Fine-Tuning on ToM States & Utterances

  • Fine-tuning uses selected context–mental-state–utterance pairs to train both mental-state prediction from context and utterance prediction conditioned on context and mental state.The model is trained with token-level cross-entropy on pairs selected for high simulated goal scores.Source: §2.3 · p. 3

3 Experimental Setup

The experiments evaluate dialogue agents on Sotopia-Eval’s all and hard scenario sets using goal, relationship, and knowledge scores. Training compares base and fine-tuned models, including variants trained with mental states, utterances, or both.

3.1 Evaluation

  • Sotopia-Eval includes 450 instances across 90 scenarios in the all set and 70 instances across 14 GPT-4-challenging scenarios in the hard set.Each scenario in the all set is paired with five agent pairs.Source: §3.1 · p. 4
  • An LLM judge scores goal achievement, relationship preservation or improvement, and new knowledge gained for each agent in a conversation.GPT-5-mini is the primary evaluator, and reported scores are single-run results after stable three-run measurements.Source: §3.1 · p. 4
  • The baseline comparison includes vanilla models and Base+MS models prompted to generate mental states before producing utterances.Base+MS measures the quality and utility of mental states generated by the base model.Source: §3.1 · p. 4

3.2 Training

  • Training compares utterance-only fine-tuning, mental-state-only fine-tuning, and TOMA’s joint mental-state and utterance training.The reported tables cover Rel, Know, and Goal on the all and hard splits, including GPT-5-nano performance on the hard split.Source: §3.2 · p. 4, Table 1 · p. 5, Table 2 · p. 5

4 Experiments

Across Sotopia experiments, ToMA outperforms base variants and shows distinct effects across partner sizes, conversation types, and reasoning strategies. The analyses examine performance dimensions, longer interactions, strategy patterns, and generated mental states.

4.1 Does ToM Help with Social Reasoning?

  • Compared with Base+MS, TOMA improves scores by 16.8%, 6.6%, and 23.45% for Qwen2.5-3B, 7B, and Llama-3.1, respectively, averaged across all and hard sets.Across both subsets, TOMA also consistently outperforms model variants on relationship, knowledge, and goal completion.Source: §4.1 · p. 5

4.2 How do Partners Affect Performance?

  • TOMA produces consistently better combined target-and-partner outcomes; 7B target outcomes improve consistently, while 3B goal-achievement winners vary against Base+MS.For 3B targets, the authors suggest that a larger, socially unaware partner may make goal achievement harder.Source: §4.2 · p. 6, Table 4 · p. 6

4.3 How does TOMA Perform Across Different Conversation Types?

  • TOMA outperforms Base in at least 75% of conversations in each of four scenario types, with greater gains in conflicts.The analysis covers five conversations for each of 90 scenarios; types are cooperation, negotiation, persuasion, and conflict.Source: §4.3 · p. 7

4.4 What Strategies Does TOMA Employ?

  • Successful TOMA conversations use compromise, accommodation, and solution offering while retaining comparable rapport-building and cooperative responses; these strategies also support competitive settings.The reported strategic behavior tends to increase with model size.Source: §4.4 · p. 7, Figure 3 · p. 8
  • TOMA failures more often involve active but unsuccessful strategies or neglecting relationship-building, while 7B TOMA has fewer such failures and more failures from inaction.The authors hypothesize that 7B's greater sensitivity to partner emotions reduces ignored preferences and prioritizing-self failures.Source: §4.4 · p. 7
  • TOMA generates more hypotheses about intentions and fewer about emotions than Base+MS, while keeping belief, desire, and knowledge levels similar.The authors connect intention-focused hypotheses with strategic, goal-oriented behavior and emotion-focused hypotheses with rapport building.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
  • TOMA produces 6.3% more first-order beliefs on 3B and 5.0% more on 7B than Base+MS, despite equal prompting proportions.First-order beliefs are the target agent's beliefs about others; the table reports zero- versus first-order reasoning percentages.Source: Table 5 · p. 7, §4.4 · p. 8

5 Further Analysis

Further analyses test whether evaluation trends persist across LLM judges and examine variations in simulation and sampling settings.

Validity of the LLM-as-a-judge evaluation protocol

  • Across four LLM judges, TOMA consistently outperforms baselines, and the reported evaluation trends remain consistent.Figure 5 summarizes relationship, knowledge, and goal scores on the hard split.Source: §5 · p. 8, Figure 5 · p. 9
  • Human validation reports 93.3% average validity for GPT-5-mini judge reasoning and 90.7% inter-annotator agreement.Ratings from different LLM judges are positively correlated across settings.Source: §5 · p. 8

Performance across different K/J numbers and simulation turns

  • More simulation turns generally improve performance, with four turns balancing data-construction efficiency and effectiveness; extra mental-state and utterance samples yield modest gains, while J = 2 is sufficient.The ablations vary K and J while holding the other parameter at 2, and test up to eight dialogue simulation turns.Source: §5 · p. 8, Table 6 · p. 8

6 Related Work

Prior work on LLM Theory of Mind reports mixed findings and often evaluates static question-answering rather than dynamic social interaction. This paper situates its training approach within work on ToM methods and look-ahead simulation.

Theory of Mind in LLMs

  • LLM ToM findings conflict: benchmark performance has been interpreted as evidence of ToM (Kosinski, 2023, 2024; Strachan et al., 2024), while other studies report inconsistent or superficial abilities (Ullman, 2023; Shapira et al., 2024; Nickel et al., 2024).Prior approaches include prompting models to reason about beliefs and mental states (Shinoda et al., 2025), belief tracking (Sclar et al., 2023), and Bayesian Inverse Planning (Ying et al., 2023).Source: §6 · p. 9
  • Many existing evaluations use static, artificial ToM benchmarks where models answer as observers, whereas this work evaluates social conversations between AI agents in Sotopia (Wagner et al., 2025; Xiao et al., 2025; Lupu et al., 2025).Belief-tracking and Bayesian approaches are described as limited in scope and applied to specific setups; exploring multiple ToM hypotheses at inference is computationally expensive.Source: §6 · p. 9

Look-Ahead Simulation in Self-Training Agents

  • Rather than directly optimizing utterances for goal achievement, which could lead to reward hacking, this work trains models to use ToM in dialogue and consider mental-state information when generating utterances.The paper uses simulation to generate training examples, distinguishing its approach from methods that use simulation for outcome alignment through reinforcement learning.Source: §6 · p. 9

7 Conclusion

ToMA combines ToM-driven mental-state and utterance prediction with conversation simulation to select goal-supporting interaction trajectories. On Sotopia, it performs competitively with GPT-5-nano using much smaller models and supports strategic, goal-oriented behavior, long-horizon adaptation, and relationship management.

7 Conclusion

  • ToMA combines ToM-driven mental-state and utterance prediction with conversation simulation to select trajectories that support goal achievement.The conclusion describes this as a training framework for social reasoning.Source: §7 · p. 9
  • On Sotopia, ToMA achieves competitive performance with GPT-5-nano using much smaller models.The conclusion reports effectiveness across a range of baselines.Source: §7 · p. 9
  • ToMA better infers others’ mental states, supporting more strategic, goal-oriented behavior, long-horizon adaptation, and improved relationship management.Source: §7 · p. 9

Limitations

The paper treats useful mental states as goal-supporting interpretations rather than a single ground truth, and notes limits in evaluating them. Its evaluation also relies on LLM judges that may miss subtle multi-turn social interactions.

Limitations

  • The approach does not target a single ground-truth mental state because beliefs, desires, emotions, intentions, and knowledge can admit multiple valid interpretations.Useful states are defined by whether they help the agent reason about its partner and advance social goals.Source: Limitations · p. 10
  • Fine-grained evaluation methods for mental-state representations remain future work.The paper points to its distributional and qualitative analyses as illustrations of how these dimensions are used in LLM reasoning.Source: Limitations · p. 10
  • LLM judges may introduce bias and remain limited in capturing subtle multi-turn social interactions, despite comparisons across judges and validation with human annotations.The authors report consistent trends and strong agreement across evaluators.Source: Limitations · p. 10

Ethics Statement

The ethics statement identifies potential misuse of social capabilities and recommends safeguards for LLM-powered applications. The inference-time analysis reports compact additional tokens but also an extra call and added per-turn latency.

Ethics Statement

  • Social capabilities may be exploited for manipulation or deception, so the authors recommend AI disclosure and avoiding human names or avatars in LLM-powered applications.They identify social media bots, political persuasion, and scams as risk contexts.Source: Ethics Statement · p. 10
  • Human evaluations received institutional ethics approval and informed consent; no personal information was collected, and annotators averaged $10 per hour.The procedures followed applicable institutional and federal guidelines.Source: Ethics Statement · p. 10

A Inference Time Costs

  • ToMA adds an average of 47 conversation tokens and around 4.5 seconds per turn on one NVIDIA L40S GPU, while requiring one additional inference-time call.The reported latency applies to both Qwen-3B and Qwen-7B; latency and compute overhead depend on hardware configuration.Source: A Inference Time Costs · p. 14

B Further Analysis

The supplementary analysis describes the judge comparisons and human validation presented in Tables 8–10. It also specifies model-loading and generation settings, plus LoRA fine-tuning and validation-based checkpoint selection.

B Further Analysis

  • Tables 8–10 cover four LLM judges’ scores on the hard split, pairwise Pearson correlations, and human validation of GPT-5-mini judge reasoning.Source: B Further Analysis · p. 14

C.1 Model Settings

  • Open-weight models use Hugging Face Transformers checkpoints in bfloat16, with a 4096-token context, seed 42, temperature 0.7, and top-p 0.9; proprietary models use APIs at temperature 1.0.The open-weight settings apply to models such as Qwen; proprietary models include GPT-5 and Gemini.Source: §C.1 · p. 14

C.2 Training Details

  • Training uses LoRA fine-tuning, grid search over learning rate and LoRA rank and alpha, and validation-based checkpoint selection with early stopping after three unchanged updates.Validation samples 20 testing instances and generates 10 conversation turns per instance.Source: §C.2 · p. 14

C.3 Experimental Costs

The section reports costs for training-data construction and comprehensive evaluation, along with open-weight model training hardware and duration.

C.3 Experimental Costs

  • Gemini API calls for constructing training data with mental states and utterances cost less than 5 USD.Source: §C.3 · p. 14
  • GPT-5 (gpt-5-mini) evaluation for the comprehensive experiments cost roughly 100 USD.Source: §C.3 · p. 14
  • Open-weight LLM experiments used one NVIDIA L40S GPU per session, with each training session taking about 4 hours using unsloth.Source: §C.3 · p. 14

D.1 How does TOMA Perform Across Different Evaluation Dimensions?

TOMA outperforms Base on more instances, especially for goal and relationship scores, while analyses examine scenario-specific gains, reasoning strategies, and mental-state use.

D.1 How does TOMA Perform Across Different Evaluation Dimensions?

  • More TOMA scores fall above Base on the Goal and Relationship dimensions, with many Goal cases favoring TOMA substantially.The plots compare paired scores, with the neutral line marking draws.Source: §D.1 · p. 14, Figure 6 · p. 16, Figure 7 · p. 16

D.2 How does TOMA Perform Across Different Conversation Types?

D.3 What Strategies Does TOMA Employ?

  • Success and failure explanations are clustered into 25 manually verified canonical labels, then classified per conversation; figures show the top seven labels for Base and TOMA on 7B.Source: §D.3 · p. 22, Figure 11 · p. 23

D.4 How different mental state dimensions contribute to goal achievement

  • Across scenarios, TOMA uses mental-state dimensions consistently and more evenly than Base+MS, with greater emphasis on intention; an example shows motivations guiding compromise proposals.Source: §D.4 · p. 25, Figure 12 · p. 26, Figure 13 · p. 27

E LLM Prompts

  • For utterance training, prompts provide scenario, goal, mental state, and conversation history; mental-state generation uses the same inputs without the mental state.Source: E LLM Prompts · p. 28

F Human Annotation Instructions

Related papers

Loading 2509.22887v2…