PaperlayerView exact revision on arXiv

Source-linked AI summary

SocialArena: Evaluating Social Intelligence in Large Language Models

Zongqi Wang, Rui Wang, Yuchuan Wu, Hua Yang, Bingxue Qiu, Tianle Gu, Bing Zhao, Yujiu Yang, Jieping Ye, Yongbin Li

arXiv:2610.39247v1cs.AI

TL;DR

Existing evaluations of LLM social intelligence are fragmented across isolated capabilities, despite social intelligence being relevant to applications such as counseling and education. SocialArena defines and evaluates social intelligence through a three-capability framework, broad scenario coverage, and human arena evaluation; its leaderboard shows that models strong on other benchmarks can lag on SocialArena.

  • Problem

    Evaluations of LLM social intelligence typically focus on only one or a few isolated capability dimensions, despite its relevance to applications such as psychological counseling and education.

    Source: §1 · p. 1

  • Method

    SocialArena evaluates role-playing, mental state reasoning, and behavioral reasoning through human arena comparisons of model responses across a broad range of synthesized social scenarios.

    Source: §1 · p. 2, §2.1 · p. 3, §2.4 · p. 5

  • Results

    Models with performance comparable on STEM, coding, and agentic benchmarks can lag substantially on SocialArena, whose top three leaderboard positions are held by closed-source models.

    Source: §3.2 · p. 6

  • Takeaways & Limitations

    SocialArena ratings provide a basis for assessing social intelligence separately from performance on problem-solving benchmarks.

    Source: §3.2 · p. 6

    The benchmark uses single-turn evaluation because preliminary small-scale annotations found multi-turn settings more costly, while single-turn scenarios sufficiently address most evaluation needs.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs). Although LLMs are increasingly deployed in socially interactive applications such as mental health counseling, companionship chatbots, and educational tutoring, existing evaluations remain fragmented focusing on isolated capabilities or single scenarios and evaluation reliability is largely unverified. This work makes three main contributions to address these limitations. (1) We first propose a unified three-tier framework that defines social intelligence along three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning. To operationalize this definition, we design a role-playing dialogue paradigm with explicit character thinking for evaluation, enabling simultaneous assessment of these capabilities in a manner that mirrors realistic human interactions. (2) To ensure diversity across social scenarios and thereby prevent evaluation conclusions from being confined to specific scenarios, we introduce the U-TRIMF framework, which guides us in synthesizing social contexts with broad coverage. (3) To obtain reliable evaluation, we invest $132.4K to recruit and train human annotators and adopt a human-annotated arena-style protocol, where annotators compare model responses and provide preference labels along with natural-language critiques. We hope that SocialArena can provide valuable insights and guidance for future LLM improvements.

1 Introduction

SocialArena addresses fragmented evaluations of LLM social intelligence with a framework spanning role-playing, mental-state reasoning, and behavioral reasoning, plus broad scenario coverage and human-centered evaluation. Its experiments find weaker behavioral reasoning than social understanding, distinct capability strengths, and rankings that vary across social contexts.

1 Introduction

  • Existing evaluations are fragmented across isolated capabilities and often single scenarios, limiting the generalizability of their conclusions.The paper notes that different social scenarios require different behaviors.Source: §1 · pp. 1–2
  • SocialArena defines social intelligence through role-playing, mental-state reasoning, and contextually appropriate behavioral reasoning.Role-playing maintains character consistency; mental-state reasoning infers one’s own and others’ mental states; behavioral reasoning selects and executes appropriate actions.Source: §1 · p. 2
  • U-TRIMF synthesizes social contexts by identifying six fundamental dimensions and enumerating subcategories to cover a broad scenario space.Its top-down design guides the construction of social scenarios.Source: §1 · p. 2

Q3: How do we ensure reliable evaluation?

  • SocialArena uses trained human annotators, arena-style response comparisons, and natural-language critiques to produce scalable rankings and diagnostically meaningful assessments.The paper contrasts this protocol with static multiple-choice QA and LLM-as-a-Judge evaluation, citing concerns about situated interaction and alignment with human judgments.Source: Q3: How do we ensure reliable… · p. 3

Q4: How do current mainstream LLMs perform on social intelligence?

  • Behavioral reasoning is consistently weaker than social understanding, models have distinct capability strengths, and rankings vary across social contexts.The authors suggest this complexity points to future directions for LLM social intelligence.Source: Q4: How do current mainstream LLMs… · p. 3

2 SocialArena

SocialArena combines broad social-scenario coverage with role-playing dialogue and explicit character thinking to assess three social-intelligence capabilities. Its arena-style evaluation pairs model responses for human preference judgments and Elo ranking.

2.1 Benchmark Overview

  • SocialArena evaluates role-playing, mental-state reasoning, and behavioral reasoning using 5,702 social scenarios and 17,676 human votes.Anonymous models are paired in the same social context, and human votes contribute to an Elo-based leaderboard.Source: §2.1 · p. 3

2.2 Social Scenarios Curation Pipeline

  • U-TRIMF constructs base scenarios by sampling across six dimensions, including user presence, characters, and events, to structure coverage of the social space.The pipeline then adds ten randomly sampled corpus keywords as soft guidance and filters scenarios using automated rules followed by manual review.Source: §2.2 · p. 4, Table 2 · p. 4

2.3 Model Output

  • Models produce a role-playing dialogue with explicit character thinking followed by the character’s response and actions, enabling assessment of all three capabilities.A pilot found free-form thinking difficult to annotate because inferences and decision paths were disorganized.Source: §2.3 · p. 4
  • Mental-state inferences are organized by seven ToM categories, while behavioral reasoning is judged from final behavior regardless of the reasoning steps used.The ToM categories are emotions, desires, intentions, percepts, knowledge, beliefs, and non-literal communication; annotators need not assess every category in each scenario.Source: §2.3 · pp. 4–5

2.4 Evaluation Protocol

  • Figure 2 presents the Elo-based leaderboard for 15 frontier LLMs.The leaderboard uses Elo scores derived from pairwise outcomes.Source: Figure 2 · p. 5, §2.4 · p. 6
  • The evaluation pairs anonymized model responses for human preference labels and optional natural-language critiques, with Elo ratings derived from pairwise outcomes.At least three annotators independently assess each sample; majority voting determines preferences, and unanimous disagreement cases are discarded.Source: §2.4 · pp. 5–6

3 Experiments

SocialArena ranks closed-source models highest overall, but results also reveal behavioral-reasoning weaknesses, uneven performance across dimensions, and scenario-specific rankings. The experiments use human-annotated pairwise battles and examine critique patterns and ranking correlations.

3.1 Evaluation Setup

  • The evaluation pairs every pair of 15 models in 50 to 80 battles, with at least three annotators judging each battle and majority voting determining the verdict.Models are subsequently ranked using Elo.Source: §3.1 · p. 6

3.2 Leaderboard

  • The top three leaderboard positions belong to closed-source models with comparable scores, while open-source models rank in the middle and smaller models at the bottom.Some models comparable on STEM, coding, and agentic benchmarks lag substantially on SocialArena.Source: §3.2 · p. 6

3.3 A Deeper Understanding of LLM Performance in Social Intelligence

  • Positive-critique proportions fluctuate across dimensions within a model and vary substantially between models on the same dimension, making small predefined dimension sets inadequate for evaluation.The authors present free-form natural-language human critiques as a practical alternative.Source: Figure 4 · p. 7, §3.3 · p. 8
  • Behavior reasoning accounts for six of the ten most frequent critique clusters, and negative critiques dominate nearly all high-frequency dimensions, especially long-horizon goals and conflict de-escalation.The authors report that critiques emphasize relational outcomes and that role-playing has a more balanced positive/negative distribution than ToM and behavior reasoning.Source: §3.3 · p. 7

3.4 A Closer Look at Scenario-level Performance

  • Several scenarios show high within-scenario rank consistency but low cross-scenario correlation, confirming that rankings are scenario-specific.The authors identify conflict, friendship, strangers, family, superior-subordinate, roleplay/TRPG, and romantic scenarios in this pattern.Source: Figure 5 · p. 8, §3.4 · p. 8
  • Workplace, neighbors, and business rankings correlate closely with conflict rankings, which the authors attribute to overlap with conflict-tagged interactions.Romantic, roleplay/TRPG, and friendship scenarios with low conflict frequency show weaker correlation with conflict rankings.Source: §3.4 · p. 8

4 Evaluation on LLM-as-a-Judge

This section tests whether LLM judges can reliably evaluate social-intelligence responses against human judgments. The reported results show that they fall substantially short of human performance on both evaluation metrics.

4 Evaluation on LLM-as-a-Judge

  • SocialArena-RewardBench uses a randomly sampled subset of SocialArena instances to assess alignment between LLM judges and human expert annotations.The subset is annotated and cross-validated by three human experts, yielding 152 samples.Source: §4 · pp. 8–9
  • The benchmark assesses agreement with expert annotations using preference-label accuracy and critique consistency.Critique consistency is based on F1 similarity meeting or exceeding 0.5.Source: §4 · p. 9
  • LLM judges fall substantially short of human performance on both label accuracy and critique consistency.The comparison uses a human performance baseline established from independent judgments by randomly selected annotators.Source: §4 · p. 9, Table 3 · p. 9

5 Related Works

Prior work evaluates social intelligence in LLMs through distinct capabilities and scenarios, while the paper reviews social intelligence as a broad set of social understanding and response abilities.

5 Related Works

  • Social intelligence includes understanding mental states, empathizing, responding to social contexts, handling cultural differences and conflicts, and persuading through others’ beliefs and knowledge states.The review also encompasses emotions, motivations, and values as mental states.Source: §5 · p. 9
  • Prior evaluations are fragmented across character consistency, Theory of Mind reasoning, and contextually appropriate responses in real or virtual scenarios.These capabilities are typically tied to their specific evaluation scenarios.Source: §5 · p. 9
  • Existing benchmarks often isolate one or a few capability dimensions, including surface-level character consistency, emotion or intention inference, and scenario-specific social responses.Examples include counseling, persuasion and negotiation, and virtual narrative progression.Source: §5 · p. 9

6 Conclusion

SocialArena evaluates LLM social intelligence through a facts-to-mental-states-to-behavior process. It combines broad scenario coverage with human-based arena evaluation to produce reliable results.

6 Conclusion

  • SocialArena structures evaluation as a facts → mental states → behavior reasoning process to assess social intelligence comprehensively.The benchmark is presented as large-scale and systematic.Source: §6 · pp. 9–10
  • U-TRIMF guides SocialArena’s synthesis of diverse social scenarios with varied interaction dynamics.Source: §6 · p. 10
  • A human-based arena-style protocol is used to yield high-quality and reliable evaluation results.Source: §6 · p. 10

Limitations

The authors identify annotator-background bias and single-turn evaluation as limitations, while proposing a public platform for broader participation and ongoing data collection. The platform also addresses unsafe-content risks and applies quality controls to collected scenarios and votes.

Limitations

  • Preference judgments may reflect annotator backgrounds and communication styles; current annotators are native English and Chinese speakers, and evaluation is single-turn.The authors cite higher annotation costs for multi-turn settings and say single-turn scenarios address most evaluation needs.Source: Limitations · p. 10

A.1 Impact Statement

  • The authors say SocialArena may support reward-modeling and RLHF research in unverifiable domains and provide data for socially intelligent character models.Source: §A.1 · p. 15
  • SocialArena scenarios may contain unhealthy or unsafe material, so the platform filters such content from displayed scenarios and leaderboard updates.Source: §A.1 · p. 15

B SocialArena-Platform

  • SocialArena-Platform is proposed as a public venue for broader participation and continuous collection of fresh social scenarios and preference judgments.Its design lets users post real-life dilemmas, receive model and community responses, and use community votes and comments as evaluation signals; filters remove non-social or low-quality prompts and suspicious or duplicate votes.Source: B SocialArena-Platform · p. 15

C Details of Benchmark Construction

The benchmark construction uses U-TRIMF to organize social scenarios and combines keyword-guided synthesis with automated filtering and human review. Its design seeks broad coverage while retaining scenarios with plausible, meaningful social interactions.

C.1 U-TRIMF Framework

  • U-TRIMF spans user presence, participant topology and relationships, plus event interests, motivations, and formality; benchmark scenarios exclude solitary individuals.User presence distinguishes real-world from user-centered scenarios, while participant relationships range from intimate to transactional or asocial.Source: §C.1 · pp. 16–17

C.2 Explanation of Keyword-Guided Scenario Synthesis

  • Ten thematically coherent keywords guide each scenario-generation call after U-TRIMF constraints are applied, countering repetitive outputs from direct category prompting.The keywords are drawn together from a single book chapter.Source: §C.2 · p. 17

C.3 Explanation of Quality Filtering

  • Generated scenarios pass through automated rule-based rejection followed by holistic human review for suitability.The rule-based filtering stage is implemented via Qwen3-235B-A22B-Instruct-2507.Source: §C.3 · p. 17

C.3.1 Rule-Based Filters

  • Rules reject trivial or extreme conflicts and require balanced, emotionally weighty, urgent stakes without a free exit.The rules also exclude serious harms and require a single coherent event; their emotional-stakes requirements apply to real scenarios, with virtual or fictional scenarios exempt.Source: §C.3.1 · pp. 18–19

C.3.2 Human Review

  • Human reviewers assess plausibility, conflict balance, role-play affordance, and subtle rule violations; a scenario is retained only when all four dimensions are satisfactory.Disagreements are discussed, and borderline cases are discarded.Source: §C.3.2 · p. 19

D Details of Evaluation Protocol

The evaluation protocol prepares annotators with guidelines and worked examples, monitors annotation quality, and uses a seven-category mental-state taxonomy for Theory of Mind assessment.

D.1 Annotation Guidelines

  • Annotators receive protocol guidelines and worked examples with full scenarios, paired responses, and expert reference annotations before live annotation.The materials are intended to calibrate understanding of each dimension and the ABTG label scheme.Source: §D.1 · p. 19

D.2 Annotator Recruitment and Quality Control

  • Third-party vendors recruited roughly equal proportions of native English and Chinese speakers, while batch quality review removes annotations from annotators below a 75% pass rate.Those annotators receive targeted feedback and further training for subsequent batches.Source: §D.2 · pp. 19–20

D.3 Details of Theory of Mind (ToM)

  • The paper adopts Beaudoin et al.’s taxonomy to operationalize Theory of Mind across seven mental-state categories.The categories are summarized in Table 6.Source: §D.3 · p. 20

E.1 Full Results of ’A Deeper Understanding of LLM Performance in Social Intelligence’

This section turns human critiques into checklist-level judgments and clusters those criteria into a compact evaluation taxonomy. It reports 51,203 dimension-level records and 100 representative evaluation dimensions.

E.1 Full Results of ’A Deeper Understanding of LLM Performance in Social Intelligence’

  • Table 6 summarizes the mental-state categories used in the paper’s Theory of Mind evaluation.Source: Table 6 · p. 21
  • The extraction process yields 51,203 dimension-level records by scoring each checklist item as positive or not based on the human critique.An LLM extracts fine-grained checklist items from critiques and records both candidate models’ performance on each item.Source: §E.1 · pp. 20–21
  • Checklist items are grouped into role-playing, internal state understanding, and decision-making categories, then clustered into 100 representative evaluation dimensions.A language model generates concise labels for initial clusters before a second clustering stage consolidates the taxonomy.Source: §E.1 · p. 21

E.2 Inter-Annotator Agreement

The agreement analysis compares three annotators’ labels with a uniform-random baseline. Observed full agreement substantially exceeds that baseline, while full disagreement is less frequent.

E.2 Inter-Annotator Agreement

  • Each sample receives independent labels from three annotators across the four-class space {A, B, T, G}.Source: §E.2 · p. 21
  • The analysis distinguishes full agreement, majority agreement, and full disagreement, comparing their observed frequencies with uniform random guessing.Source: §E.2 · pp. 21–22
  • 39.18% full agreement is more than 6× the 6.25% random baseline, while 8.29% full disagreement is more than 4× below the 37.50% chance rate.The authors interpret these patterns as indicating substantially greater-than-chance convergence on shared preferences.Source: Table 7 · p. 22, §E.2 · p. 22

E.3 Analysis of Typical Failure Modes in Arena-Style Benchmarks

This section describes SocialArena’s structured role-play prompts, response format, and evaluation criteria, and analyzes measured length and position effects alongside its design’s mitigation of other common biases.

E.3 Analysis of Typical Failure Modes in Arena-Style Benchmarks

  • Short role-playing outputs, detailed criteria, and required justifications mitigate stylistic shortcuts; format, tone, and confidence biases are inapplicable by design.The authors conclude that typical biases are absent or controlled in this setting.Source: §E.3 · p. 22
  • 51.05% of winners were longer than losers (mean length difference 14.0 tokens; median 3.0 tokens), while votes favored option B over A (43.63% vs. 39.23%).The authors describe the length results as close to chance and the position effect as mild.Source: §E.3 · p. 22

Prompt: Real Social Contexts Generation

2.1 Plot Overview.

  • The plot overview should derive its complexity from authentic sources rather than artificially contrived setups.Source: §2.1 · p. 23

2.2 Current Scene.

  • The current-scene prompt asks for the characters’ relationship, relevant preceding events, the present discussion, and the source of its complexity, in concise, restrained prose.The opening dialogue should sound natural, fit the scene, and leave room for the model to improvise.Source: §2.2 · p. 23

2.4 Character Information.

  • The character-information prompt asks for dialogue participants’ names in English.Source: §2.4 · p. 24

3. Output Format.

  • The output-format prompt requires a quality check against scenario constraints, including avoiding extreme or contrived plots and ensuring conflicts have comparable weight.If a check finds an issue, the prompt directs the generator to re-output and check again.Source: §3 · p. 24

3. Output Format

  • The response format asks characters to analyze 2–4 relevant mental-state dimensions, describe a behavior, optionally validate it, and iterate if the behavior seems suboptimal.Behavior reasoning may explicitly connect the response to the preceding mental-state analysis.Source: §3 · p. 27
  • The instructions require adaptive dimension selection, first-person character perspective, structured output, and reasoning directed toward the final behavioral decision.Characters should reflect their own cognitive limitations and personality traits.Source: §3 · p. 27
  • The listed criteria include perceiving others’ emotions, intent, needs, and implicit signals, and responding with empathy, nuance, and relationship awareness.They also cover understanding others’ knowledge, beliefs, and non-literal communication.Source: §3 · p. 31
  • Behavioral criteria span constructive conflict handling, balancing interests and constraints, maintaining boundaries, and advancing dialogue toward practical outcomes.They also include de-escalation, collaboration, and converting conflicts into constructive action.Source: §3 · pp. 31–32

Related papers

Loading 2610.39247v1…