PaperlayerView exact revision on arXiv

Source-linked AI summary

SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents

Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, Maarten Sap

arXiv:2310.11667v2cs.AIcs.CLcs.LG

TL;DR

AI systems’ social intelligence remains difficult to assess robustly, especially in interactive, diverse, goal-driven settings. SOTOPIA provides an open-ended environment and holistic evaluation framework for role-play interactions, finding differences among models and between models and humans while exposing challenges on difficult tasks.

  • Problem

    AI systems’ social intelligence has not yet been robustly ascertained, and existing benchmarks often lack interaction or diverse goal-driven behaviors.

  • Method

    SOTOPIA simulates role-play interactions among language-model agents and humans across generated scenarios, characters, relationships, and social goals, evaluated with SOTOPIA-EVAL.

  • Results

    SOTOPIA reveals differences in social intelligence among models and between models and humans, with larger models still falling short of humans on challenging collaboration and competition tasks.

  • Takeaways & Limitations

    SOTOPIA has potential as a platform for assessing and enhancing social intelligence in language agents.

  • Takeaways & Limitations

    The paper focuses on two-agent tasks with locally consistent goals over relatively short timespans, and GPT-4 evaluations may be affected by known evaluator biases.

Abstract

from arXiv · show

Humans are social beings; we pursue social goals in our daily interactions, which is a crucial aspect of social intelligence. Yet, AI systems' abilities in this realm remain elusive. We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We simulate the role-play interaction between LLM-based agents and humans within this task space and evaluate their performance with a holistic evaluation framework called SOTOPIA-Eval. With SOTOPIA, we find significant differences between these models in terms of their social intelligence, and we identify a subset of SOTOPIA scenarios, SOTOPIA-hard, that is generally challenging for all models. We find that on this subset, GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills. These findings demonstrate SOTOPIA's promise as a general platform for research on evaluating and improving social intelligence in artificial agents.

1 INTRODUCTION

SOTOPIA addresses the difficulty of robustly evaluating social intelligence by providing an open-ended, interactive environment for diverse, goal-driven social interactions. Its evaluation framework reveals differences among models and between models and humans, while challenging tasks expose shortcomings in larger language models.

  • Social intelligence involves balancing complex, potentially conflicting goals while understanding others’ intentions, beliefs, norms, and expectations.
  • Existing evaluations are limited because many benchmarks are non-interactive, while interactive evaluations often omit diverse goal-driven behaviors or focus on specific tasks.
  • SOTOPIA combines open-ended interaction, diverse social scenarios, multi-turn communication, physical actions, and multi-dimensional performance evaluation.
  • SOTOPIA simulates role-play episodes by combining 90 social scenarios with 40 characters, relationships, private goals, and agent policies.
  • SOTOPIA-EVAL assesses social interactions across multiple dimensions using human and GPT-4 judges, with GPT-4 serving as a proxy for human judgments on selected criteria.
  • Larger language models generally outperform smaller ones, yet they struggle to collaborate and compete with humans on challenging tasks and may divulge secrets or violate social rules.
  • SOTOPIA is presented as an extensible benchmark and test-bed for assessing disparities in social intelligence among models and between models and humans.

2 SOTOPIA INTERACTION ENVIRONMENT

SOTOPIA constructs a large, realistic, open-ended task space from generated scenarios, characters, relationships, and social goals. Episodes expose agents to partially observable, multi-turn interactions in which they use verbal, non-verbal, and physical actions.

  • The environment is designed to be realistic, support mixed explicit and implicit utilities, and generate open-ended tasks without heavy human intervention.
  • A task combines scenario context, characters, and social goals, while episodes consist of multiple turns between agents role-playing their assigned characters.
  • The paper samples two-agent tasks with locally consistent goals over short timespans, although SOTOPIA could support more than two agents.
  • Agents privately observe their own goals and profiles, while other agents’ goals remain invisible and their profiles are only partially observable.
  • Scenario contexts, characters, and relationships jointly shape the space of possible optimal behaviors, making the task space behaviorally complex.
  • Relationships constrain which scenarios are appropriate and determine how much of each character’s profile agents can observe.
  • Scenarios include shared context and private social goals, with relationship constraints limiting which characters can be sampled together.
  • At each turn, agents can speak, use non-verbal communication, or perform physical actions to pursue their social goals.

3 SOTOPIA-EVAL: HOLISTIC SOCIAL AGENT EVALUATION FRAMEWORK

SOTOPIA-EVAL evaluates social agents holistically rather than measuring goal completion alone. Its seven dimensions cover goal achievement, believable behavior, knowledge acquisition, secrecy, relationships, social rules, and financial or material benefits.

  • The framework scores each episode along seven dimensions inspired by sociology, psychology, and economics.
  • Goal Completion (GOAL) [0–10] measures the extent to which an agent achieved its goals.
  • Believability (BEL) [0–10] measures whether behavior is natural, realistic, and consistent with the agent’s character profile.
  • Knowledge (KNO) [0–10] measures whether agents acquire new and important information during interaction.
  • Secret (SEC) [-10-0] measures whether agents successfully keep secret information or intentions private.
  • Relationship (REL) [-5–5] evaluates whether interaction preserves or improves relationships, social status, or reputation.
  • Social Rules (SOC) [-10–0] covers adherence to social norms and legal rules, while Financial and Material Benefits (FIN) [-5–5] captures short- and long-term economic utility.

4 RESEARCH QUESTIONS AND EXPERIMENTAL SETUP

The experiments use SOTOPIA to simulate diverse, goal-oriented social interactions and compare models, including interactions with humans on a challenging subset. The setup evaluates GPT-3.5, GPT-4, Llama-2-70b-chat, and MPT-30b-chat under fixed model settings.

  • SOTOPIA simulations examine differences among models and between models and humans in goal-oriented social intelligence.
  • The study creates 40 agents, 90 relationships, and 90 scenarios, yielding 450 sampled tasks.Each scenario contributes five character pairs based on its constraints.
  • GPT-4 interacts with humans on SOTOPIA-hard because available resources limit this part of the study.
  • GPT-3.5, GPT-4, Llama-2-70b-chat, and MPT-30b-chat are compared as interacting agents.
  • Model agents use temperature 1 for response diversity, while evaluators use temperature 0 for stable evaluation.
  • The study leaves newer prompting methods, including Chain-of-Thought and ReAct, for future work.

5 CAN GPT-4 EVALUATE SOCIAL INTERACTIONS?

The authors test whether GPT-4 can automate SOTOPIA-EVAL by comparing its ratings with human judgments across interactions and dimensions. GPT-4 generally tracks human scores, especially for model outputs on selected dimensions, but evaluation remains affected by known LLM biases.

  • 5.1 DATA COLLECTION PROCEDURE: 200 episodes are rated by multiple Mechanical Turk annotators and GPT-4 using the same seven-dimension evaluation instructions.Human scores average several annotators’ ratings, while GPT-4 produces an integer score and rationale for each episode, agent, and dimension.
  • 5.2 ANALYZING GPT-4 EVALUATIONS WITH HUMAN EVALUATIONS: More than 74% of GPT-4 scores fall within one standard deviation of the corresponding human scores.
  • 5.2 ANALYZING GPT-4 EVALUATIONS WITH HUMAN EVALUATIONS: GPT-4 is more likely than humans to assign higher rather than lower scores when its judgment disagrees with average human judgment.
  • 5.2 ANALYZING GPT-4 EVALUATIONS WITH HUMAN EVALUATIONS: For model outputs, GPT-4 scores strongly and significantly correlate with human scores on GOAL, FIN, and REL, whereas correlations for human outputs drop on all but GOAL.
  • 5.2 ANALYZING GPT-4 EVALUATIONS WITH HUMAN EVALUATIONS: GPT-4 scores usually fall within human score ranges, except on SOC and SEC, where it often rates higher than humans.
  • 5.2 ANALYZING GPT-4 EVALUATIONS WITH HUMAN EVALUATIONS: The authors conclude that GPT-4 can cautiously proxy human judgments for model performance on some dimensions, particularly GOAL.They warn that LLM evaluation biases can influence interpretation.

6 EVALUATING SOCIAL INTERACTION BETWEEN LLMS IN SOTOPIA

Across SOTOPIA interactions, GPT-4 performs best on most dimensions, but interactive performance depends strongly on partner quality and includes weaknesses in cooperation, secrecy, and norm adherence.

  • GPT-4 performs best on most dimensions, followed by GPT-3.5, Llama-2-70b-chat, and MPT-30b-chat.
  • Different trends from static benchmarks: Llama-2-70b-chat scores relatively lower than GPT-3.5 across dimensions, diverging from static benchmarks where it is reported as comparable or better.
  • Different trends from static benchmarks: Interactive success requires maintaining personas, advancing conversations, and actively responding to partners, abilities with which weaker models often struggle.
  • Weaker partners models weaken their conversation partners: Underperforming reference models can reduce the overall performance of their interaction partners.
  • All models are at risk of divulging secrets and violating norms: All models receive negative scores in SOC and SEC, indicating risks of divulging secrets and violating social norms even when GPT-4 leads elsewhere.
  • Models sometimes use creative strategies to accomplish goals: Models, especially GPT-4, sometimes use creative strategies such as proposing rest stops or parallel supplier negotiations to accomplish social goals.

7 DIFFERENCES BETWEEN MODELS AND HUMANS IN SOCIAL INTERACTION

SOTOPIA-hard comprises 20 scenarios selected as challenging for GPT-4 and generally challenging for other models. In these tasks, humans outperform GPT-4 on goal achievement and interact more efficiently and strategically.

  • 20 tasks selected as challenging for GPT-4 are also commonly challenging for other models, forming SOTOPIA-hard.
  • Humans perform significantly better than GPT-4 in the GOAL dimension on SOTOPIA-hard.
  • Humans average 16.8 words per turn compared with GPT-4’s 45.5 words, indicating greater conversational efficiency.
  • GPT-4 frequently rephrases the other agent’s utterance before answering, whereas humans typically answer directly.
  • Humans are usually more strategic, opening bargaining below their target and persisting more often with their preferred goals than GPT-4.

8 RELATED WORK

SOTOPIA builds on research in social intelligence, dialogue systems, social-interaction simulation, and multi-agent coordination. It differs from prior benchmarks through interactive, goal-driven, and broader social tasks.

  • The work draws on literature concerning social intelligence, dialogue systems, and simulations of social interactions.
  • Static social intelligence benchmarks: Static social-intelligence benchmarks include clinical-style tests and datasets for social commonsense reasoning.
  • Task-oriented and open-domain dialogue systems: Task-oriented dialogue systems target specific tasks, while open-domain systems are designed for chit-chat rather than broad goal generalization.
  • Simulations of social interactions with LLMs: LLMs have been used to simulate social interactions for platform design, believable human-behavior proxies, and software collaboration.
  • Multi-agent coordination: SOTOPIA’s multi-agent setting avoids assumptions of either zero or extensive knowledge about interaction partners’ policies.

9 CONCLUSION

The paper presents SOTOPIA as an interactive environment for evaluating goal-driven social interactions among models and humans. Its findings support using the platform to assess and potentially enhance social skills in language-based agents.

  • SOTOPIA simulates goal-driven social interactions across varied social scenarios and supports comparisons among agents and humans.
  • SOTOPIA differs from most prior social-intelligence benchmarks by being interactive, goal-oriented, and broad in realistic social tasks.
  • The paper reports that GPT-4 can automate agent-performance evaluation based on SOTOPIA-EVAL.
  • The paper includes appendix material on SOTOPIA’s limitations and future directions, formalization, task generation, evaluation prompts, interfaces, quantitative results, and qualitative examples.

B LIMITATIONS & FUTURE DIRECTIONS

SOTOPIA is presented as a platform for evaluating and improving social intelligence, while its simulated world and automated evaluation introduce important scope and bias considerations.

  • SOTOPIA is proposed as a general platform for realistic evaluation of social intelligence in AI agents.
  • Limitations of the simplified simulated “world”: The simulated world abstracts real social interaction through limited relationship types, character and scenario pools, and dyadic fixed-turn interactions.Future work could expand relationship granularity, scenario coverage, and interaction complexity.
  • Social impact and ethical considerations: Automated evaluation, which is largely supported by GPT-4, may embed social stereotypes and biases.The authors propose investigating when such biases emerge, how they affect evaluation, and how to mitigate them.
  • Social impact and ethical considerations: Training agents to persuade or negotiate with humans may create risks of social manipulation.The authors do not endorse using SOTOPIA to create manipulative agents and plan to release it under the AI2 impact license.
  • Improving LLM social intelligence: SOTOPIA and SOTOPIA-EVAL could support training socially intelligent language agents using automated evaluations as rewards.This direction relies on GPT-4 providing reasonable evaluations for interactions involving humans.

C FORMAL DEFINITIONS AND TECHNICAL DETAILS

SOTOPIA formalizes social interaction as procedurally generated mixed-motive games with vector rewards, role-play characters, and multi-dimensional evaluation.

  • C.1 FORMAL FORMULATION OF THE TASKS IN SOTOPIA: SOTOPIA models interactions as mixed-motive Markov games extending Dec-POMDPs with vector rewards across M social dimensions and procedurally generated tasks.
  • C.1 FORMAL FORMULATION OF THE TASKS IN SOTOPIA: Experiments use N = 2 agents in fixed round-robin turns, although the environment supports any N ≥2.Agent i acts at time step t when t ≡i mod N.
  • C.2.1 CHARACTERS: Character generation includes free-text identity attributes and predefined personality, moral-value, personal-value, and decision-making-style types.GPT-4 also generates secrets and public information, with manual correction of some inconsistent profiles.
  • C.2.1 CHARACTERS: Relationships are generated from family, friend, romantic, acquaintance, and stranger categories, then manually checked for consistency.The generated set contains 31 family, 30 friend, 30 romantic, 29 acquaintance, and 30 stranger pairs.
  • C.2.1 CHARACTERS: Scenarios and social goals are created by refining existing vignettes or using templates, with manual inspection for feasibility and realism.The first method asks GPT-4 to refine vignettes and encourages goals requiring collaborative resolution.
  • C.2.1 CHARACTERS: Template-generated goals encode roles, target prices, penalties, and bonuses for buyer-seller interactions, with prices derived from source data and sampled markups.This process produces 30 scenarios from CraigslistBargins and MutualFriends.
  • D INSTRUCTIONS FOR SOTOPIA-EVAL: SOTOPIA-EVAL uses shared GPT-4 and human instructions covering each dimension, reasoning requirements, and score ranges.The Relationship dimension asks evaluators to assess how interactions change personal relations and social status or reputation.

E.1 INTERACTION DATA

The interaction dataset combines procedurally specified environment profiles with human annotation protocols designed to assess seven social capabilities consistently.

  • The study samples 222 episodes: 180 model-model episodes and 42 episodes involving humans, with two annotators per episode.Annotations average 84.85% pairwise agreement.
  • Environment profiles define social context, social goals, and possible relationship constraints for two-agent interactions.
  • Profile annotation requires plausible, gender-neutral, and mutually consistent scenarios, goals, and constraints.Annotators may modify these elements and assign a binary validity label.
  • Human annotators rate each agent on seven dimensions using 11-point Likert scales after reading dimension-specific instructions and examples.The evaluation task takes about 15 minutes and includes clarification that agents have partial knowledge of one another.
  • Annotator quality is controlled through qualification, rationale review, and re-annotation of episodes with low agreement.The process retained 43 of 235 annotators, with two workers per episode.
  • Agreement analysis groups ratings into equal-width bins, with the main text reporting results using five bins.
  • The human-evaluation interface supports interaction with GPT-4 and entry of dimension scores with free-form rationales.The bot displays instructions but does not participate in the interaction.

G.1 NON-GPT-BASED MODELS FOR EVALUATION

GPT-4 is selected as the strongest tested proxy for human evaluation, while model performance varies by partner and dimension, with qualitative failures in interactive coordination and persona maintenance.

  • GPT-4 is reported as the best proxy for human evaluation among the tested LLMs in a pilot study.
  • Providing evaluators with fine-grained quantitative scale descriptions did not significantly improve results and slightly worsened correlation with humans.
  • Human judgments are summarized as perceived score ranges, with average GPT-4 scores plotted against those ranges for each social dimension.
  • Aggregated model performance is averaged across reference models, while SOTOPIA-hard results and human performance are reported separately.MPT-30b-chat is excluded from human evaluation because of relatively weak SOTOPIA performance.
  • A weaker reference model can reduce the overall performance of its conversation partner.The overall score averages performance across dimensions and model pairs.
  • Qualitative examples: Models may reveal secrets, lack strategic persistence, or be less strategic than humans, although GPT-4 also produces creative solutions in some episodes.
Loading 2310.11667v2…