SOTOPIA-$\pi$: Interactive Learning of Socially Intelligent Language Agents

• LLMs still lag humans in theory of mind, following social norms, and navigating diverse goal-driven social scenarios.Source: §1 · p. 1
• SOTOPIA-π uses generated social tasks, expert and agent interaction data, and GPT-4 ratings to train agents through behavior cloning and self-reinforcement.Source: §1 · pp. 1–2
• The behavior-cloning-plus-self-reinforcement model approaches GPT-4’s social goal-completion performance under GPT-4-based evaluation.Source: §1 · p. 2
• Training widens the gap between GPT-4-based and human evaluation, exposing limitations in relying solely on GPT-4 to optimize or evaluate social performance.Source: §1 · p. 2
• The method improves safety and reduces toxicity in social tasks while preserving models’ MMLU question-answering ability.Source: §1 · p. 2

2.1 SOTOPIA environment

• SOTOPIA tasks combine scenarios, character profiles, and private social goals across negotiation, collaboration, and competition, with agents communicating through speech, non-verbal behavior, and actions.Source: §2.1 · p. 2
• SOTOPIA-EVAL rates social intelligence across seven dimensions, with GPT-4 and human evaluators assigning Likert-scale scores.Source: §2.1 · p. 3

2.2 Interactive learning

• Behavior cloning learns from expert behavioral trajectories; because extensive high-quality human conversation data is difficult to collect, the authors use GPT-4-based agents as experts.Source: §2.2 · p. 3
• Ratio-based filtering enables self-reinforcement without iterations, addressing threshold tuning costs and poor performance across tasks with varied difficulty.Source: §2.2 · p. 3

Step 1: Social task generation

• GPT-4 generates scenarios and social goals from sampled social-activity keywords, while training tasks use randomly paired SOTOPIA character profiles and exclude evaluation tasks.Tasks are produced automatically without manual quality filtering; the authors report gains from large quantities of lower-quality tasks and suggest that more selective filtering could improve results further.Source: Step 1: Social task generation · pp. 3–4

Step 2: Training data collection

• Behavior cloning uses expert-policy interactions, while self-reinforcement uses the agent policy’s interactions; expert data can be costly or unavailable.The authors report that one behavior-cloning round with a GPT-4-based expert can surpass a GPT-3.5-based agent, and leave human conversation trajectories for future work.Source: Step 2: Training data collection · p. 4

Step 3: Agent policy update

• GPT-4 rates goal completion on a 0–10 scale, and examples above a score threshold become supervised fine-tuning data.Each interaction turn is parsed into input-output training pairs; the input includes task information visible to the agent and conversation history.Source: Step 3: Agent policy update · p. 4
• GPT-4-based evaluation uses the combined behavior-cloning-then-self-reinforcement model as the sequential policy-update approach.The policy is first updated by behavior cloning, then generates interaction data for self-reinforcement.Source: Step 3: Agent policy update · p. 4

Agent models

• Mistral-7B is trained with behavior cloning, self-reinforcement, or both sequentially, with GPT-4 as expert and GPT-3.5-turbo as the fixed interaction partner baseline.GPT-4 and Mistral-7B also serve as prompting-based baselines; all models share a prompt format and use few-shot prompting.Source: §4 · p. 5

Training

• QLoRA fine-tuning uses 100 GPT-4-generated social tasks per training round and 10 interactions per task across different character pairs.The tasks cover negotiation, collaboration, and competition.Source: §4 · p. 5

Evaluation

• Human and GPT-4 evaluators rate agents on seven social dimensions, with performance reported on all 90 tasks and a 14-task hard subset.The hard-task figure compares the base model, trained agents, and GPT-4 on goal completion.Source: §4 · p. 5, Figure 4 · p. 5

5 Does SOTOPIA-π improve the social intelligence of language agents?

• 5.71 (ours) vs 5.89 (GPT-4) for goal completion on hard SOTOPIA tasks, as rated by GPT-4, with behavior cloning followed by self-reinforcement.The study reports similar improvement trends on all SOTOPIA tasks, where BC+SR is comparable to the expert model.Source: §5 · pp. 5–6, Table 2 · p. 6
• Human goal-completion scores show a larger gap between BC+SR and the GPT-4 agent than GPT-4-based scores do: 4.29 versus 5.25.GPT-4-based evaluation increasingly overestimates models trained specifically for social interaction, motivating more robust evaluation models.Source: §5 · pp. 5–6
• Human evaluations on hard social tasks show improvements in believability, relationship, social rules, and overall score, with only slight effects on other dimensions.Table 1 reports improvements for BC+SR over the Mistral-7B base model.Source: §5 · p. 6, Table 1 · p. 6
• Human evaluation was not conducted on all SOTOPIA tasks because of its high cost.The all-task results cited here are GPT-4-based evaluations.Source: §5 · p. 6, Table 2 · p. 6

6 How does SOTOPIA-π influence other capabilities of LLMs

• The safety evaluation uses one scenario in which one character seeks to injure a third person and another seeks to prevent violence.The study considers nine examples for each of five agent models and manually labels behavioral quantities, including engagement, injury intent, prevention, alternatives, and toxic words.Source: §6 · pp. 6–7
• In the examined task, trained models engage more, are safer and more persuasive, and use fewer toxic words, though the best method still produces more toxic words than GPT-4.The best model is less likely to pursue an injury plan and more likely to persuade the other agent to abandon it; it also proposes more alternatives than the expert model.Source: §6 · p. 7, Table 3 · p. 7
• The best SOTOPIA-π models perform comparably to the base model on MMLU, preserving its question-answering capability.MMLU is evaluated using standard 5-shot instruction-based prompting.Source: §6 · p. 7, Table 4 · p. 7

7 Related work

• Prior studies find that LLMs struggle with theory-of-mind tasks and with deception, deduction, and negotiation in multi-agent settings; the authors leave explicit theory-of-mind modeling for future work.Work on behavior cloning from observational interaction data and inferential social learning provides related approaches.Source: §7 · pp. 7–8
• Unlike online RL methods that often require data collection with longer latency in multi-agent settings, SOTOPIA-π applies offline self-reinforcement using GPT-4 rewards for multi-turn social interaction.The paper leaves comparison of different offline methods for training social intelligence to future work.Source: §7 · p. 8
• The paper evaluates alignment with human social norms through safety and toxicity, and uses MMLU to test general question-answering and reasoning capabilities.The related work notes that continual fine-tuning can lead to forgetting domain knowledge, reasoning, and reading comprehension.Source: §7 · p. 8

8 Conclusion and future work

• SOTOPIA-π improves social intelligence without loss of general QA ability and with improved safety.Source: §8 · p. 8
• The method is offline and cannot improve iteratively; future work could explore online reinforcement learning without costly LLM ratings.Source: §8 · p. 8

Limitations

• GPT-4-based and human evaluations diverge increasingly as models optimize GPT-4 scores, motivating more robust evaluation and learning signals.Source: §8 · p. 8, Limitations · p. 9
• The safety study covers only one social task, leaving other social-alignment dimensions, including privacy, fairness, and reliability, unstudied.Source: §8 · p. 8, Limitations · p. 9
• GPT-4-generated content in the interactive environment may introduce social biases and stereotypes into agent training.Source: Limitations · p. 9

Ethical Statement

• The study focuses on developing social intelligence rather than human-indistinguishable systems, while recognizing that strategic interactions can have unexpected outcomes and negative impacts.Source: Ethical Statement · p. 9

A Detailed Results

• The appendix reports detailed seven-dimension results and paired t-tests alongside the overall score.Source: A Detailed Results · p. 13

A.1 Main Results

• Table 5 compares baselines and training settings across three SOTOPIA data settings, seven social-intelligence dimensions, and an overall score.Source: Table 5 · p. 13, §A.1 · p. 13

A.2 Statistic Test

• Table 6 reports paired t-tests comparing BC+SR with other methods; positive t-values favor BC+SR, and p-values below 0.05 indicate significance.Source: §A.2 · p. 13, Table 6 · p. 14

B.1 Social Task Generation

• The authors generated 462 social tasks from 510 unused prompts sampled equally across Social Chemistry, Social IQa, and Normbank, excluding sources for availability or similarity reasons.Source: §B.1 · pp. 14–15

B.2 Interaction Data Filtering Strategy

• Behavior cloning filtering uses within-task ranks and global score thresholds, while self-reinforcement keeps the top 20 percent by within-task goal-score rank.Source: §B.2 · p. 15

B.3 Training Data Format

• Training inputs combine the social task and conversation history; outputs are JSON strings containing the agent’s action type and current response content.Source: §B.3 · p. 15

B.4 Involved Model Versions

• The experiments use GPT-4 gpt-4-0613, GPT-3.5 gpt-3.5-turbo-0613, and Mistral-7B-Instruct-v0.1, with GPT-4 task generation using gpt-4-1106-preview.Source: §B.4 · p. 16

B.5 Training Setup

• Mistral checkpoints were trained for 20 epochs on four A6000 80G GPUs using QLoRA, batch size 4, and a 5.0e-5 initial learning rate.The cutoff length was 4096; QLoRA rank, alpha, and dropout were 8, 16, and 0.05.Source: §B.5 · p. 17

B.6 Checkpoint Selection

• The selected checkpoint is epoch 20 for behavior cloning and epoch 5 for self-reinforcement, based on training loss.Source: §B.6 · p. 17

C.1 Social Interaction Data for Annotation

• Human evaluation used 14 hard scenarios selected from SOTOPIA’s 90 scenarios, sampling two agent pairs per scenario to produce 28 examples per model setting.Source: §C.1 · p. 17

C.2 Human Annotation System

• Annotators received definitions, examples, and scoring guidance for seven social dimensions, then scored both agents and supplied reasoning for each dimension.Source: §C.2 · pp. 17–18, Figure 5 · p. 18, Figure 6 · p. 19

C.3 Human Annotator Selection

• The team invited 30 qualified Prolific annotators and four internal annotators after a qualification process combining score-range screening with manual review of reasoning.Source: §C.3 · pp. 18, 20

C.4 Demographic and Geographic Information about Human Annotators

• Annotators were recruited from the United Kingdom and United States, with ages ranging from 23 to 53 among qualified annotators.Source: §C.4 · p. 20

C.5 Human Annotation Data Collection

• About 20% of collected annotation data points were filtered, with roughly 1.8% of annotations separately removed for uncorrelated reasoning.Each datapoint was assigned to two annotators; filtering also checked reasoning quality, score consistency, and strong disagreement.Source: §C.5 · pp. 20, 22
• Target-model identities were concealed from annotators and the filtering author, and either conversation side could trigger filtering.Source: §C.5 · p. 22

C.7 Human Annotator Consent

• Annotators consented to academic use of their data.Source: §C.7 · p. 22

C.8 Correlation between Automatic Evaluation and Human Evaluation

• GPT-4-prompting-based automatic evaluation had high Pearson correlation with human evaluation across training settings for goal-completion scores.Source: §C.8 · p. 22, Table 7 · p. 23

C.9 Inter-annotator Agreement

• The study reports pairwise agreement and Randolph’s Kappa because annotators were not paired across datapoints.Source: §C.9 · p. 22, Table 8 · p. 23

C.10 Additional Human Evaluation Results

• As target models improved through behavior cloning and self-reinforcement, their fixed GPT-3.5-turbo conversation partner also improved in goal-completion and overall scores.The authors interpret this pattern as indicating greater likelihood of agreement and mutual satisfaction of requirements.Source: §C.10 · p. 23, Table 9 · p. 23

D LLM Safety

• The appendix presents example model responses when asked to injure a person and when acting to prevent violence, without specifying a comparative outcome in these passages.Source: D LLM Safety · p. 24, Figure 9 · p. 24, Figure 10 · p. 24

E LLM Secret Keeping Ability

• The evaluation selects SOTOPIA tasks asking both agents to reveal a secret without disclosing that it is theirs.Source: E LLM Secret Keeping Ability · p. 25
• Across 10 setups, the behavior-cloned model usually does not discuss the secret, but when it reveals one, it does so explicitly and fails to hide the identity.Not discussing the secret could count as successfully hiding identity, though it does not accomplish the reveal goal.Source: E LLM Secret Keeping Ability · p. 25
• GPT-3.5 almost always explicitly reveals the secret without hiding identity, often phrasing it exactly as in the profile background.It discusses irrelevant content less often than the behavior-cloned model.Source: E LLM Secret Keeping Ability · p. 25
• GPT-4 uses wrappers such as “heard a story” or “a friend of mine” to hide identity when revealing secrets, and teaches the other agent to use similar phrasing.Source: E LLM Secret Keeping Ability · p. 25

F Detailed MMLU Results

• The MMLU results are shown as per-subject comparisons across 57 subjects, with Figures 12–14 covering the model comparisons.The supplied passages do not state the per-subject outcomes.Source: F Detailed MMLU Results · pp. 25–26, Figure 12 · p. 27, Figure 13 · p. 28

G Contributions

• All authors contributed to writing; listed roles include co-leads, co-advisors, and an overall project lead.Source: G Contributions · p. 26