PaperlayerView exact revision on arXiv

Source-linked AI summary

Sotopia-RL: Reward Design for Social Intelligence

Haofei Yu, Zhengyang Qi, Yining Zhao, Kolby Nottingham, Keyang Xuan, Bodhisattwa Prasad Majumder, Hao Zhu, Paul Pu Liang, Jiaxuan You

arXiv:2508.03905v3cs.CL

TL;DR

Social interactions make reward design difficult because utterance quality may not track episode success and social outcomes span multiple dimensions. SOTOPIA-RL refines episode feedback into utterance-level, multi-dimensional rewards for RL training. On SOTOPIA, it reports goal completion scores of 7.17 on SOTOPIA-hard and 8.31 on the full dataset, outperforming baselines.

  • Problem

    Social RL lacks a unified reward design for diverse scenarios, while utterance quality may be loosely related to success and social interactions require multidimensional evaluation.

    Source: §1 · p. 1

  • Method

    SOTOPIA-RL uses offline LLM-based, rubric-guided utterance reward assignment to train an utterance-level reward model for online RL.

    Source: §1 · p. 2

  • Results

    7.17 on SOTOPIA-hard and 8.31 on the full SOTOPIA dataset are the goal completion scores achieved by SOTOPIA-RL agents, which outperform all baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that both reward components are essential for stable, effective RL training and that the framework mitigates reward hacking.

    Source: §8 · p. 10

    The authors acknowledge potential social or cultural biases from using models such as GPT-4o for automated evaluation.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Social intelligence has become a critical capability for large language models (LLMs), enabling them to engage effectively in real-world social tasks such as collaboration and negotiation. Reinforcement learning (RL) is a natural fit for training socially intelligent agents because it allows models to learn sophisticated strategies directly through social interactions without requiring human annotations. However, there are two unique parts about social intelligence tasks: (1) the quality of individual utterances in social interactions is not strictly related to final success; (2) social interactions require multi-dimensional rubrics for success. Therefore, we argue that it is necessary to design rewards for building utterance-level multi-dimensional reward models to facilitate RL training for social intelligence tasks. To address these challenges, we propose Sotopia-RL, a novel framework that refines coarse episode-level feedback into utterance-level, multi-dimensional rewards. Utterance-level credit assignment attributes outcomes to individual utterances, while multi-dimensional rewards capture the full richness of social interactions and reduce reward hacking. Experiments in Sotopia, an open-ended social learning environment, demonstrate that Sotopia-RL achieves state-of-the-art social goal completion scores (7.17 on Sotopia-hard and 8.31 on Sotopia-full), significantly outperforming existing approaches. Ablation studies confirm the necessity of both utterance-level credit assignment and multi-dimensional reward design for RL training.

1 INTRODUCTION

Social tasks challenge RL reward design because individual utterances may not track final success, while interactions involve multiple dimensions. SOTOPIA-RL addresses this with utterance-level, multi-dimensional rewards and achieves strong goal-completion results.

1 INTRODUCTION

  • Social utterance quality can be loosely related to final success, and interactions require evaluation across dimensions such as task progress, rapport, and conversational flow.For example, a misleading negotiation claim may still help secure a better outcome.Source: §1 · p. 2
  • LLMs can infer which utterances helped or hindered success, while separate rubric dimensions simplify credit assignment and reduce reward-estimate variance.The cited empirical observation is strong attribution agreement across diverse LLM families, with Spearman correlations above 0.7.Source: §1 · p. 2
  • SOTOPIA-RL collects episode outcomes offline, uses LLMs and multi-dimensional rubrics for utterance-level reward assignment, then trains a reward model for online RL.The proposed practical reward-design recipe is intended to train agents to generate high-quality social utterances.Source: §1 · p. 2
  • 7.17 on SOTOPIA-hard and 8.31 on the full SOTOPIA dataset are the goal completion scores achieved by agents trained with SOTOPIA-RL, which outperform all baselines.The experiments use SOTOPIA, an open-ended social learning environment.Source: §1 · p. 2
  • The authors report that utterance-level and multi-dimensional reward design is critical for stable, effective RL training in complex social scenarios.They also identify evaluating interaction quality across diverse dimensions as important.Source: §1 · p. 3

2 RELATED WORK

Prior social-agent training has used varied reward and feedback strategies, while process and multi-objective reward modeling offer related approaches. This paper focuses on utterance-level, multi-dimensional reward learning tailored to social tasks.

2 RELATED WORK

  • A social utterance is a deliberate communicative act pursuing social goals; sequences of utterances form multi-turn interaction episodes.Utterances can convey intentions and emotions that shape the conversation.Source: §2 · p. 3
  • Prior social-learning methods use self-reinforcement, conversation-level rewards, rule-based peer feedback, or preference tuning, while SOTOPIA-RL introduces utterance-level reward modeling.The paper distinguishes this approach from methods that inject explicit strategies during training.Source: §2 · p. 3
  • Social utterance-level reward design resembles process reward modeling, but social ambiguity and multidimensionality call for evaluation beyond math and programming settings.The related work notes that process reward models have been effective for RL with verifiable rewards.Source: §2 · p. 3
  • For social tasks, the paper uses multi-dimensional reward learning with linear scalarization, including auxiliary rewards for relationship maintenance and knowledge seeking.Related multi-objective work also explores nonlinear utilities and reward decomposition.Source: §2 · p. 3

3 SOCIAL LEARNING ENVIRONMENT

The environment generates social dialogue trajectories and episode-level feedback. It models agents as partially observable conversational decision-makers and evaluates outcomes with a primary goal score plus auxiliary dimensions.

3 SOCIAL LEARNING ENVIRONMENT

  • The environment must generate simulated social-interaction data and provide training feedback; it uses multi-turn agent dialogues and LLM-based episode evaluation.Source: §3 · p. 3

3.1 SOCIAL INTERACTION

  • Social interaction is modeled as a POMDP: agents observe dialogue history and contextual cues, produce verbal or nonverbal utterances, and receive partner responses; an MDP approximation supports RL training with utterance-level rewards.The state approximation combines dialogue history with the agent’s private goal, and the reward model scores each utterance.Source: §3.1 · pp. 3–4

3.2 SOCIAL EVALUATION

  • The LLM evaluator produces goal completion as the primary training signal and six auxiliary rubric dimensions for richer analysis of interaction quality.The auxiliary dimensions are believability, knowledge seeking, relationship maintaining, secret keeping, social rule-following, and financial/material benefits.Source: §3.2 · p. 4

4 METHODOLOGY

SOTOPIA-RL turns coarse episode feedback into offline, utterance-level, multi-dimensional rewards, then distills them into a reward model for online policy training. Its design uses full-episode context for offline attribution and dialogue history available up to the current turn for online scoring.

4 METHODOLOGY

  • SOTOPIA-RL collects offline utterance-level, multi-dimensional rewards and uses them to train a reward model and conduct online RL.Offline labels use the full dialogue, while the online reward model conditions only on dialogue history up to the current utterance.Source: §4 · p. 4, Figure 3 · p. 5

4.1 SOCIAL REWARD DESIGN WITH LLMS

  • Full-episode LLM attribution assigns each utterance a contribution score, which is refined with the episode outcome to emphasize successful contributions and down-weight contributions in failed episodes.The design addresses weak links between individual utterances and final success by evaluating each utterance in the full dialogue context.Source: §4.1 · p. 5
  • The rubric scores utterances across seven evaluation dimensions, normalizes each dimension, and combines them with a weighted average.Dimensions can be tailored to social contexts; relationship maintenance and knowledge seeking are reported as particularly important for goal achievement, potentially by supporting conversational flow.Source: §4.1 · pp. 5–6

4.2 SOCIAL AGENT TRAINING WITH RL

  • The online reward model is trained to predict designed utterance rewards from the current interaction context, translating offline labels into turn-level signals.The model supports stable and fine-grained RL optimization.Source: §4.2 · p. 6
  • Policy training warms up with behavior cloning on GPT self-play rollouts, then uses single-turn GRPO updates while retaining multi-turn information distilled into the reward model.The policy deliberately omits explicit inference-time reasoning traces.Source: §4.2 · p. 6

5 EXPERIMENTAL SETUP

Experiments use Qwen2.5-7B-Instruct for policy and reward models, and evaluate on SOTOPIA-hard and SOTOPIA-all. The setup compares training, attribution, and reward-aggregation designs.

5 EXPERIMENTAL SETUP

  • SOTOPIA-hard covers 14 challenging scenarios with 10 agent pairings each, while SOTOPIA-all covers 90 scenarios with two pairings per scenario.Evaluation reports believability, relationship building, knowledge seeking, goal completion, and average scores; GPT-4o serves as judge.Source: §5 · p. 6
  • Table 1 reports a state-of-the-art comparison with GPT-4o as partner; Table 2 compares reward designs with BC as partner, with p < 0.05 on GOAL.Figures 5–6 report consistent improvement across LLM evaluators and partner models.Source: Table 2 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8
  • SOTOPIA-RL combines direct utterance-level attribution with REL, KNO, and GOAL rewards, trained using single-turn GRPO without explicit reasoning.The comparison includes behavior cloning, SOTOPIA-π, PPDPP, EPO, DAT, and DSI.Source: §5 · p. 7
  • Attribution baselines distribute episode reward uniformly, assign it to one utterance, or scale it across utterances; DIRECT independently attributes normalized weights to each utterance.DIRECT is the attribution method used in SOTOPIA-RL.Source: §5 · p. 7
  • SOTOPIA-RL averages REL, KNO, and GOAL rewards; single-dimension baselines isolate BEL, GOAL, KNO, or REL.The setup treats dimensions equally because the ablation study found weight choice had little impact on performance.Source: §5 · p. 7

6 EXPERIMENTAL RESULTS

SOTOPIA-RL reports a 7.17 goal-completion score on SOTOPIA-hard, while the authors qualify comparisons with AMPO because it uses explicit reasoning and substantially more tokens. Reward-attribution and reward-aggregation ablations also report gains, while combined rewards delay convergence relative to GOAL alone.

6 EXPERIMENTAL RESULTS

  • Combined rewards delay convergence on the GOAL score curve compared with using the GOAL reward alone.The figure caption describes the training-process curve.Source: Figure 7 · p. 8
  • On SOTOPIA-hard, Qwen-2.5-7B-Instruct trained with SOTOPIA-RL achieves the benchmark’s highest goal-completion score, 7.17, indicating that utterance-level reward modeling better guides RL training.AMPO reaches 7.50 but uses explicit reasoning and averages more than 640 inference tokens per utterance, so the authors consider the comparison unfair.Source: §6 · p. 8
  • 7.17 versus 6.97: SOTOPIA-RL surpasses GPT-4o used directly as a policy model, which the authors cite as evidence that their method goes beyond distillation.Offline GPT annotations condition on the entire episode, whereas online RL rewards condition only on preceding dialogue history.Source: §6 · p. 8
  • 6.74 to 7.21: direct reward attribution raises goal-completion score compared with the attribution baselines discussed by the authors.They report that relaxing attribution constraints gives LLMs more freedom to score within minimal range limits.Source: §6 · p. 8
  • 7.9% gain in GOAL: combining GOAL, REL, and KNO performs best, while optimizing each dimension independently also improves performance on the others.The authors report that the objectives are correlated and that combining them makes reward-model training more robust.Source: §6 · p. 8

7 DISCUSSION

SOTOPIA-RL’s gains hold across partner and evaluator models and extend to human evaluation. The discussion attributes improvements to offline, full-dialogue reward attribution and complementary reward dimensions rather than reward scaling alone.

7 DISCUSSION

  • Improvements persist across five partner models and five evaluator models, and human evaluation also confirms gains over other baselines.The authors present these results as evidence against overfitting to particular partners or evaluators.Source: §7 · p. 9, Table 3 · p. 9
  • 7.81 goal score is achieved with offline full-episode labels, compared with online labels based only on preceding dialogue history.The two reward-model settings use the same training settings.Source: §7 · p. 9
  • Weaker LLMs still produce highly correlated utterance-level labels (>0.7), so the reported attribution improvement does not rely on GPT-4o.Figure 8 reports high pairwise correlation between different LLMs’ utterance-level reward labels.Source: §7 · p. 9, Figure 8 · p. 9
  • The 10-point scale scores lower than the 3-point scale on GOAL (6.44 vs. 6.74), indicating that reward scaling alone does not explain the benefits.The comparison tests whether finer attribution granularity accounts for the observed gains.Source: §7 · p. 9
  • Knowledge, relationship, and goal rewards show positive but moderate correlations, while combined rewards stabilize training and regularize the single-dimension objective later.The authors interpret the correlations as evidence that the objectives capture distinct facets of social interaction.Source: §7 · p. 9

8 CONCLUSION

SOTOPIA-RL combines reward attribution and aggregation to train socially intelligent agents, with both components reported as essential. The authors identify personalized rewards and multi-agent group settings as directions for extending the framework.

8 CONCLUSION

  • SOTOPIA-RL combines reward attribution and reward aggregation to provide fine-grained, task-aligned supervision while mitigating reward hacking.The conclusion reports that both components are essential and that the framework achieves state-of-the-art performance on SOTOPIA.Source: §8 · p. 10
  • Experiments on the SOTOPIA benchmark demonstrate state-of-the-art performance, with both reward components reported as essential.Source: §8 · p. 10
  • The authors suggest personalized rewards and multi-agent group settings as extensions for applications such as negotiation and collaborative problem solving.Source: §8 · p. 10

ETHICS STATEMENT

The ethics statement describes role-play design choices intended to mitigate anthropomorphization and acknowledges possible bias in automated evaluation. The appendix identifies comprehensive evaluation results across dimensions and SOTOPIA benchmarks.

ETHICS STATEMENT

  • The project aims to advance social capabilities without replicating human identity or creating systems indistinguishable from people.Source: ETHICS STATEMENT · p. 10
  • Role-play scenarios avoid consistent human-like identities to mitigate anthropomorphic tendencies and associated user risks.The statement notes risks including inappropriate expectations, undue influence, and manipulative scenarios.Source: ETHICS STATEMENT · p. 10
  • The authors acknowledge potential social or cultural biases from GPT-4o-based automated evaluation and commit to ongoing analysis and mitigation.Source: ETHICS STATEMENT · p. 10

A.1 FULL MAIN RESULTS

  • Table 5 reports evaluation across all SOTOPIA dimensions on SOTOPIA-hard and SOTOPIA-all.The table caption identifies the BC model as partner and GPT-4o as evaluator.Source: §A.1 · p. 16, Table 5 · p. 16

A.2 FULL ABLATION RESULTS

The appendix reports ablations of reward aggregation and robustness, a Best-of-N evaluation, and safety and diversity assessments. It finds uniform averaging robust, while adding believability lowers final performance.

A.2 FULL ABLATION RESULTS

  • Ablations across partner and evaluator models assess robustness and provide evidence against reward hacking or overfitting to particular models.Table 6 varies partner models with GPT-4o fixed as evaluator, then varies evaluators with BC fixed as partner.Source: §A.2 · p. 16, Table 6 · p. 17
  • Changing reward-dimension weights consistently hurts performance, supporting uniform averaging; adding believability lowers final performance.The authors suggest believability may be less related to goal completion and may distract reward-model training.Source: §A.2 · p. 16, Table 7 · p. 17

A.3 BEST-OF-N EVALUATION RESULTS

  • Best-of-N evaluation ranks sampled responses with reward models trained using different attribution and reward-dimension settings, then selects the top candidate.Source: §A.3 · p. 17, Table 8 · p. 18

A.4 SAFETY EVALUATION RESULTS

  • SOTOPIA-RL preserves safety performance relative to BC and produces fewer toxic continuations under benign prompts.Source: §A.4 · p. 17

A.5 DIVERSITY EVALUATION RESULTS

  • Under matched tasks and partners, SOTOPIA-RL yields more dialogue turns and longer utterances than BC, Sotopia-π, and GOAL-RL.The authors interpret these proxies as evidence of sustained interaction and richer contributions rather than terse, templated collapse.Source: §A.5 · p. 18

B ADDITIONAL ANALYSIS OF UTTERANCE-LEVEL REWARD

The analysis reports a smoother, lower-variance distribution for combined rewards and high alignment between GPT-4o reward annotations and human judgment. It also describes correlation and interaction-length measures used in the supplementary analyses.

B ADDITIONAL ANALYSIS OF UTTERANCE-LEVEL REWARD

  • Combined rewards have lower variance and a smoother distribution than individual dimensions, suggesting a more stable estimator of the noisy latent social state.Rewards are normalized to [0,1].Source: B ADDITIONAL ANALYSIS OF… · p. 18
  • Human–GPT-4o annotation correlations are high, while correlations among human annotators are also described as quite high.Table 10 presents the Pearson correlation matrix for four independent human annotators and GPT-4o.Source: B ADDITIONAL ANALYSIS OF… · p. 18, Table 10 · p. 19
  • The diversity evaluation reports word counts and turn counts for social interactions across models.The supplied passage identifies the measures but does not give the table's values.Source: Table 11 · p. 19

C LIMITATIONS AND FUTURE WORKS

The supplementary material documents future research directions alongside artifacts, data, experimental settings, and human-evaluation procedures. It identifies synthetic SOTOPIA dialogue data, provides reproducibility resources, and describes the annotation interface and guidance.

C LIMITATIONS AND FUTURE WORKS

  • Future work includes stronger multi-turn RL, direct human–agent evaluation, and testing whether protocols capture real-world risks of manipulative or deceptive behavior.The authors note that current testing protocols may not fully capture those risks.Source: C LIMITATIONS AND FUTURE WORKS · p. 19

D.1 ARTIFACT INFORMATION

  • The artifact provides code, checkpoints, datasets, and evaluation data, and is described as publicly accessible and documented for reproducing the results.It includes training and evaluation pipelines, reward annotations, and trained reward-model and final RL checkpoints.Source: §D.1 · p. 19

D.2 ARTIFACT LICENSE

  • The listed models and datasets carry Apache 2.0, MIT, or proprietary licenses, with model use limited to research purposes and SOTOPIA data used for academic research.The SOTOPIA framework is described as permitting research and commercial reuse.Source: §D.2 · pp. 20–21

D.4 DATA STATISTICS

  • The study uses synthetic SOTOPIA dialogue data; evaluation covers SOTOPIA-all's 90 tasks and SOTOPIA-hard's 14-task subset.The material also describes self-play data from 100 scenarios and GPT-4o-generated reward annotations.Source: §D.4 · p. 21

E.1 ACRONYM FOR EXPERIMENTAL SETTINGS

  • The experimental acronym list distinguishes SOTOPIA-RL's multi-dimensional reward modeling from GOAL-RL's use of only the GOAL reward dimension.It also defines BC, GRPO, SOTOPIA-π, and self-reinforcement.Source: §E.1 · p. 22

E.2 MODEL SIZE AND BUDGET

  • Qwen2.5-7B-Instruct is the main policy and reward-model backbone, with LoRA fine-tuning and 4-bit policy quantization for GRPO.The reported training budgets are about one hour for BC, five hours for reward modeling, and 24 hours for GRPO on A100 80GB GPUs.Source: §E.2 · p. 22

E.3 HYPERPARAMETER FOR EXPERIMENTS

  • The appendix specifies training hyperparameters, model identifiers, and SOTOPIA evaluation platform version 0.1.0rc5.It lists separate learning rates and batch settings for BC, reward-model, and GRPO training.Source: §E.3 · p. 22, §E.4 · p. 23, §E.5 · p. 23

F.1 HUMAN ANNOTATION SYSTEM

  • The human annotation system presents dimension definitions, examples, and scoring guidance before annotators evaluate dialogue data.The guidance covers believability, relationship, knowledge, secret keeping, social rules, material benefits, and goal completion.Source: §F.1 · p. 23

F.2 ANNOTATION DATA PREPARATION

The appendices describe human-evaluation preparation, utterance reward attribution, reward dimensions, and illustrative conversations. They detail annotation procedures and attribution variants, then show examples in which SOTOPIA-RL combines goal pursuit with social and informational considerations.

F.2 ANNOTATION DATA PREPARATION

  • Human evaluation samples all 14 SOTOPIA-hard scenarios, using two agent pairs per scenario and 28 conversations for each of four training methods.The compared methods are Behavior Cloning, SOTOPIA-π, Goal-RL, and SOTOPIA-RL.Source: §F.2 · pp. 23–24

F.3 HUMAN ANNOTATOR INFORMATION

  • Four internal annotators calibrated their judgments by independently rating five representative conversations and agreeing on consensus scores.The annotators reviewed their scores and discussed discrepancies before the full annotation set.Source: §F.3 · p. 24

F.4 ANNOTATION PROCESS

  • Two qualified annotators score each agent's goal completion from 0 to 10 using the full transcript and both agents' goals, without knowing the training methods.Higher scores indicate greater progress toward the stated goal.Source: §F.4 · pp. 24, 26

ATTRIBUTION TEMPLATE

  • The attribution template asks an LLM to score every utterance by the evaluated agent for its contribution to a selected social dimension.The prompt supplies dialogue history, the agent's goal and background, and the dimension description.Source: ATTRIBUTION TEMPLATE · p. 27

DIRECT ATTRIBUTION

  • Direct attribution judges each utterance's importance using its contribution to the dimension and the partner's responses, rather than utterance quality alone.The instructions assign zero to utterances with no impact and request a score for each utterance by the selected agent.Source: DIRECT ATTRIBUTION · p. 28

H ADDITIONAL DETAILS ABOUT REWARD ATTRIBUTION

  • Scaled attribution normalizes direct scores to sum to the episode's final goal score, singular attribution assigns that score to key utterances, and uniform attribution divides it evenly.Uniform attribution serves as a control against more content-sensitive methods.Source: §H.3 · p. 28, §H.4 · p. 28, §H.5 · p. 29

I ADDITIONAL DETAILS ABOUT REWARD DIMENSIONS

  • The rubric combines GOAL, REL, and KNO: goal achievement, relationship or status effects, and acquisition of new, important information.The GOAL dimension concerns progress on the agent's environment-defined social goals.Source: I ADDITIONAL DETAILS ABOUT REWARD… · p. 29

J CASE STUDY

  • In a case study, SOTOPIA-RL integrates goal pursuit, relationship-building, and informativeness within an utterance, and other examples show collaborative solutions and multi-turn strategies.The illustrated examples include proposing an exchange and reaching a practical collaborative solution.Source: J CASE STUDY · p. 30, Figure 14 · p. 31, Figure 15 · p. 32

Related papers

Loading 2508.03905v3…