PaperlayerView exact revision on arXiv

Source-linked AI summary

SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design

Jianing Wang, Xintao Wang, Aili Chen, Jie Shi, Hongcheng Guo, Jun Gao, Wenxuan Zhao, Chengkun Lang, Yuanli Guo, Yanghua Xiao

arXiv:2609.09764v1cs.CL

TL;DR

Existing reinforcement learning methods often optimize single utterances or sparse outcomes, limiting long-horizon social dialogue optimization. SocialRL addresses this with PPO over complete trajectories and adaptive, multi-dimensional process rewards, improving Goal Achievement by 9.2 percentage points on average over corresponding Base models.

  • Problem

    Existing methods provide limited supervision for intermediate social behavior and often optimize single-turn utterances or sparse outcomes rather than multi-turn goal-relationship trade-offs.

  • Method

    SocialRL uses PPO and a value network to optimize complete dialogue trajectories, combining delayed outcome rewards with six-dimensional, rubric-based process rewards and stage-aware weights.

  • Results

    9.2 percentage points: SocialRL improves Goal Achievement on average across four social-dialogue benchmarks relative to corresponding Base models.

  • Takeaways & Limitations

    SocialRL improves performance across synthetic and real social scenes, including standard and challenging scenarios, against corresponding Base models and compared non-commercial baselines.

  • Takeaways & Limitations

    The counterpart’s private goal is hidden and responses are stochastic, requiring conversation-level optimization under evolving dialogue states.

Abstract

from arXiv · show

Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue. As language models become autonomous collaborators, it is central to building effective and trustworthy human-AI interaction. Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards, producing short-sighted policies that struggle to manage goal-relationship tensions across multi-turn interactions. We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges. First, we apply multi-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning. Second, we design six process reward dimensions capturing the goal-relationship trade-off, including goal advancement, relational attunement, contextual coherence, etc. A reward model dynamically generates fine-grained scoring criteria for each dimension, while a stage-aware weight schedule prioritizes relationship-building in early turns, goal advancement mid-way, and balanced closure late. Across multiple social-dialogue benchmarks, SocialRL improves Goal Achievement by an average of 9.2 percentage points over the corresponding Base models. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.

1 INTRODUCTION

SocialRL addresses the difficulty of balancing goal pursuit and relationship management across multi-turn social dialogue. It combines trajectory-level PPO optimization with dynamic process rewards and improves Goal Achievement by 9.2 percentage points on average over corresponding Base models.

  • Social dialogue requires agents to pursue private goals while preserving relationships as interactions unfold over multiple turns.
  • Existing approaches optimize isolated utterances or rely on final outcome rewards, providing limited supervision for cumulative social trade-offs.
  • SocialRL optimizes complete dialogue trajectories with PPO and a value network using turn-level rewards.
  • SocialRL decomposes social quality into six process-reward dimensions and adapts their weights across dialogue stages.
  • 9.2 percentage points: SocialRL's average Goal Achievement improvement over corresponding Base models across four benchmarks.
  • SocialRL policies outperform compared non-commercial baselines, while commercial reference models remain an upper bound in several settings.

2 RELATED WORK

Prior work evaluates or improves social behavior, but existing methods typically address benchmarks, imitation or preference learning, or single-turn reinforcement learning separately. SocialRL combines full multi-turn trajectory optimization with dense process rewards for social dialogue.

  • Social intelligence benchmarks: Social dialogue benchmarks cover negotiation, empathy, information verification, theory of mind, and long-term relationships across standardized interactive scenarios.SOTOPIA and AgentSense cover negotiation, empathy, and information verification; ToMBench probes theory of mind, and Lifelong-SOTOPIA extends evaluation to long-term relationships.
  • Behavior and reward methods: Imitation learning, preference optimization, intention-space reward aggregation, and outcome attribution improve social behavior without online value modeling.The cited approaches include behavior cloning, segment-level preference optimization, intention-space reward aggregation, and attribution of multi-turn outcomes to utterances.
  • Reinforcement learning: Single-turn reinforcement learning optimizes utterances independently and misses how early actions shape later social outcomes.Sotopia-RL applies GRPO at the utterance level, while longer-sequence methods extend reinforcement learning or value learning to broader horizons.
  • SocialRL's distinction: SocialRL jointly optimizes full multi-turn trajectories and dense process rewards, a combination not found in prior methods summarized by Table 1.The framework also includes multi-dimensional social feedback and stage-aware weighting, as illustrated in Figure 2.

3 SOCIALRL

SocialRL models social dialogue as a finite-horizon MDP and optimizes complete conversations with PPO. It combines delayed outcome rewards with dense, context-specific process feedback whose dimensions and weights adapt across dialogue stages.

  • Task formulation: SocialRL represents each task as a finite-horizon dialogue MDP with scenario, private goal, persona information, dialogue history, token actions, and utterance-level rewards.Tokens within a turn receive zero reward, while the completed utterance receives a bounded reward rm ∈[−Rmax, Rmax].
  • Multi-turn optimization: PPO optimizes multi-turn trajectories using a value network and GAE computed from turn-level composite rewards.All tokens in an utterance share its reward and advantage, treating the utterance as one optimization unit.
  • Training algorithm: PPO is selected over GRPO because state-dependent, delayed social rewards make turn-index-based group baselines vulnerable to variance in longer dialogues.The same turn index can represent negotiation, repair, compromise, or failure, so states at that index may have different values.
  • Composite rewards: SocialRL combines process rewards on intermediate turns with final outcome reward, using α = 0.3 for process feedback and β = 1 at the final turn.Intermediate turns therefore provide gradients before final success is known or when the dialogue ultimately fails.
  • Process reward dimensions: Each utterance receives scores for six dimensions: goal advancement, strategic positioning, relational attunement, persona consistency, contextual coherence, and turn quality.These dimensions separate goal progress, relationship maintenance, and general conversational validity.
  • Rubric-based scoring: A reward model generates 2–4 binary criteria per dimension from the scenario, personas, goal, history, and utterance, then uses criterion pass rates as dimension scores.Binary criteria make reward feedback more stable and inspectable than abstract scalar judgments.
  • Stage-aware weighting: Stage-aware weights emphasize coherence, persona, and rapport early; goal advancement and strategy mid-dialogue; and closure with relationship preservation late.The current stage is inferred from context rather than fixed turn-ratio thresholds, and the resulting weights are sanitized before process reward computation.

4 EXPERIMENTS

SocialRL is evaluated across complementary interactive benchmarks, backbones, opponents, and ablations using Goal Achievement and Relationship Change. It improves performance over corresponding Base models, with gains extending to multi-party and out-of-distribution scenarios.

  • Experimental setup: Experiments evaluate SocialRL across four benchmarks, four trained backbones, and four opponent models using Goal Achievement and Relationship Change.The benchmark suite includes SOTOPIA-π, SOTOPIA-All, SOTOPIA-Hard, and AgentSense.
  • Main results: 52.3% / 0.358 versus 42.3% / 0.311 shows SocialRL outperforming Qwen2.5-7B Base across opponents on SOTOPIA-π.The strongest trained-model average is LLaMA3.1-8B at 59.6% / 0.341, while GPT-5.5 reaches 69.0% / 0.388.
  • Main results: 21.8 percentage points and 0.227 are the SocialRL gains for Gemma-3-4B over Base in Goal Achievement and displayed Relationship Change.Gemma-3-4B rises from 30.5% / −0.093 to 52.3% / 0.134.
  • Main results: 80.5% / 0.444 is SocialRL’s Qwen3-8B average on AgentSense, exceeding the corresponding Base model by 7.2 percentage points and 0.293.AgentSense tests multi-party interaction and latent social reasoning in script-derived scenarios.
  • Aggregate results: 9.2 percentage points is the average Goal Achievement improvement across four benchmarks after converting [0, 10] benchmark gains to percentage-point units.The benchmark-specific gains are 10.725 on SOTOPIA-π, 9.625 on AgentSense, 0.9275 on SOTOPIA-All, and 0.7125 on SOTOPIA-Hard before scaling the latter two.
  • Ablation experiments: 9.9 percentage points and 0.080 separate multi-turn PPO from multi-turn GRPO in Goal Achievement and Relationship Change under the same reward design.PPO obtains 63.3% / 0.516 versus GRPO’s 53.4% / 0.436; process rewards have the largest ablation effect, while dynamic weights outperform fixed schedules.

5 CONCLUSION

SocialRL combines multi-turn PPO with dense process rewards to optimize social-dialogue trajectories. Its analysis compares PPO and GRPO, showing that normalization introduces a finite-group correction while adjusted discrepancies diminish with group size.

  • 5 CONCLUSION: SocialRL optimizes complete dialogue trajectories with PPO and a value network, using dense turn-level feedback and stage-aware weights.The framework targets both goal pursuit and relationship management across dialogue phases.
  • A.3 PPO WITH GAE: PPO’s ideal-value, λ = 1 estimator matches the Monte Carlo policy-gradient target, but this does not establish unbiasedness for clipped PPO, approximate critics, or λ < 1.The caveat limits the unbiasedness claim to the ideal-value, λ = 1 case.
  • A.4 GRPO AND ITS BIAS: GRPO’s reward standardization produces a global positive rescaling rather than changing the gradient direction.The rescaling can be absorbed into the learning rate.
  • A.4 GRPO AND ITS BIAS: O(1/G) finite-group discrepancies remain after scale adjustment and vanish as group size grows.The analysis identifies variance as the primary long-horizon difference after this correction.

A.5 VARIANCE COMPARISON

The comparison analyzes PPO and GRPO through gradient variance and signal-to-noise under state-dependent baselines, dialogue horizons, and critic error. It finds that GRPO’s baseline mismatch can accumulate with horizon, while practical performance depends on mismatch, critic error, and reward normalization.

  • Variance comparison: PPO and scale-adjusted GRPO are compared using gradient bias, variance, and signal-to-noise under explicit variance-decomposition assumptions.The analysis uses two approximations only for the variance decomposition.
  • Baseline comparison: PPO and unnormalized GRPO have the same per-turn mean under Assumption B, while GRPO adds variance from state-value baseline mismatch.The mismatch term measures the distance between GRPO’s state-independent group baseline and the state-dependent value baseline.
  • Baseline comparison: GRPO’s additional variance is nonnegative and vanishes exactly when its group baseline matches the state value on states with nonzero score norm.The equality condition is stated as V π(s_m−1) = µ_0D_m almost surely for every turn m.
  • Signal-to-noise comparison: GRPO has a lower gradient signal-to-noise ratio whenever its group-level baseline fails to match the state value.The ratio measures gradient signal power relative to total gradient noise, so larger values indicate more reliable update directions.
  • Dialogue horizon: For M-turn dialogues, PPO variance grows with M, and GRPO accumulates an additional gap Δ_M = Ω(M) when average baseline mismatch remains nonzero.The relative gap may grow, remain constant, or shrink according to the ratio of average mismatch to average intrinsic variance.
  • Imperfect value network: With an imperfect value network, critic error adds PPO variance, while approximate GAE with λ < 1 may introduce additional bias that is not analyzed.The practical comparison also depends on critic error and σ_0.

C ADDITIONAL IMPLEMENTATION DETAILS

The implementation uses repeated evaluations, human audits, specified PPO and reward-model settings, matched baseline reimplementations, and training-curve procedures across multiple benchmarks.

  • Evaluation: Each policy–opponent pair is evaluated five times, with means and standard deviations reported for Goal Achievement and Relationship Change.Human alignment is separately assessed by three independent annotators across PRM and ORM judgments.
  • Training setup: PPO uses γ = 0.95, GAE λ = 0.95, clip ϵ = 0.2, and 100 training iterations with trajectories typically spanning 10–20 turns.Each batch contains 265 scenes, with four sampled trajectories per scene.
  • Models and rewards: The process reward model is Qwen3.5-35B-A3B, the judge is Deepseek-v4-flash, and process and outcome reward weights are α = 0.3 and β = 1.Reward-weight safeguards include non-negativity, caps, fallback priors, overflow redistribution, and ℓ1 normalization.
  • Baselines: Sotopia-RL, SDPO, and ArCHer are reimplemented on Qwen2.5-7B using the same SOTOPIA-π data, while Behavior Cloning uses about 1,000 GPT-4o-generated expert trajectories.Opponent models share role settings and temperature T = 0.7.
  • Training curves: Figure 7 tracks training-set Goal success across Qwen3-8B, Qwen2.5-7B, LLaMA3.1-8B, and Gemma-3-4B using raw and smoothed curves.Evaluation records marked by test epoch are excluded.
  • Benchmark results: Results are presented in the order SOTOPIA-All, SOTOPIA-Hard, SOTOPIA-π, and AgentSense.On SOTOPIA-All, SocialRL (LLaMA3.1-8B) has the strongest trained-model averages overall: 8.00 Goal and 2.35 Relationship Change.

D.2 SOTOPIA-HARD

On SOTOPIA-Hard, SocialRL achieves the strongest trained-model results on Goal Achievement or Relationship Change depending on the backbone, while training can improve both metrics for Gemma-3-4B.

  • SOTOPIA-Hard results: SocialRL (Qwen3-8B) achieves the strongest trained-model average Goal Achievement score of 7.07 on SOTOPIA-Hard.Against GPT-5.5, it reaches 5.94, compared with 5.11 for Behavior Cloning.
  • SOTOPIA-Hard results: SocialRL (LLaMA3.1-8B) achieves the strongest trained-model average Relationship Change of 1.91 on SOTOPIA-Hard.The benchmark scores Goal Achievement on [0, 10] and Relationship Change on [−5, 5].
  • SOTOPIA-Hard comparison: 0.83 above Behavior Cloning, SocialRL (Qwen3-8B) reaches 5.94 Goal Achievement against GPT-5.5 on SOTOPIA-Hard.Its corresponding Relationship Change is 0.26, compared with 2.30 for Behavior Cloning.
  • SOTOPIA-Hard comparison: Training Gemma-3-4B raises average Goal Achievement from 5.82 to 6.67 and Relationship Change from −0.83 to 0.26.This is reported as a benchmark-specific trade-off.
  • Additional benchmarks: The complete SOTOPIA-π and AgentSense results include standard deviations, while SOTOPIA-π reports Goal Achievement (%) and Relationship Change multiplied by 100.The benchmark tables provide the full repeated-experiment statistics.

E ABLATION EXPERIMENTS

The ablation experiments isolate algorithm choice using the same Qwen2.5-7B backbone and reward design, comparing SocialRL’s multi-turn PPO with a multi-turn GRPO variant.

  • Algorithm ablation: 9.9 percentage points, multi-turn PPO improves Goal Achievement over multi-turn GRPO from 53.4% to 63.3%.Relationship Change also increases from 0.436 to 0.516.
  • Algorithm ablation: 0.080, multi-turn PPO improves Relationship Change over multi-turn GRPO from 0.436 to 0.516.The comparison uses the same Qwen2.5-7B backbone and reward design.

E.2 NECESSITY OF PROCESS REWARDS

Removing intermediate process rewards largely removes SocialRL’s gains over the base model, while leave-one-dimension-out results identify distinct roles for goal- and relationship-oriented rewards.

  • Process-reward ablation: 53.2%/0.460: NoProcess nearly matches Base at 52.5%/0.458 but remains below full SocialRL at 63.3%/0.516.These values report Goal Success Rate / Relationship Change against GPT-5.5.
  • Process-reward ablation: Removing process rewards largely eliminates SocialRL’s improvement over the base model, leaving a substantial goal-success gap to full SocialRL.The authors conclude that outcome feedback alone is insufficient for supervising intermediate social behavior.
  • Dimension ablation: 55.3%/0.486 and 56.1%/0.479: removing goal advancement or strategic positioning produces the lowest Goal Success results.Full SocialRL reaches 63.3%/0.516 on Goal Success Rate / Relationship Change.
  • Dimension ablation: 61.8%/0.401: removing relational attunement produces the lowest Relationship Change while retaining comparatively high Goal Success.The result is consistent with relational attunement’s role in relationship maintenance.

E.4 NECESSITY OF DYNAMIC WEIGHTS

The dynamic reward-weight schedule is evaluated against fixed alternatives on SOTOPIA-π, with fixed schemes achieving lower reported outcomes than the full SocialRL reference.

  • Comparison setup: The comparison uses the same Qwen2.5-7B policy and GPT-5.5 opponent for dynamic and fixed schedules.The ablation reports Goal Success Rate (%) and Relationship Change ([−1, 1]).
  • Comparison setup: Figure 10 compares Dynamic with Uniform, Manual, and another fixed-weight alternative using Goal Success Rate and Relationship Change.Dynamic values come from the SocialRL row in Table 2.
  • Fixed schedules: 58.7%/0.472: the Uniform fixed-weight scheme assigns equal weight to all six reward dimensions at every turn.Values are Goal Success Rate / Relationship Change.
  • Fixed schedules: 60.4%/0.487: the Manual fixed heuristic schedule outperforms Uniform on both reported metrics.Manual uses different fixed vectors for early, middle, and late dialogue stages.

F EXTENDED QUALITATIVE EXAMPLE

The board-game example shows SocialRL converting a concern about slow play into a mutually ratified, persistent pacing rule while preserving the positive relationship change achieved by Base.

  • Intervention ownership: SocialRL turns a slow-play concern into an explicit intervention, whereas Base remains warm but leaves the pacing intervention to Benjamin.The evaluated policy must communicate the group’s need and propose a face-saving solution.
  • From opening to commitment: SocialRL builds the rule through permission, proposal, parameterization, and ratification rather than imposing a timer without social permission.Benjamin offers the opening, Ethan proposes a timer, Benjamin specifies ninety seconds, and Ethan ratifies the duration and boundary.
  • Long-horizon persistence: The agreement persists beyond one turn: Benjamin starts the timer, makes a decisive move, and invites future enforcement.This sequence is presented as a reusable coordination mechanism rather than superficial assent.
  • Goal–relationship balance: +0.6 relationship change: both Base and SocialRL receive the same positive relationship outcome, while goal success rises from 0 to 1 under SocialRL.The comparison therefore attributes the difference to improved task agency without a relationship cost, not greater friendliness.
Loading 2609.09764v1…