Source-linked AI summary
PB-GRPO: Learning Socially Adaptive LLM Agents from Persona-Driven Simulation with Preference-Batched GRPO
Jingquan Wang, Jun Yin, Xu Han, Yongsheng Mei, Jie Hao, Bin Guo
TL;DR
Socially adaptive agents must infer latent user preferences across multi-turn interactions, but real interaction data is scarce. The paper trains policies in persona-driven simulation and introduces PB-GRPO, which normalizes advantages within preference-consistent groups. In that simulated setting, PB-GRPO achieves the strongest social adaptation among the compared baselines.
Problem
Socially competent behavior requires inferring latent preferences and adapting across turns, while real multi-turn interaction data is scarce.
Source: §1 · p. 1Method
The paper trains a post-training policy with conversation-level feedback in persona-driven simulation and normalizes advantages across preference-consistent user groups.
Source: §1 · pp. 1–2, §3 · p. 3Results
PB-GRPO achieves the strongest social adaptation, with higher held-out-persona success rates and faster, smoother reward gains than the baselines.
Source: §4 · p. 4Takeaways & Limitations
Preference-consistent normalization provides a structured middle ground between noisy prompt-local and globally pooled reward normalization.
Source: §3 · p. 3The framework is limited to dyadic interactions with static preferences, uses 128 personas across 32 preference types, and relies on a single LLM judge without human validation.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Building LLMs that behave well socially, not merely correctly, requires Building LLMs that behave well socially, not merely correctly, requires more than producing locally helpful responses. A socially competent agent must infer users' unstated goals, respect their preferences, and adapt as the conversation unfolds. These behaviors are inherently multi-turn and social, making them hard to optimize: real interaction data is scarce, and user preferences are typically latent rather than directly observable. To address these challenges, we build on a persona-driven social simulation environment (consisting of a persona library, LLM-based user simulators, and a user-satisfaction scoring system ranging from [0, 1]), to introduce preference-batched GRPO (PB-GRPO), a post-training algorithm that learns socially adaptive policies from conversation-level feedback. Compared to vanilla GRPO, PB-GRPO computes advantages using a normalization estimated across a bucket of users with similar preferences, stabilizing training across a diverse social population. Empirical evidence shows that PB-GRPO improves models' social behavior over strong reinforcement learning baselines in our simulated environment.
1 Introduction
Social competence requires inferring latent user preferences and adapting across turns, but real interaction data is scarce. The paper trains a post-training policy from conversation-level feedback in persona-driven simulation and introduces PB-GRPO, which normalizes advantages across users with similar preferences.
1 Introduction
- Socially competent agents must infer unstated goals, respect user constraints, adapt as users react, and decide when to clarify rather than act.Preferences are often latent, and prior approaches condition on user information at inference time or adapt parameters per user.Source: §1 · p. 1
- The work targets implicit preferences through post-training with conversation-level feedback, rather than inference-time conditioning or per-user adaptation.Its simulation combines personas, LLM user simulators, and a satisfaction score for full dialogues; PB-GRPO normalizes advantages across users with similar preferences.Source: §1 · pp. 1–2
- Real multi-turn data is scarce and preferences are latent, while preference heterogeneity can destabilize training.PB-GRPO addresses this by estimating normalization across preference-consistent user buckets.Source: §1 · pp. 1–2
2 Simulation Framework for Socially Adaptive Agents
The simulation studies agents that infer hidden user preferences from multi-turn conversational cues and adapt to improve satisfaction. It combines a persona library, a role-playing user simulator, and a judge that evaluates dialogue satisfaction.
2 Simulation Framework for Socially Adaptive Agents
- The framework pairs persona-driven multi-turn user simulation with an LLM judge that scores satisfaction of hidden preferences.Its three components are a fixed persona library, an LLM user simulator, and an LLM judge.Source: §2 · p. 2
2.1 Persona Library Construction
- The library expands PersonaChat profiles into personas with traits, scenarios, latent preferences, emotional-state behaviors, and response contingencies.It contains 128 training personas across 32 preference types, with four interaction-style and scenario variants per type, plus 100 unseen evaluation personas.Source: §2.1 · p. 2
2.2 Multi-Turn Interaction with Implicit Preferences
- Each instance pairs a persona and opening utterance with a latent preference hidden from the policy, which must infer it from conversational cues.Claude Sonnet 4 simulates users for up to eight rounds; the preference remains fixed while opening utterances are regenerated each epoch.Source: §2.2 · pp. 2–3
2.3 Preference Satisfaction Evaluation
- A Claude Sonnet 4 judge scores each turn from 0 to 1 using a structured rubric grounded in the full persona, preference, and dialogue history.The final-turn score is the conversation-level measure; success requires r > 0.70 before the turn limit.Source: §2.3 · p. 3
3 Preference-batched GRPO (PB-GRPO)
PB-GRPO addresses preference heterogeneity by pooling reward normalization within preference-consistent groups. This offers a middle ground between noisy prompt-local normalization and global normalization across incompatible preferences.
3 Preference-batched GRPO (PB-GRPO)
- PB-GRPO pools normalization within preference-consistent groups, aiming for a stable scale without mixing incompatible preference types.Vanilla GRPO normalizes each prompt independently, while REINFORCE++ pools globally across the batch.Source: §3 · p. 3
- For a prompt q in bucket B_k, PB-GRPO centers rewards using prompt-local µ_q and scales them using σ_k pooled across the bucket.The scale uses |B_k|G rewards rather than G, while the bucket preserves preference consistency.Source: §3 · p. 3
- Figure 2 reports that PB-GRPO improves faster and more stably than vanilla GRPO and REINFORCE++ in training reward and held-out-persona success rate.The figure's left panel shows mean rollout return over training steps; the right shows test-time success rate.Source: Figure 2 · p. 4
4 Preliminary Findings
In the simulated environment, PB-GRPO achieves the strongest social adaptation among the compared methods. Its training dynamics and held-out-persona success-rate ordering are consistent with that comparison.
4 Preliminary Findings
- PB-GRPO achieves the strongest social adaptation, with higher held-out-persona success rates and faster, smoother reward gains than the baselines.The stated generalization comparison covers 100 evaluation personas; Table 1 reports Max Reward and Success Rate.Source: §4 · p. 4, Table 1 · p. 4
5 Discussion and Future Work
The framework is limited to dyadic interactions with static preferences, a relatively small persona library, and satisfaction scores from a single LLM judge without human validation. Future work proposes multi-agent settings and comparing judge scores with human judgments.
5 Discussion and Future Work
- The study covers dyadic interactions with static preferences, 128 personas across 32 preference types, and satisfaction assessed by a single LLM judge without human validation.The authors describe the persona library as far from the diversity of real social environments.Source: §5 · p. 4
- The authors propose multi-agent settings to study whether socially adaptive agents converge on shared norms or develop differentiated strategies.They also propose correlating judge-based satisfaction scores with human judgments to assess fidelity and external validity.Source: §5 · p. 4
A Example Persona-Conditioned Environment
The example persona combines a detailed social scenario with a hidden preference and response contingencies that guide the simulator and judge but are not visible to the policy. Yining’s cues call for attentive listening rather than advice or comparison.
A Example Persona-Conditioned Environment
- The persona’s hidden preference and response contingencies are available to the user simulator and judge but hidden from the policy.The example is presented to make the environment concrete.Source: A Example Persona-Conditioned… · p. 6
- Yining is an introverted, cautious design student who rarely states her needs directly and is strained by her father’s repeated comparisons with a successful cousin.Her scenario describes exhaustion, isolation, and disconnection from creative work.Source: A Example Persona-Conditioned… · p. 6
- Her hidden preference is attentive, nonjudgmental listening that acknowledges her effort and discouragement, not advice, evaluation, or productivity planning.The example opening says she feels worn out and measured against others.Source: A Example Persona-Conditioned… · p. 6
- Careful reflection and affirmation lead her to share more, whereas immediate advice, achievement emphasis, or comparisons make her withdraw.Source: A Example Persona-Conditioned… · p. 6
B Extended Training-Reward Dynamics
Across the extended 400-step training horizon, PB-GRPO maintains the strongest reward trend and continues improving, preserving the qualitative ordering seen in the main results. The figure distinguishes raw rollout returns from smoothed trends.
B Extended Training-Reward Dynamics
- Light traces show raw per-step mean rollout returns, while solid curves show TW-EMAs that suppress rollout noise and emphasize the training trend.Source: B Extended Training-Reward Dynamics · p. 6
- Vanilla GRPO improves more gradually, while REINFORCE++ rises initially and largely plateaus in later training.Source: B Extended Training-Reward Dynamics · p. 6
- PB-GRPO continues improving and maintains the strongest reward trend through 400 steps, preserving the qualitative ordering of the main results.Source: B Extended Training-Reward Dynamics · p. 6, Figure 3 · p. 7