Source-linked AI summary
From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL
Wenyue Hua, Zachary Huang, Tyler Payne, Safoora Yousefi, Saleema Amershi, Asli Celikyilmaz
TL;DR
Agents acting for users must negotiate with counterparts whose goals may conflict with their principal’s interests, but conventional assistant training can produce overly accommodating behavior. The paper introduces SocialRL to train social reasoning in a 4B model across six environments, finding that transfer-aware unification yields a single policy with 0.627 average utility across them, comparable to larger GPT models.
Problem
Strategic delegation requires agents to protect private information, infer counterpart incentives, and pursue outcomes aligned with their principals, capabilities not established by cooperative assistant training.
Method
SocialRL trains 4B agents across six heterogeneous environments, evaluates cross-environment transfer, and consolidates specialists through transfer-aware cascade reinforcement learning and distillation.
Results
0.627 average utility across six environments: transfer-aware cascade RL produces one 4B policy comparable to GPT-4.1, GPT-5.1, and GPT-5.2, while related environments reinforce and other configurations interfere.
Takeaways & Limitations
Social reasoning and strong multi-domain delegated behavior can be learned in relatively small models, with transfer structure useful for consolidating domain specialists.
Takeaways & Limitations
Transfer dynamics are not uniformly positive: earlier capabilities can temporarily degrade, and later training may cause performance declines that require curriculum ordering and checkpoint selection.
Abstract
from arXiv · showhide
AI agents increasingly act on their users' behalf, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These principal-driven tasks routinely place the agent across from a counterpart (another user's agent, a seller, a recruiter) whose goals may conflict with its principal's. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly, helpful frontier model may disclose its principal's private information unprompted and concede at the first sign of resistance. We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six domains: Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace. Every domain is trained in-domain under the same recipe, and every policy is evaluated on all six. We find that (1) in-domain training reaches the frontier: on held-out scenarios the 4B matches or exceeds the GPT-5 family per domain, closing 73-122% of the baseline-to-frontier gap on the negotiation games, with 78% of buyer openings anchoring below target versus 3% untrained; (2) cross-domain transfer follows game structure: structurally paired games lift each other, a broad multi-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; (3) guided by this transfer structure, two strategies, cascade RL and multi-teacher on-policy distillation (OPD), consolidate the per-domain specialists into a single unified 4B that reaches 0.627 average utility across all six environments, matching or exceeding GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613); (4) an explicit theory-of-mind scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next-action prediction predicts negotiation outcomes.
1 INTRODUCTION
Strategic delegation is difficult because agents must pursue their principal’s interests while interacting with parties whose objectives and information differ. SocialRL addresses this gap by training and consolidating strategic social reasoning in 4B agents across six heterogeneous environments.
- Motivation: Delegated interactions require more than completing tasks: agents must represent principals faithfully against counterparts with conflicting objectives and constraints.Examples include compensation requests, seller negotiations, and scheduling conflicts.
- Motivation: Frontier models often reach agreements but abandon contested positions after limited resistance, revealing weak strategic delegation despite coherent negotiation.These behaviors are linked to assistant dispositions that favor general helpfulness over persistent advocacy.
- Approach: SocialRL trains social reasoning through targeted post-training of a 4B model, using domain specialists, cross-environment transfer evaluation, and unified-policy consolidation.The six environments span allocation, multi-issue bargaining, price negotiation, and preference-based coordination.
2 RELATED WORK
Prior work spans tool-using language-model agents, delegated assistants, and strategic-interaction training. SocialRL unifies these directions by post-training one small model across heterogeneous delegated interactions that require counterpart reasoning and principal-aligned action.
- Language-model agents: ReAct interleaves language-based reasoning with environment actions, while Toolformer learns when and how to invoke external APIs.These works helped establish language-model agents that reason, plan, and act through tools and digital interfaces.
- LLMs as delegated agents: Delegated-agent research studies LLMs acting for users or organizations in economically and socially consequential settings, including retail, airline, calendar, and private-information coordination.Examples include τ-bench, ScheduleMe, and CalBench.
- Training agents for delegated interaction: Negotiation research trains language agents through self-play, language feedback, behavior cloning, reinforcement learning with verifiable economic rewards, and synthetic-data supervision.The cited methods combine different forms of interaction, imitation, and reward-based training to induce strategic behavior.
- SocialRL: SocialRL post-trains a single 4B model across heterogeneous delegated interactions that demand different strategic capabilities while sharing counterpart-incentive reasoning and principal-aligned action.The resulting 4B policies reach the performance range of much larger GPT models.
3 ENVIRONMENT AND INFRASTRUCTURE FOR SOCIALRL
SocialRL’s infrastructure models negotiation as stateful, partially observed, multi-agent interaction and separates rollout execution from training through standardized event and API interfaces.
- Environment design: The infrastructure supports stateful, partially observed episodes with multiple agents acting on different schedules, many model calls, delayed rewards, and heterogeneous counterparts.Experiments can combine local trainable policies, remote frontier-model opponents, scripted agents, and human participants.
- Environment design: A generic multi-agent interface supports sequential or simultaneous decisions, asynchronous communication, and arbitrary participant counts across two-party and multi-party environments.Environments remain independent of agent implementations, allowing language models, external coding agents, scripted policies, and humans to participate through the same abstraction.
- Event-based interaction: Event streams encode observations and decision notifications while preserving ordering, timing, audience-specific visibility, and the distinction between actions and their later effects.This representation captures collective outcomes that may depend on several participants’ actions, such as elimination after all votes are received.
- Rollout-training separation: Rollout generation is decoupled from training through an OpenAI-compatible proxy that records authentic model inputs and outputs, attaches terminal rewards, and exposes trajectories to interchangeable training backends.The separation keeps environment and agent-harness logic outside the trainer while supporting multi-call episodes and remote or black-box counterparts.
4 SOCIAL REASONING ENVIRONMENTS AND REWARD DESIGN
SocialRL trains across six heterogeneous negotiation and coordination environments that share the need to reason strategically about counterparts with private objectives. It uses terminal outcome-only rewards normalized to [0, 1], with difficulty-aware reference agreements for multi-issue negotiation.
- Environment suite: SocialRL covers six domains: Deal-or-No-Deal, CaSiNo, Craigslist Bargains, Job Interview, Calendar, and Marketplace.Training first produces a domain-specialized policy in each environment, then consolidates specialists into one cross-domain policy.
- Environment suite: The environments span multi-issue allocation, contract negotiation, single-issue price bargaining, and preference-based coordination, while requiring private-objective reasoning, information protection, and strategic proposal decisions.They differ in action spaces, utility structures, information asymmetries, and interaction dynamics.
- Reward design: All six environments use terminal, outcome-only rewards mapped to a common [0, 1] range for cross-environment reporting, aggregation, and checkpoint selection.Each episode receives one scalar after agreement, walk-away, or timeout, without intermediate reward shaping.
- Reward design: The difficulty-aware transformation preserves reward endpoints and outcome rankings while concentrating reward discrimination near each scenario- and role-specific reference point.This makes comparable normalized utilities reflect what was reasonably attainable in each scenario and role.
- Reward design: DnD, CaSiNo, and Job Interview use scenario-specific reference agreements to account for differences in attainable utility across private-preference configurations.Allocation-game references use Pareto-optimal and envy-free outcomes when available, while Job Interview uses an egalitarian objective with ties favoring greater total utility.
5 SOCIALRL TRAINING AND UNIFICATION
SocialRL produces strong 4B domain specialists and reveals structured, directional transfer across environments. Transfer-aware cascade RL and MOPD consolidate these capabilities into unified models, with cascade RL reaching frontier-level average utility and MOPD offering a cheaper alternative.
- In-domain specialists: 0.615 average utility is achieved by specialized 4B policies across six domains, comparable to GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613).Specialists fall within or above the GPT baseline range on DnD, CaSiNo, Craigslist, and Marketplace, while remaining close on Job Interview and Calendar.
- Cross-domain transfer: +0.317 is the gain from Craigslist training on Marketplace, while Marketplace training raises Craigslist by +0.184, showing that transfer follows interaction structure and direction.DnD and CaSiNo also transfer positively in both directions, whereas transfer is not determined solely by specialist strength.
- Cascade RL: 0.627 ± 0.004 Avg-6 is reached by the transfer-aware cascade, compared with 0.584 ± 0.008 for random ordering and 0.562 ± 0.010 for anti-transfer ordering.The curriculum places Craigslist before Marketplace to exploit the stronger transfer direction and uses the transfer matrix to order environments.
- Cascade RL: 0.742 is the final Calendar score, rising from a 0.540 specialist score, while Marketplace reaches 0.803 and Craigslist reaches 0.580.Transfer accumulates during the cascade without additional Calendar-specific updates, though earlier capabilities can temporarily degrade and later stages can cause forgetting.
- MOPD: 0.597 ± 0.015 Avg-6, 92.6% mean gap closure, and 84.5% clipped gap closure are achieved after 60 MOPD optimization steps without the saturated CaSiNo teacher.MOPD recovers 85% of the Craigslist teacher’s advantage, 69% on Job Interview, and 83% on Marketplace, while surpassing specialists on DnD (0.664 vs. 0.656) and Calendar (0.564 vs. 0.540).
6 QUALITATIVE ANALYSIS
Post-training replaces rapid capitulation with utility-sensitive concession: policies preserve high-value terms, create bargaining room, protect private boundaries, and accept agreements conditionally on their value to the principal. These changes vary by interaction structure and are accompanied by shorter turns, more informative messages, more active rejection, and clearer commitment language.
- Cross-domain pattern: Across environments, training makes concession utility-sensitive: agents trade away inexpensive dimensions while protecting reservation prices, preferred bundles, high-weight issues, and preferred time slots.Trained policies also use fewer total tokens per turn, longer and more informative messages, more active rejection, and clearer commitment language.
- Negotiation behavior: 78% of SFT+PPO buyer openings fall below target, versus 3% for the base model, while the mean opening shifts from +0.04 to −1.48.Training strengthens anchoring and keeps the policy below target through its third proposal on average.
- Deal-or-No-Deal: PPO reduces Deal-or-No-Deal zero-reward episodes from 4.7% to 2.5% and no-deals from 4.5% to 0.5% by resisting collapse after rejection.The trained policy more consistently preserves high-value items and counters or accepts favorable allocations.
- CaSiNo: 17.3% to 13.6% of CaSiNo games over-concede after training, while strong preservation of the agent’s value rises from 17.3% to 24.2%.The trained policy maintains floors on high-priority resources and concedes mainly on low-priority resources.
- Marketplace: Marketplace mean opening offers fall from 0.956 to 0.361 of reservation price, while 62% of base opening messages reveal the buyer’s boundary and this behavior essentially disappears after PPO.The policy creates bargaining room, anchors aggressively, and moves gradually toward agreement.
- Calendar: Calendar maximum-utility outcomes rise from 16% to 38% and zero-utility outcomes fall from 63% to 35%, despite scheduling fewer meetings overall at 74%.The trained policy rejects or counter-proposes when a feasible agreement poorly serves the principal.
7 CONCLUSION
The paper introduces SOCIALRL as a general platform for training and studying social reasoning in language-model agents across heterogeneous delegated interactions. Its results show that structured transfer, specialist consolidation, and explicit theory-of-mind supervision support strong multi-domain behavior in relatively small models.
- Systems platform: SOCIALRL combines an implementation-independent event-based multi-agent interface with decoupled rollout generation and optimization infrastructure.The infrastructure uses an OpenAI-compatible rollout proxy and supports local or remote agents.
- Training and transfer: Domain-specific training produces strong 4B specialists, while cross-environment evaluation reveals directional transfer between structurally related interactions and interference from other configurations.The transfer structure is used to consolidate specialists into broader social-reasoning capabilities.
- Theory of mind: Distilling INFER→ACT→ANTICIPATE traces improves negotiation performance and measured theory-of-mind abilities, with next-action prediction most closely associated with negotiation outcomes.Trajectory analysis links these gains to greater sensitivity to the principal’s utility and stronger protection of the principal’s interests.
- Implications: Together, the environment abstraction, decoupled infrastructure, and training results establish a general platform for studying social reasoning in interactive agents.The findings indicate that strategic capabilities can transfer across interaction structures, support specialist consolidation, and produce strong multi-domain behavior in relatively small models.