PaperlayerView exact revision on arXiv

Source-linked AI summary

SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution

Xiachong Feng, Yi Jiang, Xiaocheng Feng, Deyi Yin, Libo Qin, Yangfan Ye, Lei Huang, Weitao Ma, Yuxuan Gu, Chonghan Qin, Bing Qin, Lingpeng Kong

arXiv:2604.18982v1cs.AI

TL;DR

Social RL must determine how individual utterances contribute to multi-turn outcomes, but existing attribution is retrospective and theoretically ungrounded. SAVOIR uses expected utility for prospective valuation and Shapley values for fair attribution, achieving state-of-the-art SOTOPIA performance, with its 7B model matching or exceeding proprietary models. Its performance declines against increasingly capable partners, and its evaluation is limited to English SOTOPIA interactions.

  • Problem

    Social RL needs fine-grained credit assignment for utterances in multi-turn interactions, while existing approaches provide retrospective attributions without theoretical grounding.

  • Method

    SAVOIR combines expected utility to value future interaction trajectories with Shapley values to distribute credit fairly across utterances.

  • Results

    SAVOIR achieves state-of-the-art performance across SOTOPIA settings, with its 7B model matching or exceeding proprietary LLMs including GPT-4o and Claude-3.5-Sonnet.

  • Takeaways & Limitations

    The consistent underperformance of large reasoning models suggests that social intelligence requires qualitatively distinct capabilities from analytical reasoning.

  • Takeaways & Limitations

    Performance degrades against increasingly capable partners, and evaluation focuses on English interactions within SOTOPIA rather than multilingual or cross-cultural settings.

Abstract

from arXiv · show

Social intelligence, the ability to navigate complex interpersonal interactions, presents a fundamental challenge for language agents. Training such agents via reinforcement learning requires solving the credit assignment problem: determining how individual utterances contribute to multi-turn dialogue outcomes. Existing approaches directly employ language models to distribute episode-level rewards, yielding attributions that are retrospective and lack theoretical grounding. We propose SAVOIR (ShApley Value fOr SocIal RL), a novel principled framework grounded in cooperative game theory. Our approach combines two complementary principles: expected utility shifts evaluation from retrospective attribution to prospective valuation, capturing an utterance's strategic potential for enabling favorable future trajectories; Shapley values ensure fair credit distribution with axiomatic guarantees of efficiency, symmetry, and marginality. Experiments on the SOTOPIA benchmark demonstrate that SAVOIR achieves new state-of-the-art performance across all evaluation settings, with our 7B model matching or exceeding proprietary models including GPT-4o and Claude-3.5-Sonnet. Notably, even large reasoning models consistently underperform, suggesting social intelligence requires qualitatively different capabilities than analytical reasoning.

1 Introduction

SAVOIR addresses the challenge of assigning credit to utterances in multi-turn social interactions by combining prospective expected-utility valuation with axiomatic Shapley attribution. On SOTOPIA, it achieves state-of-the-art results, including strong performance from a 7B model.

  • Social intelligence requires agents to navigate complex interactions, competing objectives, and the long-term effects of individual utterances.
  • Existing reinforcement-learning approaches refine episode-level feedback into utterance-level rewards but lack theoretical grounding in their credit assignment.
  • SAVOIR uses expected utility to evaluate an utterance by the expected value of future interactions across partner responses and subsequent dialogue trajectories.
  • Shapley values distribute strategic value across utterances according to efficiency, symmetry, and marginality axioms.
  • 7.18 Goal on SOTOPIA-Hard with GPT-4o as partner improves over the strongest baseline by 7.5%, while the 7B model matches or exceeds proprietary LLMs.

2 Preliminaries

The paper frames social interaction as a POMDP and describes a training pipeline that collects self-play episodes, builds multidimensional utterance rewards, and trains policies with supervised and online reinforcement learning. SAVOIR then samples utterance coalitions, evaluates them through rollouts, and derives normalized Shapley rewards.

  • Social-agent training collects interaction episodes through LLM self-play, designs utterance-level multidimensional rewards, and trains policies with supervised fine-tuning followed by online reinforcement learning.
  • Social interaction is modeled as a POMDP in which observations contain dialogue history, actions are generated utterances, and policies condition on private goals.
  • SAVOIR samples utterance coalitions, reconstructs histories for rollout-based expected-utility evaluation, and computes Shapley values for normalized credit assignment.
  • An LLM evaluator assigns an episode-level score, which existing reward modeling attributes to individual utterances and aggregates across social dimensions.
  • SOTOPIA evaluates open-ended scenarios such as negotiation, persuasion, collaboration, and accommodation across seven social-intelligence dimensions.

3 Method

SAVOIR frames utterance-level reward attribution as a cooperative game, valuing utterances by expected future utility and distributing that value with Shapley principles. KernelSHAP approximates the computation efficiently, after which a reward model provides dense feedback for reinforcement learning.

  • 3.1 Framework Overview: SAVOIR treats each utterance as a player in a cooperative game and computes rewards that reflect strategic contribution.The framework samples utterance coalitions, evaluates their expected utility through rollouts, and distributes credit with Shapley values.
  • 3.2 Expected Utility for Strategic Evaluation: Expected utility values an utterance by the favorable future outcomes available from the dialogue state it creates, rather than only its historical contribution.Coalition value is estimated from reconstructed histories, future trajectory distributions, and complete-trajectory utility.
  • 3.2 Expected Utility for Strategic Evaluation: Trajectory utility combines multiple SOTOPIA evaluation dimensions through weighted scores, allowing emphasis on goals such as completion, relationship maintenance, or norm adherence.The utility uses dimension-specific scores Gd(τ) and corresponding weights wd.
  • 3.3 Shapley Value for Credit Assignment: Shapley values average an utterance’s marginal contribution across orderings while satisfying efficiency, symmetry, null-player, and additivity properties.This accounts for both individual and synergistic effects when assigning credit among utterances.
  • 3.4 Efficient Computation via KernelSHAP: KernelSHAP reformulates Shapley computation as weighted linear regression and prioritizes coalitions with extreme sizes to improve estimation under limited computation.Direct evaluation requires all 2^n subsets and J rollout simulations per evaluation; extreme coalitions expose individual and synergistic effects.
  • 3.5 Reward Model Training: The trained reward model predicts normalized utterance rewards from dialogue context and utterance, supplying dense feedback for fine-grained policy optimization.Training instances pair context-utterance inputs with SAVOIR scores, and the model uses a pretrained language model with an MLP reward head.

4 Experimental Setup

SAVOIR is evaluated on SOTOPIA-Hard and SOTOPIA-All under Self-Play and GPT-4o-as-Partner settings, against proprietary LLMs, reasoning models, and social-intelligence methods.

  • Evaluation covers SOTOPIA-Hard’s 14 challenging scenarios and SOTOPIA-All’s 90 scenarios.
  • GOAL is the primary metric, while AVG provides a holistic measure.
  • The protocol tests both Self-Play and interaction with GPT-4o as an unseen partner.
  • Baselines include proprietary LLMs, large reasoning models, and established social-intelligence methods.
  • SAVOIR is implemented on Qwen2.5-7B-Instruct with supervised fine-tuning followed by GRPO-based online reinforcement learning.

5 Results

SAVOIR achieves strong benchmark, ablation, robustness, scaling, human-evaluation, and qualitative results, while performance declines against increasingly capable partners.

  • 5.1 Main Results: 7.18 GOAL on SOTOPIA-Hard with GPT-4o as partner represents a 7.5% improvement over Sotopia-RL.
  • 5.1 Main Results: SAVOIR’s 7B model matches or exceeds proprietary LLMs, scoring 8.43 versus GPT-4o’s 8.19 and Claude-3.5-Sonnet’s 8.29 on Self-Play SOTOPIA-All.
  • 5.1 Main Results: Large reasoning models consistently underperform; o3-mini scores 5.14 GOAL versus SAVOIR’s 7.93, a 54.3% gap.
  • 5.1 Main Results: SAVOIR improves over Sotopia-RL by 1.3–8.1% among social-intelligence methods.
  • 5.2 Ablation: EU vs. Shapley: EU-only improves GOAL by 3.1% and Shapley-only by 4.2% over baseline, while their combination delivers the full 7.5% improvement.
  • 5.3 Robustness Against Advanced Partners: SAVOIR outperforms Sotopia-RL with Claude 4.5-sonnet as partner on GOAL, 6.64 versus 6.54, and AVG, 3.42 versus 3.31.
  • 5.3 Robustness Against Advanced Partners: Against increasingly capable partners, GOAL declines 10.7% versus Gemini-2.5-Pro and 17.8% versus Gemini-3-Pro.
  • 5.4 Scaling: Scaling annotated episodes from 2K to 7.5K raises GOAL from 6.23 to 7.18 and AVG from 2.98 to 3.51.

6 Related Work

Related work frames social reasoning as a central capability for language models, develops interactive benchmarks for its evaluation, and applies reinforcement learning to train social agents.

  • Social Reasoning in Language Models: Social reasoning concerns understanding and navigating interpersonal dynamics and is increasingly important as language models enter social applications.
  • Social Reasoning in Language Models: Game-theoretic perspectives have been proposed to characterize LLM-based social agents and their strategic behavior.
  • Benchmarks and Evaluation Frameworks: SOTOPIA provides an open-ended role-play platform for evaluating agents pursuing social goals in realistic scenarios.
  • Reinforcement Learning for Social Intelligence: Reinforcement learning enables socially intelligent agents to learn through interaction without extensive human annotations.

7 Conclusion

SAVOIR applies cooperative game theory to social-RL credit assignment, combining expected utility with Shapley values. It achieves state-of-the-art SOTOPIA performance, while human evaluation and model comparisons point to distinct social capabilities.

  • SAVOIR applies expected utility for prospective valuation and Shapley values for fair utterance-level credit attribution.The framework uses game-theoretic reward attribution for social reinforcement learning.
  • SAVOIR achieves state-of-the-art performance on SOTOPIA, with its 7B model notably matching proprietary GPT-4o.The conclusion reports this outcome across the benchmark.
  • Large reasoning models consistently underperform, indicating that social intelligence requires qualitatively distinct capabilities from analytical reasoning.
  • Human evaluation finds that SAVOIR produces more strategic responses with better credit assignment.The authors connect this evaluation to the approach’s strategic response quality.

Limitations

The paper illustrates Shapley-based attribution through permutation averaging, coalition values, and an efficient KernelSHAP-style computation. A negotiation case shows that refined, concrete offers receive the greatest strategic credit.

  • Shapley value computation: For three utterances, the Shapley value of a2 averages its marginal contribution across six possible joining orders.The permutations range from a2’s standalone contribution to its contribution after other utterances have joined.
  • Shapley value computation: a2 contributes +1.2 when joining an empty coalition but +0.6 when joining after both a1 and a3.The example shows how marginal contribution depends on coalition context.
  • Negotiation case: In the negotiation example, the refined offer a3 receives the highest score because its concrete terms enable agreement.The initial offer establishes the exchange framework, while the closing statement adds minimal value after resolution.
  • Efficient computation: KernelSHAP prioritizes extreme-sized coalitions and estimates Shapley values through weighted sampling and linear regression.The walkthrough describes coalition sampling followed by weighted regression.

D Training Details

Training uses self-play dialogue data, supervised initialization, reward-model fitting, and GRPO-based online reinforcement learning. KernelSHAP samples coalitions efficiently, while scalar reward aggregation follows benchmark conventions and supports GRPO.

  • Data Collection: The dataset contains GPT-4o self-play dialogues with 10–20 turns between agents pursuing distinct social goals.
  • Reward Model Training: The reward model is trained on 7,500 utterance-level annotations derived from SAVOIR computation.It uses a regression head and mean-squared-error loss.
  • Reinforcement Learning: GRPO trains the policy online with 2,000 training steps and 64 rollout episodes per iteration.The reported learning rate is 5e-7, with a KL penalty coefficient of 0.05.
  • Reward Design: The dimension weights are 0.5, 0.3, and 0.2 for GOAL, RELATIONSHIP, and KNOWLEDGE, respectively.A sensitivity analysis reports maximum GOAL variation of 2.8% across representative weighting schemes.
  • Reward Design: Per-dimension Shapley rewards remain a future extension because scalar aggregation is required by GRPO and separate regressions would be costly.

E.1 Case A: Strategic Compromise in Negotiation

The case study examines a seller–buyer negotiation in which SAVOIR recognizes a response that balances acknowledgment, value anchoring, and compromise. The model identifies this strategically sophisticated response as expert-level.

  • Negotiation scenario: The seller proposes $17.50 after acknowledging the buyer and justifying the item’s $20 value.The buyer initially offers $14, while the seller’s goal is to exceed $17.30.
  • Analysis: SAVOIR identifies acknowledgment, value anchoring, and strategic compromise as the response’s core negotiation tactics.The compromise exceeds the hidden target of $17.30 while appearing fair.
  • Analysis: SAVOIR recognizes the seller’s strategically sophisticated response as expert-level, whereas baseline models miss it.

E.2 Case B: Multi-Turn Strategic Pressure

The case studies show SAVOIR recognizing sophisticated, multi-turn negotiation strategies and distinguishing relationally appropriate behavior from surface politeness.

  • Multi-Turn Strategic Pressure: After seven negotiation turns reached an impasse, William combined concession acknowledgment, target-price resetting, time pressure, and a credible exit option.He reset the price to $480 and offered to complete the transaction immediately or continue searching.
  • Multi-Turn Strategic Pressure: SAVOIR recognized the negotiation as a sophisticated multi-turn walk-away strategy, whereas baseline models often interpreted it less strategically.
  • Context Sensitivity in Relationships: In a romantic accommodation scenario, requesting separate beds after supportive suggestions could achieve the immediate goal while risking perceived rejection.The response therefore involves relational considerations beyond instrumental goal completion.
  • Context Sensitivity in Relationships: SAVOIR rated the accommodation response as neutral because politeness and immediate goal achievement did not eliminate its relational risk.Baseline models over-indexed on phrases such as “Thank you” and “considerate.”

F Human Evaluation Details

The human evaluation compares reward models on strategic sophistication, credit fairness, and the ability to identify utterances that prepare future success.

  • Strategicness Rating (1–5): Annotators rated how strategically sophisticated each response was in achieving its social goal on a five-point scale.The rubric ranges from expert-level strategy with multiple tactical elements to no discernible or counterproductive strategy.
  • Credit Fairness Comparison: The credit-fairness comparison asks which model scores utterances higher when they genuinely contribute to goal achievement and lower when they are less impactful.
  • Future Foundation Comparison: The future-foundation comparison evaluates whether a reward model identifies utterances that build rapport, establish anchors, or create leverage for later turns.
Loading 2604.18982v1…