PaperlayerView exact revision on arXiv

Source-linked AI summary

Playing social deduction games with reinforcement fine-tuned large language models

Lingzhe Zhang, Yunpeng Zhai, Tong Jia, Kening Zheng, Chiming Duan, Minghua He, Zhaoyang Liu, Bolin Ding, Philip S. Yu, Ying Li

arXiv:2610.04261v1cs.CLcs.AIcs.MA

TL;DR

This paper asks how reinforcement fine-tuning shapes LLM agents’ social reasoning during interaction. It studies agents playing hidden-role games and finds that targeted rewards improve social reading and can improve social influence, while terminal-outcome optimization alone is unreliable. The authors also examine combining these signals in same-side multi-agent training and evaluate how people perceive the resulting behaviours.

  • Problem

    How reinforcement fine-tuning shapes LLM agents’ social reasoning when they interact with other agents remains an open question.

    Source: §1 · p. 2

  • Method

    The study trains and evaluates LLM agents in hidden-role games, using social-reading and social-influence signals, including same-side multi-agent training.

    Source: §1 · pp. 3–4, Hidden-state inference · p. 7, §2.2.2 · p. 12

  • Results

    Terminal outcome-only training does not reliably produce social-deduction ability, while targeted rewards improve social reading and can improve social influence.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The findings support decomposing social deduction into behaviorally grounded targets and recombining them through training that respects the game’s multi-agent structure.

    Source: §1 · p. 3

    Social deduction games have explicit hidden states, discrete decisions, and defined team objectives, unlike many real social environments with ambiguous identities and changing or contested objectives.

    Source: §3 · p. 27

Abstract

from arXiv · show

Reinforcement fine-tuning (RFT) is increasingly used in applications where large language models (LLMs) interact with humans and other agents. Here we use social deduction games to study how RFT changes LLMs' social behaviour. We let fine-tuned and base LLM agents play hidden-role games that require hidden-state inference, social reading and vote steering. Our results show that LLM agents do not reliably acquire social-deduction ability by directly optimizing terminal win--loss outcomes, suggesting that final game results provide a sparse and noisy signal for socially interactive learning. However, RFT is particularly effective at improving social reading, including tasks that require agents to infer hidden roles from public discussion, update beliefs over time and predict other agents' future decisions. We further show that RFT can also improve social influence, including tasks that require agents to steer votes, team approvals and collective decisions, although these gains depend more strongly on behaviourally specific rewards and structured interaction settings. Finally, we show that LLMs' ability to play social deduction games can be further improved through multi-agent social-cognitive reinforcement fine-tuning, which combines social-reading and social-influence signals during same-side multi-agent training. These learned behaviours also receive more favourable human evaluations of strategic competence, persuasiveness and social usefulness. Together, these results enrich our understanding of how RFT changes LLMs' social behaviour and provide a step toward a behavioural learning theory for machine social intelligence.

1 Introduction

This paper uses hidden-role social deduction games to examine how reinforcement fine-tuning changes LLM agents’ social behaviour. Outcome-only training does not reliably produce social-deduction ability, while behaviorally targeted rewards improve social reading and can support social influence and multi-agent training.

1 Introduction

  • The study uses role-based hidden-information games where agents discuss publicly, propose or vote on teams, observe partial outcomes, and pursue team objectives.The overview depicts full-game trajectories being collected for reinforcement fine-tuning before updated agents play again.Source: §1 · p. 2, Figure 1 · p. 3
  • The authors propose decomposing social deduction into behaviorally grounded targets and recombining them through same-side multi-agent training, with terminal win rate as an evaluation diagnostic.The training combines social-reading and social-influence signals in complete game trajectories.Source: §1 · pp. 3–4
  • Terminal win–loss optimization does not reliably produce general social-deduction ability, whereas rewards targeting specific social-cognitive functions yield clearer improvements.The paper attributes the outcome-only difficulty to sparse feedback entangled with dialogue, belief formation, and group decisions.Source: §1 · pp. 3–4
  • Fine-tuned agents improve at inferring hidden roles, updating beliefs, and predicting other agents’ decisions; targeted rewards can also improve vote steering and team approvals.Social-influence gains depend more strongly than social-reading gains on the reward definition and interaction setting.Source: §1 · p. 4

2 Results

The study examines reinforcement fine-tuning of LLM agents across several social deduction games, with controlled experiments focused on social reading and social influence.

2 Results

  • The experiments use Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen3-4B-Instruct, GLM-4-9B-Chat, and Llama-3.1-8B-Instruct.Source: §2 · p. 4
  • The game suite includes One Night Werewolf, Werewolf/Mafia, Avalon, Secret Hitler, and Spyfall.Source: §2 · p. 4
  • The main controlled experiments focus on Avalon and One Night Werewolf, examining social reading and social influence.Source: §2 · p. 4

2.1 Analysing RFT across social deduction games

Across game families, outcome-based reinforcement fine-tuning produces uneven, often unstable results rather than reliable broad improvement. Structured institutional feedback shows the clearest positive signal, motivating controlled tests of more specific social capabilities.

2.1 Analysing RFT across social deduction games

  • The study groups games by how hidden information becomes public evidence: through elimination, institutional procedures, or asymmetric clues and consistency checks.Werewolf/Mafia and One Night Werewolf represent Elimination; Avalon and Secret Hitler, Institution; Spyfall and Deception, Asymmetry.Source: §2.1 · p. 4
  • Each game trains one focal player across available roles against fixed pretrained agents, using complete episodes and a binary reward for achieving the focal side’s objective.The focal policy is rewarded with 1 for success and 0 otherwise; mean reward over training is reported.Source: §2.1 · pp. 4–5
  • Secret Hitler reaches a final mean reward of approximately 0.63 and an average reward of approximately 0.61, while Avalon ends around 0.39 after starting around 0.23.Avalon’s average reward is about 0.25, illustrating the contrast within the Institution family.Source: §2.1 · p. 5
  • Werewolf/Mafia peaks at approximately 0.88 but ends around 0.38, while One Night Werewolf averages approximately 0.52 without sustained improvement.The curves are volatile or non-monotonic rather than showing stable gains.Source: §2.1 · pp. 5–6
  • Outcome-based RFT alone does not reliably improve broad social-deduction ability; gains are strongest with structured procedural feedback, while other families show higher variance and weaker final improvements.Training curves are often high-variance and game-dependent because dialogue, hidden-role inference, and strategic responses create long-horizon credit assignment.Source: §2.1 · pp. 5–6, Figure 3 · p. 6
  • Spyfall averages approximately 0.40 and ends around 0.50; Deception peaks around 0.75 but ends around 0.50, averaging approximately 0.48.The clue-centered games show limited sustained improvement under outcome-based training.Source: §2.1 · p. 6

2.2 Social reading and influence

The paper turns from aggregate game outcomes to controlled tests of two core social capabilities. This isolates when reinforcement fine-tuning may improve a specific skill even when overall game performance does not improve uniformly.

2.2 Social reading and influence

  • The controlled experiments isolate social reading and social influence as two capabilities required by social deduction games.Source: §2.2 · p. 7
  • Social reading means inferring hidden social states from other players’ behaviour.Source: §2.2 · p. 7
  • Social influence means shaping other players’ beliefs and decisions through communication.Source: §2.2 · p. 7

2.2.1 Social Reading

The section separates social reading into hidden-state inference, belief updating as evidence accumulates, and a role-specific test of whether reading improves game success. Across these settings, RFT improves social reading, with effects that are clearest when rewards align with a concrete inference task.

Hidden-state inference.

  • RFT improves hidden-team alignment inference in Avalon and wolf-set inference in Werewolf/Mafia across completed model-family runs.The learned policy uses public accusations, voting patterns, and consistency between claims and outcomes; the controlled task provides a clearer signal than full-game outcomes.Source: Hidden-state inference · pp. 7–8, Figure 4 · p. 8

Dynamic belief updating.

  • RFT improves beliefs at later, evidence-rich game states, including Avalon mission alignment accuracy and Werewolf/Mafia wolf-set F1.The reported gains are larger later in both settings, as team proposals, votes, quest outcomes, discussion, and elimination evidence accumulate.Source: Dynamic belief updating · pp. 8–9, Figure 5 · p. 9

Role-specialized reading.

  • In fixed-role Avalon training, the Assassin is the only role with clear, sustained outcome improvement, consistent with reward focused on identifying Merlin.The Assassin’s response length also decreases: it minimizes intermediate participation and reserves the decisive choice for the final assassination.Source: Role-specialized reading · pp. 10–11, Figure 6 · p. 10

Robustness checks.

  • Across four Werewolf social-reading runs, performance improved in every run without a uniform increase in response length or convergence to the length limit.The gains varied in magnitude across seeds, but their direction was consistent.Source: Robustness checks · pp. 11–12, Figure 7 · p. 11

2.2.2 Social Influence

The section tests whether RFT can use public language to influence individual votes and collective team approvals. It finds gains in both settings, with team-approval improvements more consistent across models and vote-steering gains varying across models and runs.

Vote steering.

  • Self-selected vote steering improved for most models, but target vote hit-rate gains varied substantially by model family.DeepSeek-7B gained 28.7%, Qwen2.5-7B 11.8% across two runs, GLM-4-9B 5.8%, Qwen3-4B 2.1%, and Qwen2.5-14B 3.2%.Source: Vote steering · pp. 12–13, Figure 8 · p. 13
  • Vote steering also increased wolf-protection rates and werewolf-team win rates, most strongly for DeepSeek-7B.Response-length changes differed across models, and verbosity alone did not explain the pattern of steering success.Source: Vote steering · pp. 13–14

Team approval.

  • RFT improved team-approval outcomes across all five evaluated model families, with gains accompanied by positive shifts in other players’ votes toward the target direction.Target approval outcome increased by 21.3% for Qwen2.5-7B, 12.5% for Qwen2.5-14B, 10.6% for Qwen3-4B, 16.3% for DeepSeek-7B, and 18.1% for GLM-4-9B.Source: Team approval · pp. 14–15, Figure 9 · p. 15

Robustness checks.

  • All four Qwen2.5-7B vote-steering runs improved, though the size of the target hit-rate gain varied across runs.Each run also increased response length, but the broader model-family results do not show verbosity as sufficient for influence.Source: Robustness checks · pp. 15–16, Figure 10 · p. 16

2.3 Joint Multi-Agent Social-Cognitive Reinforcement Fine-Tuning

The section tests prompt-level social-cognitive scaffolds and full-game reinforcement fine-tuning, then evaluates whether the learned behaviours transfer across opponents. Joint social-reading and social-influence signals improve performance across game settings, though influence gains depend more on the reward and interaction structure.

2.3.1 Prompt-level multi-agent social-cognitive ablations

  • The joint SR+SI scaffold improved good-side win rate for every evaluated model, with gains up to 13 percentage points, while SI alone varied and reduced Qwen3-4B performance.The authors suggest influence planning is most useful when grounded in a stable belief state; the joint scaffold couples belief formation with public action steering.Source: §2.3.1 · pp. 17–18, Table 1 · p. 18

2.3.2 Full-game multi-agent reinforcement fine-tuning

  • Win rates rose from approximately 0.25 to nearly 1.0 in six-player Werewolf, 0.10 to 0.95–1.00 in nine-player Werewolf, and near zero to 0.65–0.75 in Avalon.In Avalon, social reading reached approximately 0.85–0.90 and social influence approached 1.0; in nine-player Werewolf, social reading was less monotonic than win rate.Source: §2.3.2 · pp. 18–20
  • Across six-player and nine-player Werewolf and Avalon, full-game RFT improved held-out win rates alongside social-reading and social-influence measures.The authors report that these intermediate social-cognitive rewards make behaviours trainable where terminal win–loss outcomes alone provide sparse, noisy feedback.Source: §2.3.2 · pp. 18, 20, Figure 11 · p. 19
  • Late-training Werewolf rollouts shifted toward stronger social reading and influence across game days, while Avalon social reading was stronger in later missions.In Werewolf, late-training agents more often produced messages followed by aligned group decisions; the nine-player shifts were more heterogeneous across days.Source: Figure 11 · p. 19, §2.3.2 · pp. 19–20

2.3.3 Cross-model generalization.

  • Across evaluated opponents, the learned policy won 72% of games against Kimi-K2.5 and 96% against Qwen3-4B, with SR-F1 of 0.523–0.643 and SI-Follow of 0.583–0.725.The authors interpret the breadth of results as transfer beyond a single fixed training opponent.Source: Table 2 · p. 21, §2.3.3 · pp. 21–22

2.4 Human Evaluation

The blinded trajectory evaluation compares complete games from base and social-cognitive RFT policies using human ratings and matched game pairs. RFT received higher ratings for social reasoning, coordination and strategic correctness, while gains in human-like gameplay were smaller.

Human evaluation

  • Evaluators rated complete, blinded trajectories on five dimensions, judging whether beliefs, speech, votes and collective decisions fit available evidence rather than simply rewarding a win.The set contained 74 matched base–RFT game pairs under the same game identifier and opponent condition.Source: §2.4 · pp. 22–23
  • Overall value rose from 1.43 to 2.52 (+1.09), social-reading quality from 1.35 to 2.25 (+0.90), social-influence and coordination from 1.54 to 2.70 (+1.16), and strategic correctness from 1.21 to 2.38 (+1.17).Scores are means on a five-point scale over 89 trajectory ratings per condition.Source: Figure 12 · p. 22, Table 3 · p. 23, §2.4 · p. 23
  • Across 74 matched pairs, RFT received higher overall-value scores in 42, tied in 25 and scored lower in 7; paired tests confirmed gains in four dimensions but not human-like gameplay.The overall-value, social-reading, social-influence and coordination, and strategic-correctness differences had P < 0.0001; human-like gameplay had P = 0.067.Source: Figure 12 · p. 22, §2.4 · p. 24
  • Human-like gameplay rose by 0.36 points, less than the gains in social influence and strategic correctness; repetitive accusations and abrupt suspicion changes could remain.The authors describe strategic social competence and linguistic naturalness as partially dissociable.Source: §2.4 · pp. 23–24

3 Discussion

The discussion argues that RFT is more effective when rewards target specific social-cognitive behaviours than when they rely on terminal outcomes alone. Social reading and influence can combine in multi-agent training, but their interpretation and deployment remain bounded by the game setting, opponent policies and unresolved questions about influence.

3 Discussion

  • Terminal wins entangle many behaviours into sparse feedback, making them stronger as integrative policy tests than as the sole learning signal.The authors describe RFT as a feedback-mediated analogue of part of social learning, not as evidence that models acquire human mental states or mechanisms.Source: §3 · pp. 24–25
  • Social-reading rewards can improve role inference by linking public behaviour to scoreable hidden states and encouraging evidence integration over time.Improved later-day and later-mission beliefs suggest gains beyond static role priors or output formatting.Source: §3 · p. 25
  • Social influence needs action-contingent feedback that links a communicative target to subsequent movement in votes, approvals or other group decisions.When rewards specify both the intended target and downstream behaviour, RFT can select messages that become more consequential for group decisions.Source: §3 · p. 25
  • Reading and influence are complementary: influence without an explicit belief state may steer a group toward a mistaken or harmful target.The proposed closed loop links observation, inference, social objectives, communication and updating from group responses.Source: §3 · p. 25
  • Same-side multi-agent training shares a policy across role-conditioned perspectives while preserving private observations, and improves held-out win rate.Those gains test whether intermediate social-cognitive rewards combine into a more effective full-game policy.Source: §3 · p. 26
  • Specialized social agents may be trained when desired social states and downstream effects are operationalized, but rewards must distinguish useful coordination from misaligned influence.The authors propose assessing accurate representation, intended communicative effects and whether those effects advance an appropriate collective goal.Source: §3 · p. 26
  • Influence is dual-use, and stronger strategic or social capabilities do not by themselves warrant greater anthropomorphism.The discussion calls for constraints such as truthfulness, consent, autonomy and uncertainty disclosure in deployed influence-oriented systems.Source: §3 · pp. 26–27
  • Evidence remains bounded by explicit game states and mostly fixed or slowly changing model opponents; transfer to human interaction is not established.The influence metrics also leave open whether movement comes from evidence, clarity, repetition, confidence or model-specific exploitation.Source: §3 · p. 27

4 Related work

Prior work has studied LLMs as strategic players and social agents, while hidden-role games make language-mediated social reasoning observable through discussion and collective decisions. This paper places RFT in that setting, where feedback is delayed and entangled compared with many other post-training tasks.

4 Related work

  • LLMs have been evaluated in game-theoretic settings, where their choices show strategic tendencies but remain sensitive to framing, prompts, model family and interaction history.Their behaviour emerges from model priors, task structure and communicative context rather than being a fixed model property.Source: §4 · p. 27
  • Richer multi-agent studies find that LLMs can maintain roles and coordinate, yet may lose belief consistency, misuse information or produce strategically weak messages.Games and simulations make competence observable through choices, messages and downstream outcomes rather than isolated text responses.Source: §4 · pp. 27–28
  • Social deduction games probe language-mediated reasoning through hidden identities, asymmetric information, public discussion and collective decisions.Competence includes whether messages and choices change downstream beliefs, votes and outcomes.Source: §4 · p. 28
  • Most prior studies evaluate models with prompts or scaffolds, motivating treatment of social competence as multiple behaviourally grounded functions rather than one property.Fluent responses, reasonable votes or persuasive arguments may fail to improve beliefs or collective decisions.Source: §4 · p. 28
  • RFT has reshaped LLM policies across tasks, but hidden-role games add delayed, stochastic rewards entangled with other agents’ beliefs and actions.This makes RFT an operational model for studying how language agents acquire socially consequential behaviour through repeated interaction.Source: §4 · pp. 28–29

5 Methods

The methods comprise an interactive game arena, social-cognitive reinforcement fine-tuning and human-in-the-loop validation.

5 Methods

  • The AgentEvolver Game Arena generates game trajectories, records hidden and public state, and converts natural-language interactions into structured training signals.It is the framework’s trajectory-generation and signal-conversion component.Source: §5 · p. 29
  • The multi-agent social-cognitive RFT procedure optimizes language-model agents for social-reading and social-influence objectives.It forms the framework’s training component.Source: §5 · p. 29
  • A human-in-the-loop validation protocol tests whether model-only behavioural effects are recognizable and useful to human players.It provides the framework’s human-facing validation component.Source: §5 · p. 29

5.1 AgentEvolver Game Arena

AgentEvolver Game Arena provides a controlled multi-agent environment for studying language-model behaviour in social deduction games. It converts game interactions into trajectories with structured actions, social outcomes and game outcomes, supporting evaluation and reward-based training.

5.1 AgentEvolver Game Arena

  • The arena simulates hidden-role games and exports trajectories containing prompts, responses, parsed actions, hidden states, intermediate social outcomes and terminal outcomes.It supports evaluation of social reasoning and reinforcement fine-tuning across games.Source: §5.1 · p. 29, §5.1.3 · p. 31

5.1.1 Social reasoning game environments

  • One Night Werewolf, Classic Werewolf and Avalon provide varied role structures, game horizons and communication protocols through a shared computational interface.The tasks distinguish social reading of latent roles or team membership from social influence on later group decisions.Source: §5.1.1 · pp. 29–30

5.1.2 Multi-agent interaction protocol

  • A turn-based prompt-chaining protocol gives each active player role information and the public transcript, while keeping hidden-phase actions private.Controlled-policy settings train a subset of players against fixed opponents; same-side runs can train all roles on one side from a rollout.Source: §5.1.2 · p. 30

5.1.3 Trajectory and reward annotation

  • Trajectory annotations distinguish task-specific metrics from game outcomes, and training rewards weight behavioural metrics such as role inference or movement toward intended collective decisions.Terminal win rate is logged as an external outcome, while online same-side training uses task-aligned social-cognitive metrics rather than terminal win rate as its primary reward.Source: §5.1.3 · p. 31

5.2 Multi-Agent Social-Cognitive Reinforcement Fine-Tuning

The method trains LLM agents with separate social-reading and social-influence rewards, then combines these signals for coordinated same-side training. It evaluates behavioural outcomes separately from the optimization reward and uses group-relative policy optimization.

5.2 Multi-Agent Social-Cognitive Reinforcement Fine-Tuning

  • Same-side training shares policy parameters across controlled players while conditioning each response on that player's role, public history and game phase.A rollout can produce multiple role-conditioned training sequences, while fixed opponent policies remain untrained.Source: §5.2 · pp. 31–32

5.2.1 Social reading objective

  • Social reading rewards recovery of hidden social state by scoring agents' structured beliefs against the simulator's hidden state.Set-valued predictions use F1, categorical role predictions use exact-match accuracy, and episode rewards average scores across annotated reading events.Source: §5.2.1 · p. 32

5.2.2 Social influence objective

  • Social influence rewards communication when later actions by other players move toward the controlled agent's intended social outcome.The task-specific metric varies by game and side, covering decisions such as votes, approvals, team proposals and mission-related suspicion.Source: §5.2.2 · p. 33

5.2.3 Joint multi-agent social-cognitive reward

  • The joint same-side objective combines reading, influence and auxiliary game-specific outcomes, while terminal wins remain a diagnostic measure rather than the default online reward.The decomposition provides intermediate feedback and supports analysis of whether outcomes align with reading, persuasion, valid actions or formatting shortcuts.Source: §5.2.3 · pp. 33–34

5.2.4 GRPO optimization

  • GRPO compares rewards across sampled trajectories and updates the policy with a clipped objective and KL penalty to the reference model.Group-relative normalization provides a variance-reduction mechanism without a separate critic in this sparse, noisy, trajectory-dependent setting.Source: §5.2.4 · pp. 34–35

5.2.5 Training regimes

  • Training regimes target hidden-state inference, downstream behavioural change, or both social-cognitive components plus auxiliary events; evaluation uses held-out games and separate behavioural metrics.Same-side runs optimize controlled roles together, and evaluation reports outcomes including win rates, reading accuracy, valid-action rates and influence-following rates.Source: §5.2.5 · p. 35

5.3 Human evaluation

The human evaluation compared matched base and RFT Werewolf trajectories, with policy identity hidden from evaluators. Ratings covered strategic and social qualities, and analyses used trajectory-level comparisons and paired tests.

5.3 Human evaluation

  • Blinded evaluation compared complete nine-player Werewolf trajectories, with the candidate policy controlling all good-side players and fixed opponent models controlling the wolf side.The base and RFT trajectories were paired under the same game identifier and opponent condition.Source: §5.3 · p. 35

5.3.1 Trajectory selection

  • Matched trajectories were retained only if they had complete call histories and public event records, no fallback public speech and valid structured outputs.The same filtering procedure applied to both policy conditions.Source: §5.3.1 · p. 36

5.3.2 Blinded evaluation procedure

  • Evaluators reviewed anonymized trajectories and rated five dimensions on a five-point scale, including reading quality, influence and coordination, human-like gameplay and strategic correctness.They were told not to give a high score solely because the good side won.Source: §5.3.2 · p. 36

5.3.3 Statistical analysis

  • The analysis used bootstrap uncertainty intervals and matched higher/equal/lower comparisons with paired sign-flip permutation tests; a robustness check excluded overlapping evaluator-model families.The same rubric and presentation format were used across conditions.Source: §5.3.3 · p. 36, §5.3.2 · p. 36
Loading 2610.04261v1…