PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game

Zelai Xu, Chao Yu, Fei Fang, Yu Wang, Yi Wu

arXiv:2310.18940v4cs.AIcs.LGcs.MA

TL;DR

Pure LLM agents can exhibit intrinsic action-selection bias in complex decision-making despite correct reasoning, motivating strategic language agents with stronger decision-making. The paper combines LLM-based deduction and diverse language-action generation with an RL policy, and reports that the resulting Werewolf agents outperform existing LLM agents and achieve human-level play. The approach is evaluated in a text-based Werewolf setting and raises ethical concerns about deception and misuse.

  • Problem

    Pure LLM agents may inherit intrinsic action-selection bias from training data, causing suboptimal decisions in complex tasks such as deceptive multi-agent games.

  • Method

    The framework uses an LLM for hidden-role deduction and diverse action candidates, then trains a population-based RL policy to select among them.

  • Results

    The agents overcome intrinsic bias, outperform existing LLM-based agents, and achieve human-level play in Werewolf.

  • Takeaways & Limitations

    Combining LLM language generation with RL supports flexible language actions and stronger strategic decision-making in Werewolf.

  • Takeaways & Limitations

    The implementation focuses on a pure text-based Werewolf environment and cannot be directly applied to other real-world situations.

Abstract

from arXiv · show

Agents built with large language models (LLMs) have shown great potential across a wide range of domains. However, in complex decision-making tasks, pure LLM-based agents tend to exhibit intrinsic bias in their choice of actions, which is inherited from the model's training data and results in suboptimal performance. To develop strategic language agents, i.e., agents that generate flexible language actions and possess strong decision-making abilities, we propose a novel framework that powers LLM-based agents with reinforcement learning (RL). We consider Werewolf, a popular social deduction game, as a challenging testbed that emphasizes versatile communication and strategic gameplay. To mitigate the intrinsic bias in language actions, our agents use an LLM to perform deductive reasoning and generate a diverse set of action candidates. Then an RL policy trained to optimize the decision-making ability chooses an action from the candidates to play in the game. Extensive experiments show that our agents overcome the intrinsic bias and outperform existing LLM-based agents in the Werewolf game. We also conduct human-agent experiments and find that our agents achieve human-level performance and demonstrate strong strategic play.

1. Introduction

The paper targets intrinsic action-selection bias in pure LLM agents, even when their reasoning is correct, and proposes combining LLM reasoning with reinforcement learning for strategic language play. In Werewolf, the framework uses hidden-role deduction, diverse action candidates, and an RL policy, with experiments reporting stronger performance than existing LLM agents and human-level play.

  • Motivation: Pure LLM agents can reason correctly yet choose actions with biased distributions, producing suboptimal and exploitable behavior.In 100 Rock-Paper-Scissors games, the agent identified random play as optimal but favored Rock.
  • Motivation: Werewolf requires language-based deception, hidden-role deduction, and strategic interaction between Werewolves and Villagers.Werewolves conceal their roles and eliminate players, while Villagers share information and vote out Werewolves.
  • Framework: The proposed framework uses an LLM for hidden-role deduction and diverse action generation, then trains an RL policy to choose among candidates.Population-based RL improves robustness by training against a population of varied agents.
  • Results: The agents achieve the highest win rate against four state-of-the-art baselines and comparable win rates to average humans as teammates and opponents.The evaluation includes case studies, round-robin tournaments, human-agent experiments, and systematic ablations.

2. Related Work

Prior work develops LLM agents, external planning or policy modules, and agents for games, but the paper distinguishes its setting by combining RL with language-action generation in an unbounded Werewolf action space.

  • LLM Agents: LLM-agent research uses reasoning and in-context learning for websites, games, embodied tasks, and multi-agent environments.Chain-of-Thought and ReAct are representative approaches for eliciting reasoning, plans, and actions.
  • External Modules: Cicero combines LLMs with RL for Diplomacy but selects from a fixed finite set of game-specific actions.Werewolf instead requires agents to generate natural-language actions in an unbounded space.
  • External Modules: Tree-of-Thought searches over multiple thoughts, while LLM+P converts natural-language tasks into PDDL for classical planning.The passage notes that extensive tree search can be inefficient in complex environments and PDDL does not apply to non-cooperative language games like Werewolf.
  • Reinforcement Learning: Prior reinforcement-learning work mainly applies to games with vectorized actions, whereas this paper combines LLMs and RL to generate language actions in Werewolf.The related work contrasts standard RL settings with the language-action setting studied here.
  • Social Deduction Games: Earlier social-deduction agents often use predefined protocols or omit communication, while newer natural-language agents address communication with LLMs.The paper positions its approach within the progression from simplified protocols and non-communicative deduction to natural-language play.

3. The Werewolf Game

Werewolf is a text-based, mixed cooperative-competitive game in which hidden roles, deceptive discussion, secret actions, and voting create demanding reasoning and communication requirements. The paper identifies hidden-role deduction and intrinsic action bias as the central challenges for LLM-based agents.

  • Setup: The seven-player setup contains two Werewolves, one Seer, one Doctor, and three Villagers, with hidden roles dividing players into opposing teams.Werewolves know each other, while Villagers seek to identify and eliminate the Werewolves.
  • Text-Based Environment: The environment is pure text-based: observations and actions are expressed in natural language without tone or facial-expression signals.Observations include game history, roles, secret actions, announcements, discussions, and voting results.
  • Gameplay: Night rounds assign secret actions to Werewolves, the Seer, and the Doctor, while day rounds combine discussion, role claims, accusations, defenses, and voting.The Doctor can save the Werewolves’ target if they select the same player.
  • Winning Conditions: Werewolves win when their remaining number equals the remaining Villagers, whereas Villagers win when both Werewolves are eliminated.These asymmetric conditions define the teams’ competing objectives.
  • Challenges for LLM-Based Agents: The first major challenge is distinguishing truths from lies and deducing hidden roles from deceptive communication.Unknown opponents can use manipulative statements to mislead agents and impair judgments.
  • Challenges for LLM-Based Agents: The second major challenge is overcoming biased action distributions because opponents can exploit predictable behavior in competitive play.Reducing intrinsic bias is therefore tied to preventing exploitation and improving strategic performance.

4. Strategic Language Agents

Strategic language agents combine LLM-based hidden-role deduction and diverse action generation with an RL policy that selects among language-action candidates. Population-based training optimizes these decisions against diverse agents.

  • Framework overview: The framework has three components: hidden role deduction, diverse action generation, and population-based RL training.The LLM organizes information and infers roles; candidate actions address intrinsic bias; RL optimizes action selection.
  • Hidden Role Deduction: Hidden role deduction converts raw observations into organized records and structured deductions that support later decisions.Records distinguish facts, potential truths, and potential deceptions, while deductions include each player’s role, reliability, reasoning, and evidence.
  • Diverse Action Generation: Diverse action generation prompts the LLM for N strategically diverse candidates rather than a single action.Vanilla prompting generates candidates in one inference, whereas iterative prompting generates them sequentially and is used for more complex statement actions.
  • Population-Based RL Training: The RL policy learns a non-uniform distribution over generated action candidates instead of relying on random sampling.This smaller policy is trained by playing against a population of agents, while the policy’s action space consists of language actions generated during the game.
  • Population-Based RL Training: Population-based training uses the RL policy, past checkpoints, and pure LLM agents with predefined play styles to promote robust performance.The population includes three common styles for Werewolves and three for Villagers.

5. Experiments

Experiments evaluate whether the framework reduces intrinsic action bias and improves Werewolf performance through case studies, tournaments, human-agent games, and ablations. The agents show near-optimal action distributions, strongest cross-play performance, comparable human win rates, and benefits from all major components.

  • Case Studies: Our agents produce near-uniform Werewolf first-night kill choices, close to the optimal policy, unlike the pure LLM-based agent’s bias toward Player 0.The pure agent’s bias could be exploited by a Doctor who always saves Player 0.
  • Case Studies: Our agents save themselves with probability 0.94 as Doctors on the first night, compared with the pure agent saving others with probability 0.38.
  • Case Studies: Our agents vote more effectively than the pure agent when two players claim to be Seers, reducing non-voting and identifying the Werewolf impersonator.The pure agent chooses not to vote with probability 0.69.
  • Round-Robin Tournament: In the 5 × 5 cross-play tournament, our agents achieve the highest win rates as Villagers against every agent and the best performance as Werewolves.Rows represent Villager performance, while columns represent Werewolf performance.
  • Human Evaluation: Human-agent experiments show that our agents outperform the pure LLM-based agent and achieve win rates comparable to human players when playing with six humans.The agents were not trained with real humans.
  • Ablation Study: Ablations show that hidden-role deduction, diverse action generation, and the RL policy all contribute to performance, while the RL policy is needed to overcome residual intrinsic bias.Using an LLM to select among diverse candidates remains comparable to removing diversity, because that selection still suffers from intrinsic bias.
  • Ablation Study: Using all four deduction attributes gives the best performance, and population-based training outperforms self-play as both Villagers and Werewolves.

6. Conclusion

The paper proposes a framework combining LLM reasoning with RL to build strategic language agents for Werewolf. The agents address intrinsic bias through hidden-role deduction, diverse action candidates, and RL-based action selection, outperform existing LLM agents and achieve human-level play.

  • The framework combines an LLM with an RL policy to build strategic language agents that overcome intrinsic bias in pure LLM-based agents.
  • The agents use LLM-based hidden-information deduction, diverse action candidates, and an RL policy to choose among those candidates.
  • The agents outperform existing LLM-based agents and achieve human-level play in Werewolf.

Impact Statement

The study addresses ethical risks of deceptive AI agents and potential misuse while evaluating agents in a structured seven-player Werewolf environment. Its prompts specify role-conditioned reasoning, communication, and action outputs across night and day phases.

  • Ethical safeguards: The human-agent experiment addresses deception risks through approval, informed consent, explicit disclosure, and participants’ ability to withdraw.Participants were told that the game contains deceptive communications and potentially dishonest AI statements.
  • Scope boundary: The approach is scoped to a pure text-based Werewolf environment and cannot be directly applied to other real-world situations.The authors identify this setting as a boundary on potential misuse.
  • Game setting: Each game uses seven players with two Werewolves, one Seer, one Doctor, and three Villagers, whose roles determine private knowledge and available actions.Night actions include killing, role inspection, and saving; day play includes announcement, discussion, and voting.
  • Game setting: The game ends when Werewolves equal the remaining opposing players or when both Werewolves are eliminated.Werewolves win by reaching numerical parity, while Villagers win by eliminating both Werewolves.
  • Prompt design: The prompts require role- and phase-conditioned reasoning followed by structured JSON outputs for secret actions and public discussion.Secret-action prompts request kill, see, or save decisions, while discussion prompts separate private reasoning from the spoken statement.

C.1. Self-Attention Policy Architecture

The self-attention policy combines structured player information, natural-language observations, and embedded language-action candidates. It predicts state values and selects candidates using attention-based compatibility, with rewards emphasizing winning and role-specific outcomes.

  • Inputs: The policy represents player information, language observations, and language-action candidates as three input types.Player vectors encode IDs, roles, and deductions; observations and candidates are embedded from natural language.
  • Network: A residual self-attention network contextualizes the inputs, averages player and observation embeddings into a state embedding, and predicts its value with a critic head.The architecture uses self-attention without position embeddings.
  • Action selection: Action-candidate sampling probabilities are proportional to the dot-product attention between the state embedding and each action embedding.This lets the policy choose among generated language candidates rather than directly generating one action.
  • Rewards: The reward design combines a +100/-100 winning reward with shaping rewards for killing, seeing, saving, and voting outcomes.Role-specific rewards assign positive or negative values according to whether actions help Werewolves or the opposing team.
  • Rewards: Voting rewards distinguish voting for Werewolves, voting for non-Werewolves, abstaining, and individual voting outcomes.The listed rewards separately encode team-level voting results and the current player’s vote.

C.3. Population-Based RL Training and Hyperparameters

The training procedure uses MAPPO with population-based training so the learned policy encounters varied teammate and opponent behaviors. The implementation also evaluates prompting choices and varies action-candidate count and other hyperparameters.

  • Population-based training: MAPPO trains the policy through population-based training initialized with six pure LLM-based behavioral agents.The initial population includes distinct Werewolf and non-Werewolf styles, and RL checkpoints are added during training.
  • Population-based training: Each episode assigns four players to the learning policy and three to fixed policies sampled from the population.Fixed agents retain their sampled policies for the game, exposing learning agents to varied teammates and opponents.
  • Hyperparameters: The reported hyperparameters include Adam stepsize 1e-5, entropy coefficient 0.01, PPO clipping 0.2, and three action candidates.The configuration also specifies 10 PPO epochs, 12 attention heads, and attention size 128.
  • Prompting choices: The study compares hidden-role deduction with ReAct, ReCon, and other prompting methods while retaining the broader framework components.This comparison is intended to show that the current deduction design produces reasonable performance.

D.2. Ablation on Communication

The communication ablation removes discussion-phase communication from comparison agents and measures their performance against the proposed agent over 100 games.

  • Ablation design: The ablation removes communication by forcing four comparison agents to return an empty string during the discussion phase.The intervention targets communication while retaining the rest of the game interaction.
  • Evaluation: The communication-ablated agents are evaluated by playing 100 games against the proposed agent.The evaluation uses the same round-robin comparison setting described for the four agents.
  • Evaluation: The reported outcome is the mean win rate of the communication-ablated agents against the proposed agent.The passage states that this result is presented in Table 9 but does not provide the numeric value.

D.3. Ablation on Different Actions

The ablation separates statement, secret, and voting actions to assess how reinforcement learning contributes across action types. RL improves performance overall, with different action categories mattering more for Villagers and Werewolves.

  • Using RL for both statement and strategic actions improves overall performance.
  • RL for secret and voting actions contributes more to improvement for Villagers.
  • RL for statement actions contributes more to improvement for Werewolves.

D.4. Ablation on the Number of Action Candidates N

The study examines how the number of generated action candidates affects diversity and transfer. Diversity gains diminish as more candidates are added, and the RL policy requires retraining for new settings to achieve best performance.

  • The increased diversity of the last action is relatively large for N = 2 and 3 but smaller for N = 4 and 5.
  • The authors set N = 3 because additional candidates become harder to make distinct and may exceed the number of possible actions.
  • The RL policy can combine with unseen LLMs because it takes natural-language states and actions as input and is decoupled from the prior LLM.
  • The first two framework components generalize to changed player counts, roles, and winning conditions with slight prompt changes, whereas RL policies need retraining for new settings.

D.7. Human Evaluation

The human evaluation uses controlled text-based Werewolf games to compare agents with human participants. Participants receive consent information about deception, voluntary withdrawal, and study risks, while the framework is presented as adaptable to related games.

  • The study recruited 160 human players for human-agent evaluation under department approval.
  • One group played ten games with six copies of the agents, alternating five Villager and five Werewolf games.
  • A second group compared games with the agents, a pure LLM-based agent, and human-only play across 30 games per team.
  • Participants interacted with the AI agent through text-based communication in a game involving deceptive statements.
  • Participation was voluntary, and participants could withdraw without consequences or loss of benefits.
  • The framework is described as adaptable to other social deduction and negotiation games through task-specific prompt changes.

E.4. Limitations

The paper’s limitations include a simplified text-only setting and training without human gameplay data. The qualitative examples also illustrate strategic behaviors such as cooperation, bluffing, concealment, and sacrificing.

  • The study uses a pure text-based Werewolf setting that omits tone, facial expressions, and body language.
  • The agents are trained with LLM-generated human-like agents rather than human gameplay data, so their language style and strategy may differ from real players.
  • The examples cover cooperation, bluffing, concealment, and sacrificing as emergent strategic behaviors.
  • A Doctor agent protects a suspected Seer, allowing the Seer to survive and contribute to a final victory.
  • A Werewolf agent bluffs as the Seer and persuades players to eliminate an innocent player.
  • The examples show Werewolves balancing teammate support against concealment, including sacrificing or declining to vote.
Loading 2310.18940v4…