Source-linked AI summary
Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, Yang Liu
TL;DR
Communication games require agents to reason and communicate under incomplete information, but LLMs face context, reasoning, and fine-tuning constraints. This paper proposes a frozen-LLM framework using retrieval and reflection on communications and experiences, and Werewolf experiments show experience-based learning without parameter tuning alongside emergent strategic behaviors.
Problem
Communication games remain challenging for AI agents because they require natural-language interaction, historical context, complex reasoning, and learning without impractical fine-tuning.
Method
The framework keeps LLM parameters frozen and uses retrieved historical information, reflection, and suggestions extracted from past experiences.
Results
Werewolf experiments demonstrate learning from experience without tuning LLM parameters, with trust, confrontation, camouflage, and leadership emerging during gameplay.
Takeaways & Limitations
The results suggest that engaging LLMs in communication games is a fruitful direction for further research.
Takeaways & Limitations
The study does not use experience pools derived from human players and identifies hallucinations and noisy historical experience as limitations for future work.
Abstract
from arXiv · showhide
Communication games, which we refer to as incomplete information games that heavily depend on natural language communication, hold significant research value in fields such as economics, social science, and artificial intelligence. In this work, we explore the problem of how to engage large language models (LLMs) in communication games, and in response, propose a tuning-free framework. Our approach keeps LLMs frozen, and relies on the retrieval and reflection on past communications and experiences for improvement. An empirical study on the representative and widely-studied communication game, ``Werewolf'', demonstrates that our framework can effectively play Werewolf game without tuning the parameters of the LLMs. More importantly, strategic behaviors begin to emerge in our experiments, suggesting that it will be a fruitful journey to engage LLMs in communication games and associated domains.
1 Introduction
The paper studies how frozen LLMs can participate in natural-language communication games, using Werewolf as a representative testbed. It proposes retrieval, reflection, and experience-based suggestions, and reports learning without parameter tuning alongside emergent strategic behaviors.
- Communication games matter because agents must use natural-language communication under incomplete information, creating challenges beyond conventional game playing.Werewolf and Poker are presented as proxies for problems studied in economics, social science, and artificial intelligence.
- LLMs face limited context length, complex reasoning demands, and impractical fine-tuning when playing communication games.Historical information can be too large for prompts, while parameter tuning is described as time-consuming and data-intensive.
- The proposed framework retrieves and reflects on necessary history, then extracts suggestions from past experiences without tuning LLM parameters.Reflection supports reasoning, while experience retrieval aims to prevent repeated mistakes across matches.
- The framework uses frozen LLMs and does not require human-annotated data for playing communication games.
- Experiments on Werewolf show that the framework can learn from experience without tuning model parameters.
- Strategic behaviors including trust, confrontation, camouflage, and leadership begin to emerge during gameplay.
2 Background: Werewolf
Werewolf is an incomplete-information communication game in which players infer others’ roles through language and reasoning. Its combination of hidden roles, strategic objectives, and theory-of-mind demands makes it a testbed for communication-game research.
- The studied game has seven players and five roles divided between werewolves and villagers with special roles.The roles are two werewolves, two villagers, a witch, a guard, and a seer.
- Werewolves seek to eliminate villagers, while villagers and special roles cooperate to eliminate all werewolves.
- Gameplay alternates between night actions and daytime open discussion followed by voting.Werewolves vote at night; all living players discuss and vote during the day, while special roles use role-specific abilities.
- Players initially know only their own roles and must infer others’ roles through natural-language communication and reasoning.
- Success therefore requires language abilities plus understanding others’ intentions and theory of mind.
3 Playing Werewolf with LLMs
The framework represents each game participant with an individually prompted LLM agent. Its prompts combine game information, selected communication history, reflections, experience-based suggestions, and chain-of-thought reasoning.
- 3.1 Notations: A full day is one day-night cycle, while a round spans gameplay until one side wins or the maximum number of days is reached.
- 3.1 Notations: Agents are indexed by player, round, and day, and their responses, observations, and reflections are treated as messages.Responses are spoken words, observations are heard words, and reflections summarize the current situation from communication history.
- 3.2 Overall Framework: Each role is implemented as an individual LLM-based agent through prompting.
- 3.2 Overall Framework: Response prompts include rules and role objectives, recent and informative messages, reflection, suggestions from past experiences, and a chain-of-thought prompt.
- 3.2 Overall Framework: The framework selects communication history because limited context length makes supplying the entire history unrealistic.
1 Game rules and role descriptions
The game prompt specifies role abilities, objectives, and example conversational states. These examples illustrate how role-specific actions and claims appear in the generated gameplay context.
- Game rules and role descriptions: The prompt assigns players roles and describes role-specific objectives and abilities, including werewolf voting and guard protection.
- Game rules and role descriptions: Example dialogue includes a seer asking about information, a werewolf accusing another player, and a guard mentioning special abilities.
- Game rules and role descriptions: Another example has the seer claiming verification and the guard reporting whom they protected.
- Game rules and role descriptions: The witch example records saving a voted player with an antidote while leaving the poison unused.
3 Suggestion extracted from experiences
The framework learns from accumulated Werewolf experiences without tuning the frozen LLM, retrieving relevant histories and extracting suggestions to guide new-game reasoning. It also compresses communication history through freshness, informativeness, and completeness mechanisms to fit context limits.
- Experience-Based Learning: Unlike prior experience-based approaches limited to one round or two-player games, this framework uses cross-round experiences and supports multi-player scenarios.The comparison is stated for Shinn et al. (2023) and Fu et al. (2023).
- Historical Information Collecting: Historical information is collected through freshness, informativeness, and completeness because communication history matters but cannot fit entirely in the LLM context.Completeness uses question answering and reflection to extract information from the broader history.
- Prompt Construction: The prompt combines game and role information with recent messages, heuristically selected informative messages, reflections, and suggestions from past experiences.These components organize the information used for response generation.
- Experience Pool: The experience pool stores response, reflection, and score tuples collected from all agents across all rounds.Winning and losing agents receive scores based on the game outcome and duration.
- Suggestion Extraction: At the start of a new round, the agent retrieves experiences relevant to its reflection and extracts a suggestion for reasoning.This non-parametric mechanism uses past experiences without tuning model parameters.
- Suggestion Extraction: The extractor treats the lowest-scoring retrieved response as potentially bad and responses near the median score as potentially good.The LLM identifies differences between these groups to form the suggestion, while avoiding reliance on the entire retrieved subset.
4 Experiments
Experiments evaluate experience-based learning and component necessity in Werewolf using frozen LLMs. Results show benefits from moderate experience pools, instability with larger pools, and the importance of core pipeline components.
- 4.3 Analysis of Using Experience: 50 rounds were used to evaluate experience-based learning, with villager-side roles receiving experience pools while werewolves served as a reference.Experience pools were constructed from 10, 20, 30, or 40 game rounds.
- 4.3 Analysis of Using Experience: Experience learning generally increased villager-side winning rates, with 10- or 20-round pools also improving game duration.Thirty-round pools lengthened games without conspicuously changing villager-side winning rate, while 40-round pools slightly improved winning rate but shortened duration.
- 4.3 Analysis of Using Experience: The framework learned from experience without tuning LLM parameters, but effectiveness became unstable as the experience volume grew.The authors associate this instability partly with simple prompts, heuristic scores, and sparse, indirect supervision signals.
- 4.3 Analysis of Using Experience: Werewolf capabilities also changed with accumulated experience, undermining the assumption that the werewolf side remained constant as a baseline.This interaction may explain trends in villager-side winning rate and game duration.
- 4.4.1 Qualitative Analysis: Game rules, recent messages, informative messages, and selected or asked questions were treated as necessary components of the prompt pipeline.Removing informative messages could cause agents to treat dead players as alive or forget uncovered roles.
- 4.4.1 Qualitative Analysis: Removing chain-of-thought reasoning weakened decision-making, while situational reflection was identified as necessary for avoiding ambivalent reasoning.Reflection preserves historical experience and helps agents clarify current situations; extracted suggestions are analyzed as part of the experience pipeline.
- 4.4.2 Quantitative Analysis: Figure 4 reports that the complete method generated more reasonable and realistic responses than every tested variant.The authors interpret this result as evidence that each method component is necessary.
5 Emergent Strategic Behaviors
The study identifies trust, confrontation, camouflage, and leadership as strategic behaviors that emerge during Werewolf gameplay. These behaviors include evidence-based trust, resistance to opponents, identity concealment, deception, and attempts to influence others.
- LLMs exhibit four non-preprogrammed strategic behaviors: trust, confrontation, camouflage, and leadership.
- 5.1 Trust: Trust increases throughout the game, and LLMs dissolve unreasonable trust relationships based on their own analysis.This pattern appears both without experience and with an experience pool.
- 5.1 Trust: 20-round historical experience makes LLMs more inclined to establish trust relationships, especially bidirectional ones.The paper links timely necessary trust relationships to promoting victories and potentially improving winning rates.
- 5.2 Confrontation: Confrontation includes attacks, accusations, defensive actions, and disagreement that obstructs an opponent’s objective.In one example, other players reject a werewolf’s attempt to direct them toward eliminating an innocent player.
- 5.3 Camouflage: Camouflage conceals identity or misleads others; both werewolves and special roles may claim to be villagers for safety.LLMs may also fabricate events to communicate information while preserving a role’s secrecy.
- 5.4 Leadership: Leadership involves influencing other players, such as directing discussion or votes toward a werewolf’s objective.Calling for the seer to reveal information can encourage solidarity with a camouflaged werewolf.
6 Related Work
Prior game-playing systems achieved strong results in nonlinguistic or specialized settings, while communication-game agents remain constrained by language capacity, human-data requirements, or game-specific design. This work explores LLMs as a more flexible alternative for communicative gameplay.
- Game Playing: Self-play has addressed complete-information games, and superhuman systems have been developed for some incomplete-information games such as heads-up poker.
- Game Playing: These methods lack the language-processing ability required by communication games such as Werewolf and Diplomacy.
- Game Playing: Werewolf agents commonly rely on rule-based systems or talking templates, constraining expressive language within the game.
- Game Playing: Diplomacy systems achieve promising results but require substantial human data and are tailored specifically to that game.
- Learning with LLMs: Fine-tuning and parameter-efficient tuning are difficult because of computational and data demands, while inaccessible checkpoints can make parameter-based learning infeasible.
7 Conclusion and Future Work
The paper presents a framework for communicative games using Werewolf as a representative case and reports emergent strategic behaviors in LLM gameplay. It also identifies several directions for improving evaluation, robustness, and generality.
- Conclusion: The framework uses Werewolf to explore the feasibility of constructing communication-game agents with LLMs.
- Conclusion: The experiments observe nonpreprogrammed trust, confrontation, camouflage, and leadership behaviors in LLMs.
- Future Work: Future work includes advanced game techniques, invariant baselines for multi-LLM evaluation, hallucination reduction, real-world application, and broader game coverage.
Limitations
The method has limitations related to hallucinations and the use of historical experience, including noise, cross-game generalization, and the absence of human-derived experience pools.
- Hallucinations can reduce the factuality of generated content and negatively affect reasoning.
- Historical experience could be used more robustly to mitigate noise and leverage cross-game general experiences.
- The study does not incorporate experience pools derived from human players.
Ethics Statement
The study analyzes a simulated game setting, and references to killing or eliminating are confined to the game context. The authors do not condone real-life violence or illegal activities.
- References to killing and eliminating describe actions within the simulated game only.
- The game is intended for entertainment and research on game mechanics, player behavior, and artificial intelligence.
A.1 Heuristic Rules of Informativeness
The framework scores messages using predefined informativeness rules and supplies the highest-scoring messages to the prompt. This selects informative content for the agent context.
- Each message receives a score when it contains a feature listed in the predefined scoring rules.
- The system sorts messages by score and feeds the top N messages into the prompt.
A.2 Predefined Question Set
The appendix describes role-specific questions, prompting procedures, game prompts, and the complete iterative process used to generate agent responses. It also distinguishes strategic deception from hallucination and details the game interaction setting.
- A.2 Predefined Question Set: Predefined questions recall useful information that may be lost because of limited context and guide the LLMs’ initial thinking.
- A.2 Predefined Question Set: Questions are organized into one shared class and five role-specific classes, with nine candidates available for each specific role.
- Trust behaviors: Trust is operationalized as believing other agents are teammates, expressed through reflections or responses with substantive content.
- Hallucination Problems: Speaking contrary to an actual role can be deceptive gameplay, whereas inconsistent or counterfactual information within an iteration is classified as hallucination.
- A.6 A Complete Iteration Step: A complete response iteration includes question-answering, reflection, experience extraction, and chain-of-thought reasoning.
- A.6 A Complete Iteration Step: Without experience use, experience extraction is omitted and no historical experience enters the final response.
- Prompt design: The game prompt specifies five roles, alternating daytime and dark phases, moderator instructions, and an end-of-response <EOS> marker.
- Game setting: The example game contains seven players with two werewolves, two villagers, a guard, a witch, and a seer.