PaperlayerView exact revision on arXiv

Source-linked AI summary

Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Game

Silin Du, Xiaowei Zhang

arXiv:2404.01602v2cs.CLcs.AIcs.HC

TL;DR

Systematic evaluation of LLM opinion leadership has been largely overlooked, despite its relevance to multi-agent and human-AI settings. The paper uses Werewolf’s Sheriff role and two metrics to assess leadership, finding that few LLMs show it and that improving rule knowledge does not necessarily improve leadership.

  • Problem

    Systematic evaluation of LLM opinion leadership is limited, although it matters for multi-agent and human-AI interaction settings.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates LLMs in a Werewolf framework with a Sheriff role, using Ratio for reliability and DC for influence, alongside simulations, human evaluations, and the WWQA dataset.

    Source: §1 · p. 2, §3.2 · p. 7, §5 · p. 10

  • Results

    Only a few LLMs show a degree of opinion leadership, and fine-tuning for rule understanding does not necessarily improve leadership.

    Source: §4.2.2 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    The Werewolf game is a suitable test bed for evaluating LLM opinion leadership.

    Source: §5 · p. 10

    The main experiments omit Sheriff elections and have the moderator select the Sheriff, though the appendix reports settings that include elections.

    Source: §4.2.1 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have exhibited memorable strategic behaviors in social deductive games. However, the significance of opinion leadership exhibited by LLM-based agents has been largely overlooked, which is crucial for practical applications in multi-agent and human-AI interaction settings. Opinion leaders are individuals who have a noticeable impact on the beliefs and behaviors of others within a social group. In this work, we employ the Werewolf game as a simulation platform to assess the opinion leadership of LLMs. The game includes the role of the Sheriff, tasked with summarizing arguments and recommending decision options, and therefore serves as a credible proxy for an opinion leader. We develop a framework integrating the Sheriff role and devise two novel metrics based on the critical characteristics of opinion leaders. The first metric measures the reliability of the opinion leader, and the second assesses the influence of the opinion leader on other players' decisions. We conduct extensive experiments to evaluate LLMs of different scales. In addition, we collect a Werewolf question-answering dataset (WWQA) to assess and enhance LLM's grasp of the game rules, and we also incorporate human participants for further analysis. The results suggest that the Werewolf game is a suitable test bed to evaluate the opinion leadership of LLMs, and few LLMs possess the capacity for opinion leadership.

1 Introduction

The introduction identifies a gap in systematic evaluation of LLM opinion leadership, despite its relevance to human-AI interaction and other settings. It proposes using Werewolf’s Sheriff role to study how LLMs’ reliability and influence affect other players.

1 Introduction

  • Opinion leadership—combining strategies to shape followers’ decisions and public opinion—has been overlooked in studies of LLMs’ social behavior.Prior work has observed cooperation, confrontation, deception, and persuasion in social deductive games, but opinion leadership is distinct from these strategies.Source: §1 · p. 1
  • Systematic evaluation of LLM opinion leadership is absent despite its relevance to interaction design, decision optimization, regulation, and AI security.Potential applications include multi-agent systems, AI assistants, customer service, and online discourse.Source: §1 · p. 2
  • The Sheriff is a credible proxy for an opinion leader because players elect the role, which summarizes statements and recommends decisions.Assessing the Sheriff’s reliability and spillover to other players’ decisions provides a way to evaluate opinion leadership.Source: §1 · p. 2
  • The paper presents an in-depth analysis across diverse LLMs, a Sheriff-based framework with two metrics, simulations and human evaluations, and the WWQA dataset.The framework integrates LLM-based agents and human players.Source: §1 · p. 2

2 Related Work

Related work situates LLM agents in social simulations and deduction games, while noting that existing Werewolf frameworks omit the Sheriff. It also connects opinion leadership to trust and credibility in social learning.

2 Related Work

  • LLM agents have been used to simulate social dynamics and play natural-language games, but existing Werewolf frameworks omit the Sheriff and may thereby underestimate opinion leaders’ influence.Unlike Avalon’s passively assigned leader, Werewolf’s Sheriff is elected through agent voting, which may better reflect collective consensus.Source: §2 · pp. 2–3
  • Opinion leadership is linked to trust: leaders must be trustworthy, and recipients’ adoption of their views depends on credibility.The discussion extends the concept to LLM agents, proposing that the most trusted agent selected by a majority naturally emerges as an opinion leader.Source: §2 · p. 3

3 Framework and Proposed Metrics

The framework places a Sheriff in a Werewolf game with LLM or human players, then evaluates opinion leadership through reliability and influence metrics. LLM-based players assess others’ reliability before acting, while human participants use a similar process.

3 Framework and Proposed Metrics

  • The Sheriff can set the speaking order, summarize discussion, and advise players on voting; the framework supports both LLM and human players.The game uses seven players: two Werewolves, three Villagers, one Seer, and one Guard.Source: §3 · pp. 3–4, §3.3 · p. 7

3.1 LLM-based Player

  • LLM-based players use historical information to infer other players’ roles and confidence levels, then classify public statements as potential truths or falsehoods before acting.Statements from players with reliability above α = 6 are treated as potential truths; prompts combine game rules and roles, context, and task instructions.Source: §3.1 · pp. 4–5

3.2 Opinion Leadership Metrics

  • Ratio compares the Sheriff’s reliability with other players’ mutual reliability, while Decision Change measures players who shift votes to align with the Sheriff.A Ratio above 1 indicates greater Sheriff trustworthiness than that of other players; higher DC indicates greater influence on voting decisions.Source: §3.2 · p. 7

3.3 Human Player

  • Human participants reason about other players’ roles before acting, with the moderator reviewing game state and history to support their decisions.The human-player process is similar to that of LLM-based players.Source: §3.3 · p. 7

4 Experiments

The experiments use WWQA to assess and improve models’ grasp of Werewolf rules, then compare opinion leadership across models, Sheriff roles, and human interaction. Only a few large-scale LLMs show some opinion leadership, and better rule understanding does not necessarily improve it.

4.1 WWQA Dataset

  • WWQA is a self-instructionally generated dataset for assessing Werewolf-rule understanding and fine-tuning models, with rule-based and situation-based QA pairs.Its binary QA dataset uses Yes/No standard responses for rapid evaluation; the data-generation process has question- and answer-generation phases.Source: §4.1 · p. 8

4.2.1 Settings

  • For comparison, GLM-3 backs non-Sheriff players, while the tested LLM serves as Sheriff, whose role is assigned by the moderator without an election.The authors omit the election phase to compare models fairly.Source: §4.2.1 · pp. 8–9

4.2.2 Results

  • Only a few large-scale LLMs show some opinion leadership: larger models generally do better on binary QA, but scale alone does not predict leadership.GLM-4 reaches 0.846 accuracy on binary QA and a Ratio of 1.167; decision influence remains relatively weak.Source: Table 1 · p. 9, §4.2.2 · p. 9
  • Fine-tuning generally improves binary-QA results, but better rule understanding does not necessarily improve opinion leadership.In-20B’s Ratio rises 7.2% and DC grows 61.8%, while B-13B’s Ratio rises 8.6% as its DC deteriorates.Source: §4.2.2 · p. 9, Table 2 · p. 10
  • Sheriff opinion leadership varies by role: Werewolf Sheriffs fare poorly, while Seer and Guard Sheriffs show stronger leadership.The authors attribute the latter roles’ advantage to gathering more information through their nighttime actions.Source: §4.2.2 · p. 9, Figure 3 · p. 10

4.3 Human Evaluation

  • In human evaluation, GLM-3 gains trust but struggles to sway people with more stringent logic and stronger reasoning skills.The study involved eight students, each playing five games with six LLM-based players; human and LLM-inferred reliability scores were positively correlated.Source: §4.3 · p. 10, Table 3 · p. 10

5 Conclusion

The study evaluates LLM opinion leadership in Werewolf, finding that only a few models show some leadership and that improving rule understanding does not readily improve it.

5 Conclusion

  • Only a few LLMs show some opinion leadership, and initial WWQA fine-tuning attempts suggest that improving it is non-trivial.Human evaluations suggest that LLMs can gain human trust but struggle to influence human decisions.Source: §5 · p. 10

Appendices

The appendices specify the seven-player Werewolf setting and prompting setup, then document the framework’s round sequence, invalid-output handling, and logging system.

A.1 Game Settings

  • The seven-player game alternates night and day until a team wins; the Sheriff sets speaking order and summarizes discussion to persuade players’ votes.Night roles act according to their abilities, while players discuss and vote during the day.Source: §A.1 · pp. 16–17

A.2 Prompt Templates

  • Prompts combine game rules, contextual information, and action instructions, with JSON outputs and available choices specified to support parsing and valid actions.Separate templates cover role-specific night actions, statements, voting, reliability reasoning, and the Sheriff election.Source: §A.2 · pp. 17, 20

B Details of Our Framework

  • Each player reasons about other players’ hidden roles before acting, and the round sequence places the Sheriff’s statement and vote last.The framework tracks night actions, public statements, voting, and player reliability; Figure 4 depicts the sequence of events.Source: B Details of Our Framework · p. 22, Figure 4 · p. 22, Table 16 · p. 23

B.1 Resolving Invalid Output

  • Invalid outputs are handled according to game phase: night actions are randomized, discussion statements become silence, votes become abstentions, and reliability scores remain unchanged.Unavailable choices and generation failures are handled similarly to format failures.Source: §B.1 · p. 23

B.2 Logging System

  • The logging system records generation errors, game progress, and player-level context, decisions, and reasoning histories.Player logs also support monitoring the human evaluation process.Source: §B.2 · p. 24

B.3 Interface During Human Evaluation

The human-evaluation interface uses color and icons to distinguish message sources and content, and participants submit responses in a specified JSON format.

B.3 Interface During Human Evaluation

  • Figure 5 shows the evaluation interface, where colors distinguish prompts, results, other players’ statements, and the human player’s input.Icons also highlight message sources.Source: §B.3 · p. 24, Figure 5 · p. 24
  • Human participants enter responses in the specified JSON format and can select an input template with the up and down keys.The input prompt is similar to the one used in Appendix A.2.Source: §B.3 · p. 24

C WWQA Dataset

The section describes the WWQA dataset’s generation process and provides example question-answer pairs to illustrate its structure.

C WWQA Dataset

  • The section presents the WWQA generation process and sample question-answer pairs to explain the dataset’s structure.Source: C WWQA Dataset · p. 25

C.1 Data Generation Process

WWQA is generated through question- and answer-generation phases, with human-reviewed binary questions used to test models’ understanding of the game rules.

C.1 Data Generation Process

  • WWQA question generation covers rule-based questions and situation-based questions requiring reasoning about a constructed game situation.Source: §C.1 · p. 25
  • Question generation begins with separate example pools and iterative LLM generation, review, and revision, later mixing human-written examples with generated questions.The initial pools contain five examples each; from iteration four, prompts use two human-written and three LLM-generated examples.Source: §C.1 · p. 25
  • The answer phase generates concise answers, then applies LLM post-processing, correction, and filtering before pairing questions and answers.Source: §C.1 · p. 25
  • A binary yes-or-no QA set receives additional human review and labeling and serves as a test set for LLMs’ grasp of the game rules.Source: §C.1 · p. 25

C.2 Examples of QA Pair Data

The appendix details the WWQA examples and simulation variants, then reports that fine-tuning improves rule understanding but has limited effect on opinion leadership.

C.2 Examples of QA Pair Data

  • WWQA examples include rule-based, situation-based, and binary question-answer pairs.Source: §C.2 · p. 25

D.1 Simulation

  • Using a shared reference model for non-Sheriff players and a tested model for the Sheriff supports fairer leadership comparisons.The moderator secretly preassigns the Sheriff, who orders statements, summarizes discussion, and advises voting.Source: §D.1 · p. 25, Simulation · p. 27

Simulation

  • Homogeneous comparisons may be problematic because the Sheriff’s interaction objects differ, although homogeneous simulations were also conducted.One variant omits the election phase; another follows the game rules with an election.Source: Simulation · p. 27
  • In Heterogeneous Variant 1, replacing the elected Sheriff with the tested model produced results for C3-6B and In-20B close to Table 1, suggesting preassignment had minimal impact.Source: Simulation · p. 28, Table 20 · p. 28
  • Heterogeneous Variant 2 can estimate Sheriff-election probability across hidden roles, but simulations often cannot finish when the tested model is not elected.Source: Simulation · p. 28

D.2 Fine-tuning

  • Fine-tuning on WWQA improves rule understanding on binary QA but has limited effect on opinion leadership.Validation loss eventually increased before stabilizing around 0.31 for models below 10B and about 0.28 for models above 10B.Source: §D.2 · p. 28, Figure 6 · p. 29

D.3 Human Evaluation

The human evaluation combines Werewolf games with interviews and a small human-baseline study. Participants reported recognizable LLM behavior, while preliminary results suggested humans could influence decisions more strongly.

D.3 Human Evaluation

  • Eight experienced college students from a top-tier Chinese university played five GLM-3 iterations each and were interviewed after the games.Participants were trained on the rules and WWQA examples; they did not take part in the Sheriff election or serve as Sheriff.Source: §D.3 · p. 29
  • All interviewees said LLM players were easy to recognize because they urged others to provide information rather than analyze the game situation.Participants also mentioned hallucinations; examples are presented in Table 21.Source: §D.3 · p. 30, Table 21 · p. 30

D.4 Human Baseline

  • The framework can also place a human player in the Sheriff role to measure human opinion leadership as a baseline.Source: §D.4 · p. 30
  • The human-baseline study used about 20 rounds with a few college students, and preliminary observations suggested humans influenced other players’ decisions more strongly than LLMs.The authors note that a comprehensive baseline would need to account for population diversity, which the small study did not cover.Source: §D.4 · p. 30, Table 22 · p. 30

D.5 Additional Metrics

Additional metrics show that Sheriff statements often change decisions, while alignment with the Sheriff is associated with stronger opinion leadership. The appendix also reports survival patterns, role-based trust, and simulation details.

D.5 Additional Metrics

  • Sheriff statements tend to change 2–3 players’ decisions, but only relatively strong opinion leaders align others’ choices with their own.DC* counts decision changes without requiring alignment; the reported DC and DC* trends before and after fine-tuning are consistent.Source: §D.5 · p. 31, Table 23 · p. 31
  • The appendix describes alternative win-rate and completion-rate measures because simulations ending after Sheriff elimination yield no meaningful win-or-loss outcome.Source: §D.5 · p. 31
  • GLM-3, GLM-4, and GPT-4 were more likely to survive to the end and lead their teams to victory than C3-6B and M-7B, though the relationship was not strict.Source: §D.5 · p. 31, Table 24 · p. 31
  • Under the homogeneous setting, GLM-3 and GLM-4 generally gained more trust from other players through the Sheriff role, especially when playing as Werewolves.Source: §D.5 · p. 31, Table 25 · p. 32

D.6 Cost

  • The authors report API-call costs and note that batching reliability scores could save cost but may increase invalid outputs and disrupt simulation progress.The reported costs were approximately ¥1000 for GLM-series calls and $25 for GPT-4 calls.Source: §D.6 · pp. 31–32

E.1 Simulation

  • To avoid Werewolf negotiation failures, the simulation makes Werewolf decisions sequentially; a game-log example also shows players hallucinating prior statements.One example records a player attributing silence to another player on the wrong day.Source: §E.1 · pp. 32–33

E.2 Human Evaluation

Human-evaluation game logs illustrate both deceptive Sheriff-election behavior and hallucinated statements that shifted suspicion among players.

E.2 Human Evaluation

  • The human-evaluation log includes role assignments, night actions, a Sheriff election, discussion, and voting.Source: §E.2 · pp. 34–36, Table 27 · p. 37
  • A Werewolf concealed its identity during the Sheriff election by pretending to belong to the Villager team, a strategy also reported in prior game studies.Source: §E.2 · p. 35
  • A player accused another of making a statement before that player had spoken, shifting suspicion from the accused player to the accuser and affecting later voting.Source: §E.2 · p. 36
Loading 2404.01602v2…