PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf

Yu-Yu Yang, Ti-Rong Wu, Hung Guei, Hsing-Yu Chen, I-Chen Wu

arXiv:2609.12446v1cs.AIcs.CL

TL;DR

Interactive Werewolf evaluations often rely on final game outcomes, leaving communication behavior difficult to assess. This paper measures observers’ belief shifts after suspicion and accusation messages, finding that larger models better distinguish hidden roles but still follow accusations, particularly from trusted sources. Its findings indicate that open-weight models up to 120B parameters struggle to integrate accusation content with source trust.

  • Problem

    Interactive-game outcomes such as win/loss provide little insight into the communication behaviors that lead to success or failure.

    Source: §1 · p. 1

  • Method

    The benchmark annotates suspicion and accusation messages in LLM-played Werewolf games and measures a village-side observer’s beliefs about the accuser and target before and after each message.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3

  • Results

    Models become more suspicious of accused targets and less suspicious of accusers, while larger models better distinguish hidden roles and resist accusations from accusers they distrust.

    Source: §4.2 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    Open-weight LLMs up to 120B parameters still struggle to integrate accusation content with source trust when updating beliefs in strategic communication.

    Source: §1 · p. 2

    The evaluation covers only open-weight models from 1B to 120B parameters, so its scale-dependent patterns require re-examination on frontier closed-source systems before being treated as universal.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Social-deduction games such as Werewolf are increasingly used to evaluate LLM agents, but existing evaluations often rely on final game outcomes. We propose a belief-shift evaluation benchmark in Werewolf for analyzing communication skills through belief updating. Using LLM-played games, we annotate suspicion and accusation messages and measure how an observing village-side model's beliefs change after each message. We evaluate 40 open-weight LLM configurations on 1,224 annotated messages. Our results show that larger models better distinguish true wolves from villagers based on game history, but accusations still strongly influence their beliefs. Models become more suspicious of the accused target and less suspicious of the accuser, especially when the accuser is trusted, even if the accuser is wolf-aligned. Larger models better resist accusations from accusers they already distrust. Overall, our findings suggest that current open-weight LLMs up to 120B parameters still struggle to integrate accusation content with source trust in strategic communication. Our benchmark and code are available at https://rlg.iis.sinica.edu.tw/papers/werewolf-accusation-benchmark.

1 Introduction

Existing evaluations of interactive LLM communication often focus on game outcomes, which reveal little about the behaviors leading to success or failure. This paper measures belief changes after Werewolf accusations and finds that models still struggle to weigh accusation content against source trust.

1 Introduction

  • Game outcomes such as win/loss provide little insight into the communication behaviors that lead to success or failure.The paper frames measuring those behaviors beyond final outcomes as a central challenge.Source: §1 · p. 1
  • Belief change offers an intermediate measure of whether agents accept, reject, or discount messages while reasoning about the source’s knowledge, intent, and reliability.The authors connect this perspective to theory of mind.Source: §1 · p. 1
  • The benchmark annotates accusations in LLM-played Werewolf games and measures an observer’s belief changes about both the accuser and the accused target.This analyzes communication during interaction rather than relying on final game outcomes.Source: §1 · p. 1
  • Across 40 open-weight LLM configurations, models improve with scale at distinguishing true wolves from villagers, but accusations still shift beliefs, especially when the accuser is trusted.The findings suggest models up to 120B parameters still struggle to integrate message content with source trust.Source: §1 · pp. 1–2

2 Related Work

Prior work evaluates belief reasoning through fixed narratives and studies persuasion or strategic-game outcomes. This paper instead measures an observer’s belief changes during interactive accusation exchanges.

2.1 Belief Reasoning and Persuasion in LLMs

  • Fixed-narrative theory-of-mind benchmarks test whether models infer a character’s beliefs, while this work measures changes in the model’s own beliefs during an interactive game.The cited benchmarks include ToMi, BigToM, FANToM, and ToMBench.Source: §2.1 · p. 2
  • Prior persuasion studies examine persuasive arguments, multi-turn belief shifts, or acceptance of claims from purported experts, whereas this work studies observer-side updates after accusations in a strategic game.The observer reports belief shifts about both the accuser and the accused target.Source: §2.1 · p. 2

2.2 LLMs in Social-Deduction Games

  • Social-deduction evaluations often focus on game-level outcomes such as win/loss, while WOLF evaluates utterances as deception-classification problems.These approaches address game success or deception detection rather than belief updating.Source: §2.2 · p. 2
  • The benchmark examines how an accusation changes beliefs about both the accuser and target, extending WOLF from deception detection to belief change.Source: §2.2 · p. 2

3 Belief-Shift Measurement in Werewolf

The framework uses LLM-played Werewolf games to identify suspicion and accusation messages, then measures an observing village player’s beliefs before and after each message. It evaluates 40 open-weight configurations on 1,224 annotated messages.

3 Belief-Shift Measurement in Werewolf

  • The measurement pipeline generates a dataset from LLM-played Werewolf games, annotates accusation messages, and measures observers’ belief changes before and after each message.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Dataset Generation

  • The dataset contains 1,224 suspicion or accusation messages from 200 seven-player games, with each marked message recording the accuser, target, and their roles and teams.Source: §3.1 · p. 2

3.2 Measuring Belief Shifts

  • For each message, one or two randomly selected surviving village players serve as observers, whose beliefs about the accuser and target are elicited on a seven-level scale from −3 to +3.The endpoints indicate strongly village-side and strongly wolf-side beliefs, respectively.Source: §3.2 · pp. 2–3
  • After the message, the model receives the same game history, the accusation, and its prior belief, then reports belief shifts for both players on the same −3 to +3 scale.Source: §3.2 · p. 3, Figure 1 · p. 3
  • The evaluation covers 40 open-weight LLM configurations ranging from 1B to 120B parameters.Source: §3.2 · p. 3

4 Results

Models become more suspicious of accused targets and less suspicious of accusers, with stronger shifts after direct accusations and especially when the accuser is trusted. Larger models better distinguish hidden roles and make better use of distrust, but accusation effects remain.

4.1 Prior Belief Discrimination

  • The separation between prior beliefs for true wolves and villagers grows from 0.80 in the ≤4B group to around 1.3 for models above 10B (Spearman ρ = 0.54, p = 4 × 10−4).Figure 2 presents prior beliefs by model-size group on the wolf-positive scale.Source: §4.1 · p. 3, Figure 2 · p. 3

4.2 Belief Shifts after Accusations

  • Across model sizes, direct accusations produce stronger target belief shifts than soft suspicions, while messages generally increase suspicion of the target and decrease suspicion of the accuser.For village accusers, target shifts are 0.49 for soft suspicion and 0.98 for direct accusation; for wolf accusers, they are 0.30 and 0.74, respectively.Source: §4.2 · p. 4, Table 1 · p. 4
  • Even when accusers are wolf-aligned, accused targets still shift toward the wolf side; under direct accusations, the accuser shift changes from −0.47 in the ≤4B group to −0.08 in the ≥30B group (Spearman ρ = 0.44, p = 0.004).Source: §4.2 · p. 4

4.3 Belief Shifts by Prior Trust

  • When the accuser is trusted, the target shift is similarly large for village and wolf accusers, 0.94 versus 0.96, indicating acceptance even when the accuser is wolf-aligned.Table 2 groups shifts by prior trust, accuser team, and model size.Source: §4.3 · p. 4, Table 2 · p. 4
  • For distrusted wolf accusers, larger models show greater resistance: target shift falls from 0.30 in the ≤4B group to −0.12 in the ≥30B group, while accuser shift changes from −0.56 to 0.47.The reported correlations are Spearman ρ = −0.37, p = 0.018 for target shift and ρ = 0.63, p < 10−4 for accuser shift.Source: §4.3 · p. 4, Table 2 · p. 4
  • Distrust reduces but does not eliminate the shifts, and larger models make better use of prior distrust when an accuser is viewed as unreliable.Source: §4.3 · pp. 4–5

5 Discussion

The framework measures receiver-side belief shifts around accusations, revealing that larger models discriminate hidden roles better while still tending to accept accusations, especially from trusted accusers. The authors identify broader communication settings, speaker-side behavior, and prompting as directions for extending the work.

5 Discussion

  • Larger models better discriminate hidden roles from game history, but still shift suspicion toward the accused and away from the accuser, especially when the accuser is trusted.The trusted-source effect persists even when the accuser is wolf-aligned.Source: §5 · p. 5
  • The framework measures receiver-side belief changes around accusations rather than treating final game outcomes as the sole communication measure.It compares the observer’s beliefs about both the accuser and accused target before and after a message.Source: §5 · p. 5
  • Future work could broaden the framework to other communication skills and games, analyze models as accusation-producing speakers, test skeptical instructions, and include mixed human–LLM games.The current evaluation uses pre-collected logs to give models the same contexts.Source: §5 · p. 5

Limitations

The evaluation is bounded to open-weight models in Werewolf and measures only receiver-side belief updating around accusations. Its prompting design and controlled setting also leave relevant alternatives and other interaction environments for future comparison.

Limitations

  • The evaluation covers open-weight models from 1B to 120B parameters, not frontier closed-source systems, so the reported scale patterns should be re-examined before being treated as universal.The paper specifically flags the reported regime change around 10B.Source: Limitations · p. 5
  • Werewolf’s turn-based, wolf-identification setting makes accusations dense and measurable, but may amplify credulity relative to negotiation, debate, or cooperative tasks.Those settings differ in accusation frequency, audience, or whether trust is zero-sum.Source: Limitations · p. 5
  • The study measures receiver-side belief updating only; a complete account of communication skill would also need speaker-side analysis of accusation production.Source: Limitations · p. 5
  • Providing the model’s prior belief may anchor its update; eliciting prior and posterior independently could instead prompt reassessment of the full context.The paper leaves direct comparison of the two settings to future work.Source: Limitations · p. 5
  • The framework isolates one receiver-side skill in one controlled environment and is not intended as a comprehensive evaluation of LLM communication skill.The authors describe it as a diagnostic instrument for the source-versus-content trade-off in listener belief updating.Source: Limitations · p. 5

Ethical Considerations

The paper identifies potential misuse through credulity toward trusted speakers and reports a human-checked subset of accusation annotations. It also documents its LLM-generated corpus, model licensing and intended use, and AI-assistant contributions.

Ethical Considerations

  • Uncritical credulity toward trusted speakers could be exploited by first establishing trust with an LLM agent and then injecting misleading content into its pipeline.The authors surface the risk to support defense against it.Source: Ethical Considerations · p. 5
  • Claude Opus 4.6’s labels agreed with human consensus on 144 of 150 re-labelled utterances (96%), though residual annotation bias may still affect category statistics.The authors state that the verified agreement bounds its expected magnitude.Source: Ethical Considerations · p. 6
  • The inference-only evaluation took approximately 72 wall-clock hours, or about 288 GPU-hours, on four NVIDIA RTX Pro 6000 Blackwell GPUs.Source: Ethical Considerations · p. 6
  • The study has no human participants: dialogue is LLM self-play, beliefs come from evaluated models, and the corpus contains hostile or deceptive utterances but no real-world targets or personal information.Source: Ethical Considerations · p. 6
  • The paper discloses AI-assistant use for language polishing, code drafting, and experimental-code drafting, with author review of code and outputs and authors responsible for research decisions and claims.Source: Ethical Considerations · p. 6

A.1 Model List / A.2 Licenses and Intended Use

  • The evaluation uses 29 open-weight checkpoints across 40 reasoning-mode configurations, and the listed model families’ licenses permit academic evaluation under their respective terms.The paper states that the experiments are inference-only academic benchmarking and notes GPT-OSS usage-policy compliance.Source: §A.1 · p. 8, §A.2 · p. 8

A.3 Inference Setup

Inference uses each model’s released precision and default decoding settings, while belief responses are constrained to a shared seven-point label set. The paper also reports completion reliability and enumerates the checkpoint and reasoning-mode configurations.

A.3 Inference Setup

  • Inference uses each model’s native released precision, default temperature and top-p, a 32,768-token context limit, and a 16,384-token response limit.The authors leave decoding untuned so models are compared under default behavior.Source: §A.3 · p. 8
  • Belief elicitation constrains outputs to a shared seven-point Likert label set through a JSON schema, avoiding free-form parsing errors.Source: §A.3 · p. 8
  • Reasoning-enabled checkpoints use family-specific parsers to keep captured chain-of-thought separate from final content, with reasoning toggled through family-specific chat-template settings.Source: §A.3 · p. 8
  • Only 5 of 40 configurations fall below 99% completion; remaining failures arise from overlong reasoning chains or outputs that fail JSON-schema parsing.Calls are retried on failure, with a per-configuration completion target of at least 95%.Source: §A.3 · p. 8
  • The 29 evaluated checkpoints total 40 configurations because reasoning modes may be evaluated separately, with some model families toggled at inference and Olmo variants released as distinct checkpoints.Source: Table 3 · p. 9
  • The complete evaluation, including game collection, belief elicitation, and ablations, took approximately 288 GPU-hours.Source: §A.3 · p. 9

B Dataset Details

The dataset documents English-language, seven-player Werewolf games and the annotation and evaluation procedures used to study accusation messages. It reports a 200-game corpus with 1,224 annotated messages and no train/validation/test split.

B.1 Documentation

  • The documented setting is English dialogue in turn-based, seven-player Werewolf, with no real-world text, human-authored content, or demographic data.Games proceed through day/night cycles.Source: §B.1 · p. 9
  • Games use seven LLM agents from a seven-model pool, with roles assigned randomly; belief elicitation begins only after gameplay ends.The generating pool is distinct from the 29 evaluated checkpoints described in Appendix A.Source: §B.1 · p. 9
  • Daytime messages are labeled as uncertain suspicions or direct accusations, and a 150-message stratified subset received independent author relabeling with 96% agreement.The verification protocol is reported in the Ethics Statement.Source: §B.1 · pp. 9–10

B.2 Statistics

  • The corpus contains 200 completed games and 1,224 annotated messages; all messages form one evaluation set because no model is trained or fine-tuned.The work uses no train/validation/test partition.Source: §B.2 · p. 10

B.3 Result Reporting

The appendix tests accusation-correctness effects, reports how frontier models behave, and specifies confidence-interval reporting. Separating true from false accusations preserves the main trends.

B.3 Result Reporting

  • Tables 4 and 5 report the same belief shifts as the main result tables, with means accompanied by 95% confidence intervals as subscripts.Source: §B.3 · p. 10

B.4 Frontier Closed-Source Models

  • Frontier models’ prior-role discrimination rose from 1.31 for the ≥30B group to 2.01; distrusted accusers produced negative target shifts of −0.46 and −0.62.Frontier models continued to follow trusted accusers, but with smaller shifts.Source: §B.4 · p. 10

B.5 Belief Shifts by Accusation Correctness

  • Trusted accusers produced similarly large target shifts for true and false accusations, while larger models resisted distrusted wolf accusers in both groups; the trend was not due to mixing them.Table 8 separates results by whether the accused target was wolf-aligned or village-aligned.Source: §B.5 · p. 10

B.6 Response Distributions Over the Seven-Level Scale

Responses use the full seven-level scale across model-size groups, although zero is the most frequent belief shift.

B.6 Response Distributions Over the Seven-Level Scale

  • All seven response categories appear in every model-size group, so the reported distributions are not confined to a subset of the scale.Table 9 reports percentages for prior beliefs and belief shifts by model-size group.Source: §B.6 · p. 10, Table 9 · p. 12
  • Prior-belief responses are broadly distributed across the seven-level scale in every model-size group.Source: §B.6 · p. 10
  • Zero is the most frequent belief shift, but the extreme shifts −3 and +3 occur across all model-size groups.Source: §B.6 · p. 10
Loading 2609.12446v1…