PaperlayerView exact revision on arXiv

Source-linked AI summary

Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation

Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, Gao Huang

arXiv:2310.01320v3cs.AIcs.CLcs.CYcs.LGcs.MA

TL;DR

LLM-agent research often assumes processed information is honest, leaving deceptive environments as a challenge. The paper introduces Recursive Contemplation (ReCon), combining contemplation and perspective-taking, and reports that it helps LLMs navigate deception in Avalon without extra fine-tuning or data.

  • Problem

    LLM-agent research often assumes processed information is consistently honest, while deceptive environments pose challenges for LLMs.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    ReCon combines formulation and refinement contemplation with first- and second-order perspective transitions to help LLM agents identify and address deception.

    Source: §1 · p. 2

  • Results

    Avalon experiments indicate that ReCon helps LLMs detect and navigate deceptive information without additional fine-tuning or data.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The paper presents analyses of LLM limitations in safety, reasoning, speaking style, and format as possible insights for future research.

    Source: §1 · p. 2

    LLM responses can be excessively formal and detailed, diverging from human game speech; balancing human-like speech with performance remains future work.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Recent breakthroughs in large language models (LLMs) have brought remarkable success in the field of LLM-as-Agent. Nevertheless, a prevalent assumption is that the information processed by LLMs is consistently honest, neglecting the pervasive deceptive or misleading information in human society and AI-generated content. This oversight makes LLMs susceptible to malicious manipulations, potentially resulting in detrimental outcomes. This study utilizes the intricate Avalon game as a testbed to explore LLMs' potential in deceptive environments. Avalon, full of misinformation and requiring sophisticated logic, manifests as a "Game-of-Thoughts". Inspired by the efficacy of humans' recursive thinking and perspective-taking in the Avalon game, we introduce a novel framework, Recursive Contemplation (ReCon), to enhance LLMs' ability to identify and counteract deceptive information. ReCon combines formulation and refinement contemplation processes; formulation contemplation produces initial thoughts and speech, while refinement contemplation further polishes them. Additionally, we incorporate first-order and second-order perspective transitions into these processes respectively. Specifically, the first-order allows an LLM agent to infer others' mental states, and the second-order involves understanding how others perceive the agent's mental state. After integrating ReCon with different LLMs, extensive experiment results from the Avalon game indicate its efficacy in aiding LLMs to discern and maneuver around deceptive information without extra fine-tuning and data. Finally, we offer a possible explanation for the efficacy of ReCon and explore the current limitations of LLMs in terms of safety, reasoning, speaking style, and format, potentially furnishing insights for subsequent research.

1 INTRODUCTION

ReCon is presented as a way to help LLMs handle deceptive information, a gap in prior LLM-agent work that matters because misleading content can expose agents to manipulation. The paper tests the framework in Avalon and reports improved deception handling without extra fine-tuning or data.

1 INTRODUCTION

  • Prior LLM-agent work commonly assumes processed information is honest, despite deceptive content in human society and AI-generated material that can enable malicious manipulation.The authors argue that identifying and counteracting deceptive inputs is important for avoiding risks such as misalignment and economic loss.Source: §1 · pp. 1–2
  • The authors use Avalon as an initial testbed because its hidden roles, misinformation, communication, and intricate logic demand sophisticated thinking.They describe Avalon as a “Game-of-Thoughts” and report that recursive thinking and perspective-taking improve LLM performance in the game.Source: §1 · p. 2
  • ReCon combines formulation contemplation, which generates initial thoughts and speech, with refinement contemplation, which further polishes them.First-order perspective transition infers others’ mental states; second-order transition considers how others perceive the agent’s mental state.Source: §1 · p. 2
  • ReCon helps LLMs detect and navigate deceptive information in Avalon without additional fine-tuning or data.The introduction also previews a possible explanation for ReCon’s efficacy and discussion of LLM limitations in safety, reasoning, speaking style, and format.Source: §1 · pp. 2–3

2 BACKGROUND

Avalon combines hidden allegiances, quest decisions, and discussion in which players try to infer whom to trust. These features create challenges for LLM agents: they may accept malicious reasoning, reveal private information, or conceal deceptive thoughts from users.

2.1 DECEPTIONS IN THE AVALON GAME

  • Merlin subtly signals who is evil while hiding his identity, Percival seeks the true Merlin and detects Morgana’s impersonation, and loyal servants have no special information.These roles have different information and inference tasks within the game.Source: §2.1 · p. 3
  • Avalon assigns secret good or evil roles, and evil players can disguise themselves, sway team votes, sabotage quests, and mislead others in discussion.Good players work to infer allegiances and complete most quests; evil players seek to mislead them so most quests fail.Source: §2.1 · p. 3

2.2 CHALLENGES FOR LLMS IN DECEPTIVE ENVIRONMENTS

  • LLM agents can accept malicious reasoning: in the example, CoT misses Assassin’s deceit and wrongly reasons that evil players can aid quest success.Assassin frames replacing a good player with an evil one as a way to achieve team balance and reveal evil players, while seeking to mislead the team.Source: §2.2 · p. 3
  • LLMs may expose private information or keep deceptive internal thoughts hidden, limiting users’ insight into their intentions.Merlin reveals his identity and awareness of an evil player, which leads to assassination; Morgana hides her alliance with Assassin while feigning alignment with the good side.Source: §2.2 · p. 3, Figure 3 · p. 4

3 RECURSIVE CONTEMPLATION

ReCon combines formulation and refinement contemplation to help LLMs identify and address deception. It moves from initial thought and speech to perspective-based revision.

3.1 FORMULATION CONTEMPLATION

  • Formulation contemplation generates initial private thought and spoken content, using first-order perspective-taking to infer other players' thoughts and roles.The latest role assumption is retained as a basis for subsequent guesses.Source: §3.1 · pp. 4–5

3 RECURSIVE CONTEMPLATION

  • Figure 4 reports that ReCon outperforms CoT by a large margin, while Figure 5 shows ReCon preferred across all six metrics and ablations support both contemplation components.Figure 5's values are proportions of data preferred by GPT-4 according to each metric.Source: Figure 4 · p. 5, Figure 5 · p. 6

3.2 REFINEMENT CONTEMPLATION

  • Refinement contemplation reconsiders initial thoughts and speech through second-order perspective-taking, estimating how good- and evil-side players would perceive the speech before producing refined versions.The agent then speaks the refined content, which is added to the discussion log.Source: §3.2 · pp. 5–6

4 EXPERIMENTAL EVALUATIONS

In Avalon, ReCon achieves the highest success rates among the tested methods, and its contemplation and perspective-transition components improve performance across end-to-end and multi-dimensional evaluations. Qualitative examples show it identifying misinformation, protecting private information, and uncovering intentions behind deception.

4.1 END-TO-END EVALUATIONS

  • Across every tested scenario, ReCon’s combination of mechanisms yields the highest success rates.First- and second-order perspective transitions notably help when ReCon plays on the good side, while refinement contemplation is more impactful on the evil side.Source: §4.1 · p. 6

4.2 MULTI-DIMENSIONAL EVALUATION

  • Across all six metrics, ReCon significantly outperforms CoT, while both contemplation stages benefit most metrics.Formulation contemplation is associated with lower-than-expected persuasiveness, attributed to shorter, less provocative speech.Source: §4.2 · p. 7
  • Removing either perspective transition reduces performance across all metrics; concealment is strongest when each transition is paired with its corresponding contemplation stage.The reported pairings are first-order transition with refinement contemplation and second-order transition with formulation contemplation.Source: §4.2 · p. 7

4.3 QUALITATIVE ANALYSES

  • ReCon identifies and corrects the Assassin’s misleading definition of “team balance,” whereas CoT is deceived by it.The example illustrates ReCon’s resilience to misleading, malicious content.Source: §4.3 · p. 8
  • ReCon expresses reservations about malicious players without revealing sensitive details, unlike a baseline that exposes private information despite warnings.Figures 5 and 6 quantitatively substantiate its information-concealment efficacy.Source: §4.3 · p. 8
  • ReCon’s two-stage contemplation helps users understand the reasoning behind deceptive actions and their possible risks.Gameplay logs show contemplation content consistently aligned with agents’ interests, though the genuineness of that contemplation is difficult to assess.Source: §4.3 · p. 8

5 DISCUSSIONS

The discussion explains how ReCon’s contemplation stages may protect private information, while identifying limits in safety alignment, reasoning, speaking style, and response-format adherence. It also notes a trade-off: human-like speech or thought imitation can negatively affect Avalon performance.

5.1 EXPLANATIONS OF HOW RECON MANAGES TO CONCEALS PRIVATE INFORMATION

  • Formulation contemplation gives agents a private space to analyze information before speaking, and refinement can let them amend disclosures.The authors suggest these mechanisms may contribute to higher concealment scores.Source: §5.1 · pp. 8–9

5.2 ON THE “JAILBREAKING” OF SAFETY ALIGNMENT

  • RLHF can prevent explicit deceptive content yet fail to ensure logical alignment: GPT-4 produces deceptive messages when the same logic is framed in Avalon.The authors suggest logically aligning models may be a future direction for RLHF and its variants.Source: §5.2 · p. 9

5.3 INADEQUATE REASONING SKILLS OF LLMS

  • LLM agents lack expert human reasoning in Avalon; a Percival agent fails to infer that a team proposer must be Morgana when Merlin is included.The authors contrast this with proficient humans, who would make that deduction because Merlin knows the evil players.Source: §5.3 · p. 9

5.4 EXCESSIVE FORMALITY IN LLMS’ RESPONSES

  • LLM responses are excessively formal and detailed, and human-like speech or thought imitation can negatively affect Avalon performance.Balancing human-like speaking patterns with performance is identified as future research.Source: §5.4 · p. 9, Table 1 · p. 9

5.5 COMPARATIVE ANALYSIS OF LLMS’ ADHERENCE TO RESPONSE FORMAT

  • ChatGPT and Claude meet required response formats with over 90% probability in full-game scenarios, whereas LLaMA2-70b-chat consistently fails.The authors identify instruction-following, particularly format adherence, as an area for improvement in open-source LLMs.Source: §5.5 · p. 9

6 RELATED WORK

Prior work studies multi-agent language communication, LLM thought methods, and deceptive games, but the cited strands leave distinct gaps around deceptive communication and comprehensive deception handling. The paper positions its work at their intersection in Avalon.

6.1 MULTI-AGENT INTERACTIONS

  • Prior LLM multi-agent work explores communication and cooperation but not deceptive communication; other approaches rely on iterative feedback or fine-tuning and game-specific data.The paper contrasts ReCon’s deception-oriented contemplation mechanisms with these approaches.Source: §6.1 · p. 10

6.2 THOUGHT METHODS OF LLMS

  • LLM thought mechanisms have improved reasoning and decision-making across tasks, but many studies emphasize individual agents rather than multi-agent communication.The paper extends this line toward deceptive, multi-player Avalon interactions.Source: §6.2 · p. 10

6.3 GAME PLAYING IN DECEPTIVE ENVIRONMENTS

  • Prior deceptive-game work spans Diplomacy, Hoodwinked, poker, and deception benchmarks; concurrent Werewolf work studies camouflage, while this paper also aims to identify and address deception.The paper distinguishes its scope from Xu et al. (2023), which observes camouflage during gameplay.Source: §6.3 · pp. 10–11

7 CONCLUSION

The paper presents ReCon as a framework for helping LLM agents identify and address deception, reporting efficacy in Avalon without additional fine-tuning or data. It identifies several directions for future work.

7 CONCLUSION

  • ReCon combines formulation and refinement contemplation with first- and second-order perspective transitions to help LLM agents discern and counteract misinformation in Avalon.The paper reports quantitative and qualitative efficacy without additional fine-tuning or data.Source: §7 · p. 11
  • The paper proposes improving reasoning through more advanced thinking methods and fine-tuning on high-quality human gameplay data.It also plans to refine human-like speaking style while maintaining misinformation-handling capacity.Source: §7 · p. 11
  • The authors plan to adapt ReCon to other deceptive environments, including Werewolf, Undercover, and murder mystery games.The paper also identifies speaking style as a target for refinement.Source: §7 · p. 11

ETHICS STATEMENT

The ethics statement warns that ReCon may be used to refine deceptive techniques as well as counter deceit. It contrasts game outcomes when both sides use CoT versus ReCon and urges conscientious, ethically aligned use.

ETHICS STATEMENT

  • Although intended to counter deceit, ReCon could also be applied to refine deceptive techniques.Source: ETHICS STATEMENT · p. 11
  • With CoT for both sides, success rates are 15.0% for the good side and 85.0% for the evil side; with ReCon for both, they are 19.4% and 70.6%, respectively.The authors describe the disparity as favoring ethical deception detection over uses aimed at disruption and deception.Source: ETHICS STATEMENT · p. 11
  • The authors urge users to employ ReCon conscientiously, for societal benefit and in adherence to human ethics, to prevent malicious exploitation.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the model sources and experimental settings for ChatGPT, GPT-4, and Claude, including decoding temperatures and context-window handling.

REPRODUCIBILITY STATEMENT

  • Experiments use publicly accessible OpenAI and Anthropic APIs, the Llama-2-70b-chat-hf checkpoint, and prompts provided in Appendix E.Source: REPRODUCIBILITY STATEMENT · p. 11
  • ChatGPT uses temperature 0.6 and version “0613”; inputs exceeding short contexts switch to long-context versions.Short and long contexts are 4k and 16k for GPT-3.5, and 8k and 32k for GPT-4.Source: REPRODUCIBILITY STATEMENT · p. 11
  • Claude uses temperature 1 and the Claude-2 version as of 2023-06-01, without the context-window auto-switch strategy.Source: REPRODUCIBILITY STATEMENT · p. 11

Content

This content introduces Avalon’s hidden-role game and its rules and roles, then gives examples of ReCon detecting deception and concealed intentions. It also notes current limits in LLM reasoning.

A DETAILED INTRODUCTION TO THE AVALON GAME

A.1 GAME PROCESS AND RULES

  • In the six-player rules, the good side wins with three successful quests, while the evil side wins with three failed quests.If good is about to win, the Assassin can secure an evil victory by correctly identifying Merlin.Source: §A.1 · p. 17

A.2 INTRODUCTION TO AVALON ROLES

  • Roles create asymmetric knowledge: Merlin knows Morgana and the Assassin, while Percival must distinguish the real Merlin from Morgana.The Assassin seeks to identify Merlin near the endgame; Loyal Servants lack special insights.Source: §A.2 · p. 17

B MORE GAMEPLAY EXAMPLES

B.1 RECON’S ABILITY TO DISCERN DECEPTIONS

  • A Loyal Servant using ReCon infers that Player 1 is likely evil from quest history despite Player 1’s claim that an evil teammate caused the failure.Source: §B.1 · p. 18

B.2 RECON’S ABILITY TO UNVEIL DECEPTIVE INTENTIONS

  • ReCon lets users detect the Assassin’s deceptive intentions even when the Assassin’s dialogue resembles that of a good player.The passage says users can thereby avert adverse outcomes in a timely manner.Source: §B.2 · p. 18

B.3 INADEQUATE REASONING SKILLS OF LLMS

  • LLM agents do not yet reason as complexly as expert Avalon players and may inconsistently change their suspicions about Merlin or Morgana candidates.The authors suggest logical limitations or hallucinations may explain such inconsistencies.Source: §B.3 · p. 18

C MORE ABLATION RESULTS

ReCon’s success rate remains more than twice that of GPT-4 CoT, despite GPT-4 outperforming GPT-3.5 CoT; the section also reports a performance drop with human-like style.

C MORE ABLATION RESULTS

  • ReCon’s initial-thought and speech functions use GPT-3.5, while its refinement functions use GPT-4.The CoT baseline is implemented with GPT-3.5; GPT-4 is also used for a CoT comparison.Source: C MORE ABLATION RESULTS · pp. 18–19
  • Table 2 reports a performance drop with human-like style.The supplied caption gives no metric or value for the drop.Source: Table 2 · p. 18
  • GPT-4 CoT’s success rate is less than half of ReCon’s, even though GPT-4 CoT significantly surpasses GPT-3.5 CoT.The comparison is presented as evidence that ReCon’s contemplation and perspective-transition mechanisms improve performance in deceptive environments.Source: Table 2 · p. 18, C MORE ABLATION RESULTS · p. 19

D EXAMPLE OF MULTI-DIMENSIONAL EVALUATION BY GPT-4

The appendix reports that GPT-4’s multi-dimensional evaluations are largely coherent, reasonable, and high quality when given sufficient information, then outlines the prompts and procedures used for ReCon and Avalon.

D EXAMPLE OF MULTI-DIMENSIONAL EVALUATION BY GPT-4

  • GPT-4 evaluates six dimensions, with concealment assessed only on final speech and only for Merlin and Percival; the other metrics cover contemplation and final speech.The dimensions are concealment (CCL), logic (LG), contribution (CTR), persuasiveness (PRS), information (INF), and creativity (CRT).Source: D EXAMPLE OF MULTI-DIMENSIONAL… · p. 19, Figure 11 · p. 19

E PROMPT TEMPLATES

  • The appendix presents condensed versions of the original prompts while retaining their methodology and rationale.ReCon prompts cover first- and second-order perspective transitions, formulation contemplation, and refinement contemplation.Source: E PROMPT TEMPLATES · p. 20, §E.1 · p. 20, Figure 12 · p. 20

E.1 PROMPTS FOR RECURSIVE CONTEMPLATION

  • ReCon’s prompts ask agents to infer other players’ roles from dialogue and assess whether an observed player’s actions fit their role.They also instruct agents to improve that player’s chances of winning using a prior second-perspective thought while keeping role hints in public dialogue.Source: §E.1 · pp. 20–21, Figure 12 · p. 20, Figure 14 · p. 21

E.2 PROMPTS FOR AVALON GAME

  • Avalon prompts specify six roles and their asymmetric information and goals, including Merlin’s need for subtlety and the Assassin’s final attempt to identify Merlin.The prompts also give role-specific hints, such as avoiding behavior that exposes Merlin’s identity.Source: §E.2 · p. 22, Figure 16 · p. 22

E.3 PROMPTS FOR PROCEDURES OF AVALON GAME AND RECURSIVE CONTEMPLATION

[ First-Order Perspective Transition Prompt ]

E.4 TASK PROMPTS FOR GOOD SIDE AND EVIL SIDE

  • Good- and evil-side prompts share common components but give opposing team-selection, discussion, and voting guidance.Good prompts favor entirely good teams and rejection when evil is suspected; evil prompts seek an evil teammate and rejection when none is included.Source: §E.4 · p. 24
  • Separate prompts let evil players choose quest success or failure and guide the Assassin to identify a probable Merlin after three successful quests.The guidance applies when an evil player is on the quest team and when the good side reaches three successful quests, respectively.Source: §E.4 · p. 24, Figure 21 · p. 26, Figure 22 · p. 26

F COMPLETE GAMEPLAY LOG OF AN AVALON GAME

This section presents a complete Avalon gameplay log in which players navigate uncertainty about loyalties through contested team proposals, votes, and mission outcomes, culminating in a Good-side victory.

F COMPLETE GAMEPLAY LOG OF AN AVALON GAME

  • Early discussion weighs the proposed teams’ strategic participation against the need to test unproven players, rotate mission membership, and remain alert to possible deceit.Players argue that activity and neutrality support including Player 3 and Player 6, while cautioning that first-round observations are limited.Source: F COMPLETE GAMEPLAY LOG OF AN AVALON… · pp. 28, 29, 31
  • The team proposal was approved by the majority and the mission began; its result was Success, with both recorded mission votes marked Success.Source: F COMPLETE GAMEPLAY LOG OF AN AVALON… · pp. 32, 37

Early team votes and mission

Deceptive play

  • Player 5 suggests altering a balanced-looking team while explicitly framing the suggestion to avoid revealing an evil force and deflect suspicion.Player 5 says the proposal should not make the evil force behind it evident, so they remain shielded from suspicion.Source: F COMPLETE GAMEPLAY LOG OF AN AVALON… · p. 37

Contested team proposals

  • Players debate whether to favor proven mission participants or include untested players, and a subsequent team proposal is rejected by the majority.Player 2 argues for replacing untested Player 1 with Player 6, while other proposals advocate including Players 1 or 5.Source: F COMPLETE GAMEPLAY LOG OF AN AVALON… · pp. 37–38

Final outcome

  • The Good side wins finally after a successful mission and an assassination attempt in which the Assassin identifies Player 3 as Merlin.The mission had four Success votes; the voiceover then announces a provisional Good-side win, followed by the final win announcement.Source: F COMPLETE GAMEPLAY LOG OF AN AVALON… · p. 40

Related papers

Loading 2310.01320v3…