Source-linked AI summary
Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy
Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, Yizhou Wang
TL;DR
Diplomacy demands long-horizon planning and negotiation alongside social reasoning under partial observation, while prior approaches relied heavily on domain-specific human data. Richelieu combines social reasoning, planning, memory, and reflection in a self-evolving LLM agent trained through self-play without human data; the authors report that it outperforms existing models such as Cicero.
Problem
Prior approaches relied heavily on domain-specific human data, while Diplomacy requires long-horizon planning and social reasoning under partial observation.
Source: §1 · pp. 1–2Method
Richelieu integrates social reasoning, strategic planning, memory, and reflection in a self-evolving LLM agent that acquires experience through self-play without task-specific human data.
Source: §1 · p. 2Results
Richelieu ultimately outperforms existing models such as Cicero, with its win rate approximately 0.7% higher than Cicero’s.
Source: §5.2 · p. 8, §6 · p. 10Takeaways & Limitations
The authors conclude that LLM-based agents may become an effective approach in social science.
Source: §6 · p. 10Diplomacy constrains the action space and makes information public and certain outside negotiation, unlike real-world diplomacy’s virtually boundless decisions and incomplete information.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Diplomacy is one of the most sophisticated activities in human society, involving complex interactions among multiple parties that require skills in social reasoning, negotiation, and long-term strategic planning. Previous AI agents have demonstrated their ability to handle multi-step games and large action spaces in multi-agent tasks. However, diplomacy involves a staggering magnitude of decision spaces, especially considering the negotiation stage required. While recent agents based on large language models (LLMs) have shown potential in various applications, they still struggle with extended planning periods in complex multi-agent settings. Leveraging recent technologies for LLM-based agents, we aim to explore AI's potential to create a human-like agent capable of executing comprehensive multi-agent missions by integrating three fundamental capabilities: 1) strategic planning with memory and reflection; 2) goal-oriented negotiation with social reasoning; and 3) augmenting memory through self-play games for self-evolution without human in the loop.
1 Introduction
Diplomacy combines long-horizon planning and negotiation with social reasoning under partial observation, while prior approaches relied heavily on domain-specific human data. Richelieu is presented as a self-evolving LLM agent that integrates social reasoning, planning, memory, and reflection.
1 Introduction
- Prior Diplomacy agents relied heavily on domain-specific human data, motivating the question of whether an agent can excel without it.Diplomacy also requires long-horizon planning, natural-language communication, and social reasoning under partial observation.Source: §1 · pp. 1–2
- Richelieu is introduced as an LLM-based agent built to explore human-like AI diplomacy through four core capabilities.The introduction identifies social reasoning, balanced short- and long-term planning, memory management, and self-reflection.Source: §1 · p. 2
- Its memory system recalls and integrates past negotiations and actions to inform current and future decisions and support agent evolution.Self-reflection analyzes decisions and memory experience so the agent can adapt its strategies.Source: §1 · p. 2
- The authors report that Richelieu self-evolves through self-play without task-specific human data and outperforms Cicero, while also analyzing modules and LLM generalization.They name GPT4.0 and Llama 3 as examples of alternative LLMs considered.Source: §1 · p. 2
2 Related work
Prior Diplomacy work addressed planning in a huge action space and later expanded to press settings, but existing approaches often depended on human data or separated negotiation from planning. Richelieu is positioned as a unified, self-evolving LLM agent without pre-collected human expert data.
2 Related work
- Diplomacy presents 1021 to 1064 actions per turn, and many planning approaches—including DipNet, BRPI, SearchBot, and DORA—used human data.Earlier work largely focused on No-Press Diplomacy, while press-enabled studies later emerged.Source: §2 · pp. 2–3
- Cicero combines a language model for negotiation with an RL model for planning, while another approach learns negotiation agents from predefined contracts or protocols.The passage characterizes these separately trained models as inconvenient for continual evolution.Source: §2 · p. 3
- The paper distinguishes Richelieu from prior approaches by addressing negotiation and planning in one self-evolving LLM agent without pre-collected human expert data.The related-work discussion contrasts this with LLM agents focused only on negotiation or planning.Source: §2 · p. 3
3 Problem Statement
Diplomacy models seven Great Powers competing to control supply centers, with victory determined by holding a majority rather than eliminating opponents. Each turn combines observation of the game state and other players’ prior actions with negotiation and military or diplomatic actions.
3 Problem Statement
- Seven players represent the Great Powers and seek to control a majority of supply centers by the end of the Fall phase.Victory is not achieved by eliminating other players or their units.Source: §3 · p. 3
- An agent’s turn begins with the current state and other players’ actions from the previous turn, then includes negotiation messages and the agent’s own action.The state includes territory ownership and army locations.Source: §3 · p. 3
- Available actions include military commands such as moving, supporting, or holding, as well as diplomatic moves such as proposing or withdrawing from an alliance.Diplomatic moves are less formalized in the game mechanics.Source: §3 · p. 3
4 Self-Evolving LLM-based Diplomat
Richelieu integrates social reasoning, reflective subgoal planning, negotiation, action selection, memory management, and self-play-based evolution. Its memory stores experience to inform later decisions, while self-play supplies experience without human demonstrations.
4 Self-Evolving LLM-based Diplomat
- Each turn combines social-belief reasoning, reflective subgoal planning, negotiation, action selection, and logging of states, negotiations, actions, and subgoals for future use.Richelieu also evolves through multi-agent self-play, in which its agents control all countries to acquire diverse experience without human demonstrations.Source: §4 · p. 4
4.1 Social Reasoning
- Social reasoning identifies shifting relationships and intentions under partial observation, informing subgoals, negotiation, and judgments about other players’ statements.Richelieu models potential allies, enemies, vulnerable neighbors, and long-term threats.Source: §4.1 · pp. 4–5
4.2 Strategic Planner with Reflection
- The planner links immediate actions to victory by proposing subgoals that account for inferred social beliefs and can contain rivals or expand strategically.Reflection retrieves similar past experiences using goal and state similarity, then uses their evaluations to reinforce or adjust proposed subgoals.Source: §4.2 · p. 5
4.3 Negotiator and Actor
- Negotiation updates social beliefs and pursues subgoals through agreements or conflict management, while the agent assesses opponents’ statements using inferred intentions, relevant experience, and honesty scores.After negotiation, the actor selects an action based on the updated social beliefs.Source: §4.3 · p. 5
4.4 Memory Management and Evolution in Self-Play Games
- Memory archives game states, subgoals, messages, and actions, then summarizes experience with subgoal evaluations and player-credibility assessments.Subgoal evaluations are updated as the game unfolds and stop when the subgoal fundamentally changes; player credibility is short-term memory, while other records form long-term memory.Source: §4.4 · p. 6
- Self-play training increased Richelieu’s win rate against Cicero before performance plateaued, while its defeated rate consistently decreased toward an asymptotic value.The authors report these trends as evidence of self-play’s effectiveness in Richelieu’s evolution.Source: §4.4 · pp. 6–7
5 Experiment
The experiments test Richelieu in no-press and press Diplomacy, comparing it with prior agents and AutoGPT while assessing generalization, module contributions, and social reasoning. Richelieu outperforms prior models in no-press play and reports better outcomes than Cicero and AutoGPT in press play.
5 Experiment
- The experiment questions cover non-press mastery, press performance, compatibility with different LLMs, module contributions, and social reasoning.In each game, a model plays one randomly selected country against countries controlled by other methods.Source: §5 · p. 7, §5.1 · p. 7
5.1 Experimental Setup
- The evaluation uses randomly assigned countries, typically 1,000 game rounds, outcome rates and C-Diplo scores, and six prior-model baselines plus AutoGPT.The experiments cover no-press and press settings; LLM policy generation uses temperature 0.3.Source: §5.1 · pp. 7–8
5.2 Results
- In no-press play, Richelieu outperforms the six prior models by a clear margin despite not using their human game data.Figure 4 presents relative scores among seven agents, with each point giving the vertical-axis model’s score divided by the horizontal-axis model’s score.Source: §5.2 · p. 8, Figure 4 · p. 8
- In press play, Richelieu’s win rate is about 0.7% higher than Cicero’s and its score is about 10% higher.Its Most SC rate is about 2% higher and its loss rate 0.6% lower than Cicero’s.Source: §5.2 · p. 8
- Richelieu outperforms the AutoGPT baseline in press play when three countries are assigned to Richelieu and four to AutoGPT.Each agent controls its assigned countries independently.Source: §5.2 · p. 8, Table 2 · p. 8
- Across four LLMs, Richelieu’s win rate rises with self-play training; GPT4.0 moves from 1.5% below Cicero to about 0.7% above.Llama 3 moves from 2.3% below Cicero to almost equal; Spark Desk and ERNIE Bot move from 3% and 4% below to 0.7% and 1.6% below, respectively.Source: §5.2 · p. 9, Figure 5 · p. 9
- Ablations show steady improvement from adding individual modules, with gains in planning, negotiation, and performance after self-play.The authors report that modules help Richelieu use other players’ actions, weigh short- and long-term benefits, express cooperative intentions, and avoid deception.Source: §5.2 · p. 9, Table 3 · p. 9
- Social-reasoning accuracy is consistent with Richelieu’s overall performance, evaluated in games with four GPT-4.0-controlled countries and three Llama3-controlled countries.The authors note that baselines do not explicitly model relationships and intentions, so ground-truth comparison is unavailable.Source: §5.2 · p. 10
6 Conclusion
Richelieu combines hierarchical planning, reflective memory, and self-play without human data, and the paper reports that it outperforms existing models like Cicero in Diplomacy. The conclusion also identifies social-relationship and intention inference as an evaluated capability.
6 Conclusion
- Table 4 concerns success rates for identifying social relationships and inferring others’ intentions.Source: Table 4 · p. 10
- Richelieu ultimately outperforms existing models like Cicero in Diplomacy.The paper also reports ablation evidence for its modules and generalization across different LLMs.Source: §6 · p. 10
- The agent uses hierarchical planning and reflective memory, and can evolve through self-play without human data.Source: §6 · p. 10
7 Limitations and Future Work
The authors distinguish Diplomacy from real-world diplomacy by its constrained action space and by public, certain information outside negotiation. They propose a more realistic game space and extension to other multi-agent scenarios as future work.
7 Limitations and Future Work
- Diplomacy constrains the action space, and all information except negotiation exchanges is public and certain, unlike real-world diplomacy’s incomplete information.Source: §7 · p. 10
- Future work will develop a more realistic game space with incomplete information and multi-player games, and extend the framework to other multi-agent scenarios.The authors name embodied interactions, sensor networks, and video games as possible scenarios, and mention business and finance analytics and negotiation models.Source: §7 · p. 10
A Implementation Details
The implementation appendix outlines Diplomacy’s rules, strategic domain knowledge, and prompt templates for Richelieu’s modules. The templates include state and relationship assessment as well as reflective sub-goal optimization.
A.1 Rules of Diplomacy Game
- The game rules cover supply-center victory and elimination, unit movement and support, autumn occupation, transport, and unit limits tied to controlled centers.Source: §A.1 · p. 16
A.2 Domain Knowledge
- The domain knowledge prompts Richelieu to manage alliances and balance of power, divide opponents, negotiate ceasefires, and guard against or use deception.Source: §A.2 · p. 16
A.3 Prompt Templates
- The prompt templates provide country and map context, then ask for assessments of other countries’ intentions, threats, and potential allies.Source: §A.3 · pp. 16–17
- The reflective planner sets a country sub-goal using ally and enemy context, historical experiences, and evaluation of the eventual result.Source: §A.3 · p. 17
B.1 Cases of the Effect of the Memory from Self-Playing and Collaboration
The before-and-after case shows Richelieu missing Russia’s growing threat before self-play, then using historical memory to redirect strategy and secure cooperation with Austria. The post-self-play round achieved the most supply centers.
B.1 Cases of the Effect of the Memory from Self-Playing and Collaboration
- The case begins with France at war with Austria while Russia advances against Turkey, despite France and Russia having no border or conflict of interest.Source: §B.1 · p. 17
- Before self-play, Richelieu failed to anticipate Russia’s threat and continued attacking Austria; Russia ultimately won the game.Source: §B.1 · p. 17, Figure 6 · p. 18
- After self-play, Richelieu shifted to weakening Russia, allied with Austria and Turkey, and coordinated successfully with Austria, achieving the most supply centers that round.The figure compares the case before and after self-play.Source: §B.1 · p. 17, Figure 6 · p. 18
B.2 Case of Avoiding Deception
Richelieu detects England’s deceptive alliance proposal and counters by preserving its position in Belgium while seeking an understanding with France. England’s subsequent attack on Belgium confirms the suspicion and fails.
B.2 Case of Avoiding Deception
- England proposed a ceasefire and alliance against France, while Richelieu suspected England intended to attack Belgium or Berlin after German units were redirected.Richelieu pretended to accept the proposal while seeking a ceasefire with France.Source: §B.2 · p. 18
- England’s attack from Holland on Belgium failed because Richelieu had not moved its units from Belgium.The action phase confirmed Richelieu’s suspicions.Source: §B.2 · p. 18
- Richelieu’s response paired a feigned acceptance of England’s alliance with an effort to let France focus on defending against England.Source: §B.2 · p. 18
C More application
The authors present the framework as applicable beyond Diplomacy, including economic games and daily interactions, and report a werewolf-game evaluation. They also flag possible misuse for fraud.
C More application
- The authors state that the framework’s modules cover challenges in multi-agent interactions, including economic games and daily interactions.Source: C More application · p. 18
- Figure 7 is described as an example of avoiding deception during negotiations.Source: Figure 7 · p. 19
- 59.2% was Richelieu’s win rate as a werewolf, versus 65% for the strongest specifically designed LLM-based agent Xu et al. [2023].The seven-player experiment used Richelieu as one of two werewolves and did not apply self-play.Source: C More application · pp. 18–19
D Ethical Consideration
- The authors identify possible fraud as a negative use, alongside potential uses such as cooperation through negotiation or avoiding deception.They urge further research to foster safe applications.Source: D Ethical Consideration · p. 19
NeurIPS Paper Checklist
The checklist records the authors’ claims about the paper’s scope, limitations, and supporting materials. Its responses also address experimental reporting, ethics, assets, and human-subjects research.
NeurIPS Paper Checklist
- The authors state that the introduction and abstract fully cover the paper’s claims and scope, and that limitations are discussed in “Limitation and Future Work.”Source: NeurIPS Paper Checklist · pp. 20–21
- The checklist marks the theoretical-result question “Yes,” asserting that assumptions and complete proofs appear in the Method and Experiment sections.Source: NeurIPS Paper Checklist · p. 21
- The authors claim the paper provides information to reproduce its main experiments, open access to data and code, and training and test details.Source: NeurIPS Paper Checklist · pp. 21–23
- The checklist justifications say the paper reports statistical-significance information and sufficient computer-resource details for each experiment.Source: NeurIPS Paper Checklist · p. 23
- The authors report addressing societal impacts and release safeguards, documenting assets, and not involving human subjects.Source: NeurIPS Paper Checklist · pp. 24–26