Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning

1 INTRODUCTION

• Self-play can succeed in two-player zero-sum games, but cooperation with humans may require conventions that self-play does not learn, and prior no-press Diplomacy and Hanabi agents struggled with humans.The paper uses communication conventions such as English as an example of behavior self-play without human data may not acquire.Source: §1 · p. 1
• DiL-piKL extends piKL, and RL-DiL-piKL combines it with self-play reinforcement learning to train an agent intended to perform well with humans.piKL regularizes inference-time planning toward a behavioral-cloning policy.Source: §1 · p. 2
• Both Diplodocus agents ranked among the top two average scores for participants with more than two games and first and third overall by Elo in a 200-game tournament.The tournament included one bot and six humans per game; the comparison covered 48 participants who played more than two games.Source: §1 · p. 2

2 BACKGROUND AND PRIOR WORK

• No-press Diplomacy is a seven-player mixed cooperative-competitive game in which players coordinate implicitly through moves rather than cheap-talk communication.Players compete for control of 34 supply centers, and turns involve simultaneous orders for their units.Source: §2 · p. 2

2.1 MARKOV GAMES

• A Markov game models states, player action spaces, rewards, and transitions; each player chooses a policy to maximize expected reward given the other players’ policies.With one player, it reduces to an MDP with a fixed environment.Source: §2.1 · p. 3

2.2 HEDGE

• Hedge and regret matching are equilibrium-finding approaches used in planning; hedge computes an equilibrium policy on each turn, while regret matching is evaluated only as a baseline here.Hedge’s average policy converges to a coarse correlated equilibrium as iterations grow.Source: §2.2 · p. 3

2.3 DORA: SELF-PLAY LEARNING IN MARKOV GAMES

• DORA uses one-step equilibrium search with policy and value networks, then trains through self-play; it performs poorly with human-like agents despite doing well against copies of itself.The policy proposal network limits the actions considered to a tractable subset.Source: §2.3 · pp. 3–4

2.4 PIKL: MODELING HUMANS WITH IMITATION-ANCHORED PLANNING

• Behavioral cloning can underperform the humans it imitates, while piKL regularizes planning toward an imitation policy using λ to trade off reward against closeness to that anchor.Large λ emphasizes closeness to the anchor; small λ emphasizes reward.Source: §2.4 · pp. 4–5

3 DISTRIBUTIONAL LAMBDA PIKL (DIL-PIKL)

• DiL-piKL samples λ from a distribution on each iteration instead of using piKL’s single fixed value, producing a population of agent types with different degrees of regularization.High-λ types stay closer to the anchor; low-λ types are more reward-oriented and less constrained by it.Source: §3 · p. 5

3.1 COORDINATING WITH PIKL POLICIES

• The agent models the population by sampling λ during equilibrium computation but plays a low-λ policy, interpolating among behavioral cloning, its best response, and equilibrium policies.This distinguishes the λ values assumed for other players from the value used for the agent’s own policy.Source: §3.1 · p. 6

3.2 THEORETICAL PROPERTIES OF DIL-PIKL

• DiL-piKL is analyzed as sampled follow-the-regularized-leader and has a regret bound against the regularized utilities for each player and type.The analysis notes that standard FTRL analysis does not directly apply because utilities and gradients can be unbounded.Source: §3.2 · p. 6
• In two-player zero-sum games, DiL-piKL policies converge almost surely to the unique Bayes-Nash equilibrium; the last-iterate guarantee relies on strongly convex regularized utilities.The proof also handles agents observing sampled opponent actions.Source: §3.2 · p. 6

4 DESCRIPTION OF DIPLODOCUS

• RL-DiL-piKL uses DiL-piKL in place of DORA’s equilibrium-finding algorithm to train policy and value proposal networks, with DiL-piKL also used in test-time search.The authors hypothesize this learns a strong, human-compatible policy and a value function that evaluates states under strong, human-like continuations.Source: §4 · p. 6

4.1 TRAINING

• Training largely follows DORA, but uses DiL-piKL with a λ distribution and a fixed human imitation anchor instead of regret matching to compute each turn’s equilibrium policy.The loss functions are identical to DORA’s.Source: §4.1 · p. 7

4.2 TEST-TIME SEARCH

• At test time, one-ply search samples up to 30 likely actions per player from the policy proposal network and applies DiL-piKL instead of regret matching.The agent uses the same behavioral-cloning anchor in all experiments; its first-turn λ settings are changed to produce more diverse openings resembling human openings.Source: §4.2 · p. 7

5.1 EXPERIMENTAL SETUP

• The baseline-population evaluation sampled all seven players from baseline agents or the tested agent; the proposed variants used different two-type λ distributions.Diplodocus-Low used {10−4, 10−1}, while Diplodocus-High used {10−2, 10−1}.Source: §5.1 · p. 7
• The human tournament tested BRBot, DORA, Diplodocus-Low, and Diplodocus-High for 50 games each; each game included one agent and six humans.Across 200 games, 62 human participants took part; games used five-minute turns and a stochastic end rule.Source: §5.1 · pp. 7–8

5.2 EXPERIMENTAL RESULTS

• Diplodocus-Low and Diplodocus-High performed best by a wide margin in the fixed population of baseline agents.Source: §5.2 · p. 8
• Large statistical error margins prevent conclusive claims that Diplodocus beat the best humans or clearly separated from BRBot; the results indicate performance at least at expert level.Two participants had higher average scores but played only two games and one game, respectively.Source: §5.2 · p. 8
• Diplodocus-High performed best among human participants by both Elo and average score; Diplodocus-Low ranked second by average score and third by Elo.The table covers 43 participants who played at least five games.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 RL TRAINING COMPARISON

• As RL training progressed, Diplodocus-High improved its score while keeping human-move prediction accuracy high; unregularized DORA and DNVI diverged from human play.Scores were measured against six HedgeBot copies, and prediction accuracy on held-out human games.Source: §5.3 · p. 9, Figure 3 · p. 9

6 DISCUSSION

• DiL-piKL models varied human player types while playing a strong low-λ policy, and RL self-play learns values across situations generated by strong and human-like players.The human continuation model makes fewer blunders than behavioral cloning, supporting value estimates for positions requiring accurate play, such as stalemate lines.Source: §6 · p. 10

B DESCRIPTION OF DIPLOMACY

• No-press Diplomacy uses simultaneous orders and no cheap-talk communication, so players communicate implicitly through moves.Players compete for supply centers even though cooperation is almost essential to victory.Source: B DESCRIPTION OF DIPLOMACY · p. 12

WHAT IS THE STRONGEST AGENT?

• Experts disagreed on the strongest agent: one favored Diplodocus-Low, another Diplodocus-High, and a third favored Diplodocus-Low.The first expert placed BRBot close behind Diplodocus-Low and DORA a distant third.Source: WHAT IS THE STRONGEST AGENT? · p. 13

WHAT IS THE MOST HUMAN-LIKE/BOT-LIKE AGENT?

• Two experts identified Diplodocus-High as most human-like, while one described it as human-like and rated Diplodocus-Low not far behind.That expert characterized BRBot and DORA as very non-human, in different ways.Source: WHAT IS THE MOST HUMAN-LIKE/BOT-LIKE… · p. 13

WHAT IS THE AGENT YOU’D LIKE TO COOPERATE WITH?

• Experts differed on whom they would cooperate with: two chose Diplodocus-Low and one chose Diplodocus-High.One expert found High less responsive to who would work with it; another described Low as predictable in alliance changes and potentially exploitable.Source: WHAT IS THE AGENT YOU’D LIKE TO… · p. 13

C.3 BRBOT

• BRBot’s overall-strength assessments ranged from weak to better-than-average-human, with one expert noting its performance exceeded the impression it gave.Other assessments cited tactical weaknesses, dependence on neighboring mistakes, and variable defense.Source: HOW WOULD YOU EVALUATE THE OVERALL… · p. 14, §C.3 · p. 14
• Experts described BRBot’s cooperation as inconsistent: signaling and coordination could be effective, but indiscriminate signaling and weak support choices limited them.One expert said signaling willingness to work with every power conveys little, especially when the agent may still stab.Source: §C.3 · pp. 14–15

C.4 DIPLODOCUS-HIGH

• One expert described Diplodocus-High’s tactics as unadventurous and sometimes below human standards, while also crediting it with a longer strategic view.That expert called its play practical and boring.Source: §C.4 · p. 15
• Experts described Diplodocus-High as generally strong, with good strategic sense and discipline, though some early strategies harmed its ability to get out of the box.One expert praised its handling of an unusual Austrian position under pressure on multiple fronts; another said it may fail to adapt when its favored alliance is declined.Source: §C.4 · p. 15
• Assessments of Diplodocus-High’s cooperation ranged from low to strong, including reports of useful signaling and alliance formation alongside difficulty choosing or adapting allies.One expert warned that the behavior would not hold up in press games; another described world-wide fights as self-defeating.Source: §C.4 · p. 15

C.5 DIPLODOCUS-LOW

• Experts split sharply on Diplodocus-Low’s strength: one called it exceptional, another very weak, and a third very strong.The favorable assessments cited tactical strength, strategic direction, and alliance signaling; the negative one described abandoning its position for questionable adventures.Source: §C.5 · p. 16
• Diplodocus-Low’s cooperation reviews ranged from poor to good, with experts citing both effective signaling and support and cases of meaningless support or rapid alliance reassessment.One expert said it sometimes undid good alliance-building work by reassessing alliances every turn.Source: §C.5 · p. 16

D POPULATION BASED EVALUATION

• Because nontransitivity and incompatible equilibria can distort head-to-head results, the authors test agents against populations intended to represent a wider suite of human-like agents.Each test adds the agent being evaluated to a fixed baseline population, samples seven players uniformly with replacement, and retains games in which the tested agent appears.Source: D POPULATION BASED EVALUATION · p. 16
• The population tests use eight baseline agents, including best-response, regret-matching, behavioral-cloning, and DiL-piKL variants.The listed variants include policies and value functions trained with behavioral cloning or self-play.Source: D POPULATION BASED EVALUATION · p. 17

E THEORETICAL PROPERTIES OF DIL-PIKL

• In two-player zero-sum games, DiL-piKL converges to the unique Bayes-Nash equilibrium of the regularized Bayesian game.With one type per player, the corollary is convergence of piKL to the Nash equilibrium of the regularized game.Source: E THEORETICAL PROPERTIES OF DIL-PIKL · pp. 17–18

E.1 REGRET ANALYSIS

• DiL-piKL satisfies a regret bound for each player and supported type under the regularized utilities.The analysis derives the bound using convexity and strong convexity of KL divergence, then telescopes the resulting inequality.Source: §E.1 · pp. 19–20

E.2 LAST-ITERATE CONVERGENCE IN TWO-PLAYER ZERO-SUM GAMES

• When both players use DiL-piKL, their policies converge to the unique Bayes-Nash equilibrium of the regularized game.The theorem states convergence in expectation, with high probability, and almost surely; the expected rate is roughly log T/(λ_i T).Source: §E.2 · p. 24
• The high-probability convergence argument uses Azuma-Hoeffding concentration for a martingale difference sequence.The proof obtains the almost-sure statement from the high-probability result using the Borel-Cantelli lemma.Source: §E.2 · pp. 24–25

F MODEL ARCHITECTURE

• Each of 81 board locations receives 38 feature channels, alongside previous-state and order-history encodings.The order-history encoding adds 202 channels per board location.Source: F MODEL ARCHITECTURE · p. 25
• Per-location, per-player, and global tensors are projected to 224 channels, concatenated, and processed by a 10-layer transformer encoder.The encoder has width 224, eight self-attention heads per layer, and GeLU activation.Source: F MODEL ARCHITECTURE · pp. 25–26
• An LSTM decoder produces the policy, while a separate attention-based value head predicts values for all seven players.The value head ends with a seven-channel linear layer and softmax.Source: F MODEL ARCHITECTURE · p. 26

G HUMAN IMITATION ANCHOR POLICY TRAINING

• The human imitation anchor is trained jointly on full-press and no-press games from roughly 46,000 online Diplomacy games.At inference and/or during RL, a global feature indicates that the model should predict for no-press Diplomacy.Source: G HUMAN IMITATION ANCHOR POLICY TRAINING · p. 27
• Training skips timed-out default actions and keeps actions from the top half of rated players.Player labels are randomly permuted during training to augment data, reflecting the game’s permutation-equivalent rules.Source: G HUMAN IMITATION ANCHOR POLICY TRAINING · p. 27
• The resulting human imitation policy and value function initialize the RL policy and serve as the human anchor for piKL regularization.The architecture is described in Appendix F; the passage refers to Table 6 for training hyperparameters.Source: G HUMAN IMITATION ANCHOR POLICY TRAINING · p. 27, Table 6 · p. 27

H SELF-PLAY TRAINING

• The RL policy and value functions start from the human imitation anchor, and DiL-piKL replaces regret matching for computing the one-step equilibrium.The distribution of λ and the human anchor policy remain fixed throughout training.Source: H SELF-PLAY TRAINING · p. 27
• Training samples a random λ for exploration and policy prediction, exposing the models to states from both human-like and more optimized actions.Higher λ yields more human imitation-like actions; lower λ yields more optimized actions.Source: H SELF-PLAY TRAINING · p. 27
• Unlike DORA, the training procedure omits double-oracle action exploration, which the authors found unnecessary with the human anchor’s diversity and regularization.The procedure otherwise closely matches DORA in its recorded data, loss, and sampling methods.Source: H SELF-PLAY TRAINING · p. 27

H.1 HYPER-PARAMETERS FOR RL TRAINING

• The evaluation retrains DNVI, DNVI-NPU, and DORA and trains Diplodocus and BRBot agents; the training hyperparameters are listed in Table 7.The passage says three parameters from Bakhtin et al. (2021) are shown because they are the same.Source: §H.1 · p. 28

H.2 INFERENCE TIME EFFECT OF DIL-PIKL

• Applying DiL-piKL at evaluation time to human imitation-trained policy and value functions does not improve population score over Hedge, whereas search over RL-trained policies and values does.The reported comparison is shown in Figure 6.Source: §H.2 · p. 28

• Applying DiL-piKL at inference time rather than Hedge slightly improves an IL-based search agent but greatly improves RL agents trained with DiL-piKL.Source: Figure 6 · p. 29
• 400 Elo points generally correspond to a 10-fold increase in expected winning odds or average score in the model’s rating scale.Source: I BAYES-ELO · pp. 28–29
• The model jointly estimates player ratings and starting-position biases by maximum a posteriori, using a weak prior centered at 0 with about 350 Elo standard deviation.Source: I BAYES-ELO · p. 29
• Extended to seven players, BayesElo jointly fits seven starting-country biases, producing ratings that reflect empirical winning percentages while accounting for opposition strength and starting advantage.A player with the same results after drawing harder starting countries may receive a higher rating than one who was luckier in country assignments.Source: I BAYES-ELO · p. 29
• France is among the strongest no-press starts, while Austria and England are among the weaker starts, according to fitted country advantages across the 200 tournament games.Source: Table 8 · p. 30, I BAYES-ELO · p. 30