Multi-Agent Training for Pommerman: Curriculum Learning and Population-based Self-Play Approach

• Pommerman is an open-source Python environment introduced in 2018, with team communication and challenges including sparse rewards and delayed bomb effects [1] [2].Games can last up to 800 timesteps, and bomb effects are delayed by 10 timesteps.Source: §1 · p. 1
• The training system first uses three curriculum phases to build gameplay skills, then uses population-based self-play for strategy improvement.The curriculum targets exploration, item collection, evasion, defense, and survival before self-play begins.Source: §1 · p. 2
• Performance-based adaptive annealing adjusts dense exploration rewards, while Elo-based matchmaking pairs agents for progressive learning.The paper presents these as responses to sparse-reward exploration and effective-matchmaking challenges in competitive self-play [3].Source: §1 · p. 2
• The trained agent outperforms learning agents without requiring communication among allied agents.This is the paper’s stated experimental contribution.Source: §1 · p. 2

2.1.1 Curriculum Learning with Incremental Difficulty Agents

• Three curriculum phases progress from exploring the map with static opponents, to placing bombs against moving opponents, to surviving and fighting bomb-placing opponents.The next phase activates when the training agent reaches a 55% win rate over the current rule-based agent.Source: §2.1.1 · p. 3

2.1.2 Adaptive Exploration Reward by Performance

• The adaptive annealing factor uses agent performance, measured as average enemy deaths, to shift the weight from dense exploration reward toward game reward.The performance range is [0, 2], and k is set to 1.2; the approach responds to the conflict between early exploration needs and sparse-reward self-play.Source: §2.1.2 · pp. 3–4

2.2.1 Population-based Self-play System

• Population self-play uses eight agents: three curriculum rule-based agents and five learning agents initialized from the curriculum-trained agent.An agent below a 45% win rate over the population is replaced by a stronger agent selected randomly from the four remaining agents.Source: §2.2.1 · p. 4

2.2.2 Match Making Probability

• Elo-based matchmaking applies softmax to expected win rates so higher-rated agents have a higher chance of being selected as opponents.Ratings are updated after matches using the actual score and expected win rate, with K as the maximum per-game adjustment.Source: §2.2.2 · pp. 4–5

3 Experiment Results

• Training agents used actor-critic with Proximal Policy Optimization (PPO).The network architecture is detailed in Appendix D.Source: §3 · p. 5

3.1 Curriculum Learning Stage

• After about 5 million timesteps, the agent reached a 55% win rate against the static agent; entering the simple-moving phase dropped performance and increased annealing, prompting more item collection and bombing.The agent had to find better enemy-elimination strategies as the second phase became harder.Source: §3.1 · p. 5
• In the third curriculum phase, the agent learned to dodge explosions while threatening and eliminating opponents.After curriculum training, dense exploration reward was annealed to zero, leaving sparse game rewards for self-play.Source: §3.1 · pp. 5, 7
• With linear annealing, the agent could not pass the second phase because the schedule did not adjust to encourage exploration; manual tuning might improve results but adds hyperparameter work.Figure 5 depicts enemy deaths with the linear annealing factor.Source: §3.1 · pp. 5, 7, Figure 5 · p. 6

3.2 Self-play Stage

• During self-play, the agents’ Elo rating rose from 1040 after curriculum training to 1160 after 200 million training timesteps.Figure 6 shows the ratings of five training agents.Source: §3.2 · p. 7, Figure 6 · p. 7
• In 100 round-robin matches, the method reached 982 Elo, exceeding two top 2018 learning agents and Neoteric; win rates were 96.23% against Skynet955 and 88.33% against Navocado.Its Elo almost equaled dypm, a tree-search-based agent.Source: §3.2 · pp. 7–8, Figure 7 · p. 8, Table 1 · p. 8
• The evaluation also reports that the method was defeated by tree-search-based agents and teams using communication; these cases were out of scope.Table 2 covers win rates against tree-search-based and communicated agents.Source: §3.2 · p. 8, Table 2 · p. 8

• Without communication, the trained agent defeats the top learning agent and the top four rule-based agents in the 2019 competition.Source: §4 · p. 8
• The training system combines curriculum learning with population-based self-play, teaching skills that include bombing walls, collecting items, kicking bombs, and trapping enemies.Source: §4 · p. 8
• Performance-based annealing adjusts dense exploration rewards, while Elo-based matchmaking pairs agents to support incremental self-play learning.Source: §4 · p. 8

• The 11 x 11 board is symmetric along its main diagonal and places four agents at its corners.Source: §A.1 · p. 10, Figure 8 · p. 10
• Bombs explode after 10 timesteps, creating cross-shaped flames that can eliminate agents, wooden walls, and items, and trigger other bombs.Source: §A.1 · p. 10
• Destroyed wooden walls may reveal extra-bomb, increased-range, or kick-ability items, while agents cannot pass through bombs or obstacles.Source: §A.1 · p. 10, Figure 9 · p. 10

• In Free-For-All mode, all agents fight individually, and the last survivor wins.Source: §A.2 · p. 10
• Team mode is a 2-versus-2 contest in which one team must eliminate the other; if both teams retain a survivor at the end, the game is a tie.Source: §A.2 · p. 10
• Team play is partially observable within a 9x9 grid around each agent, and allied agents can communicate by sending a list of two numbers from 1 to 8.Source: §A.2 · p. 10

• The agent’s 2D state features encode spatial information in a 9x9 grid centered on its position, matching its limited observation.Source: B State Representation · p. 11
• Scalar features represent game information such as ammunition, bomb blast strength, and equipped item ability.Source: B State Representation · p. 11
• The example map encodes passages and rigid walls as binary features, represents bomb strength and time-to-explosion, and fills off-board cells with rigid walls.This compact local representation is intended to help the agent focus on nearby objects and speed training.Source: B State Representation · p. 11, Figure 10 · p. 11

C Reward System

• Players must navigate through wooden walls and stones from map corners, so gameplay divides into exploring to reach opponents and combat to eliminate them.Source: C Reward System · p. 12
• The timestep exploration reward encourages bomb placement to uncover items hidden behind wooden walls.Source: C Reward System · p. 12, Table 5 · p. 12
• The end-of-game reward reflects a win, tie, or loss and motivates combat to eliminate opponents.Source: C Reward System · p. 12, Table 5 · p. 12

D Network Architecture and Training Parameters

• Agents use actor-critic Proximal Policy Optimization with four 3x3 convolutional layers, followed by flattened features concatenated with scalar features.Source: D Network Architecture and Training… · p. 12
• The network feeds extracted features into an LSTM that retains information across a sequence of 10 steps.Source: D Network Architecture and Training… · p. 12, Figure 11 · p. 12
• A 128-unit fully connected layer splits into a value head estimating current-state value and a policy head predicting probabilities for six actions.The actions are stay, up, down, left, right, and bomb.Source: D Network Architecture and Training… · p. 12, Figure 11 · p. 12