Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

• MARL lacks a broadly accepted benchmark test set, and the field has not yet made generalization its primary evaluation goal.This limits like-for-like comparisons; creating test environments is also labor-intensive.Source: §1 · p. 1
• Melting Pot uses pre-trained background populations held out from training to create diverse tests, because one agent’s policy changes the environment experienced by others.Source: §1 · pp. 1–2
• Its protocol compares algorithms at test time in zero-shot scenarios that retain a familiar substrate but substitute an unfamiliar background population.Training-time access to each test substrate is allowed, and the protocol is mostly agnostic to training method.Source: §1 · p. 2
• The suite spans competitive, common-interest, team-competitive, and mixed-motivation games, with two to sixteen simultaneous players.Most substrates have around eight players.Source: §1 · p. 2
• Policies maximizing collective reward are often less robust to novel social situations than policies maximizing individual reward.Source: §1 · p. 2

What does Melting Pot evaluate?

• MAPLAs are evaluated on performing across interdependent social situations, interacting effectively with unfamiliar individuals, and passing the universalization test of asking “what if everyone behaved like that?”.The unfamiliar individuals may include people not seen during training.Source: §2 · p. 2
• The scenarios omit communication and language, turn-based games, and games involving physics.The authors note these topics could be added in the future; examples include communication and language (Lazaridou & Baroni (2020); Lowe et al. (2019); Mordatch & Abbeel (2018)), turn-based games (Lanctot et al. (2019)), and physics (Liu et al. (2018)).Source: §2 · p. 2
• The motivating self-driving-car example requires a decentralized population to work whether it is a minority or majority, even as its share changes over time.The population may interact with human drivers and vehicles from competing manufacturers.Source: §2 · p. 2
• MAPLAs encompass most multi-agent reinforcement-learning approaches that produce populations, including self-play schemes and algorithms such as MADDPG, LOLA, and PSRO.Examples of self-play include AlphaGo, AlphaZero, FTW, hide and seek, and AlphaStar.Source: §2 · p. 2

3. Related work

• Agent-generated tasks appeared in competitive games (Berner et al., 2019), while interaction-driven learning did not create generalization benchmarks.Zero-shot transfer to new co-players in coordination games was investigated by Hu et al. (2020).Source: §3 · p. 2
• Unlike domain-specific MARL benchmarks and strategy competitions, Melting Pot explicitly benchmarks generalization across a broader range of multi-agent interactions.Domain benchmarks include Song et al. (2020b) and Lowe et al. (2017); Perez-Liebana et al. (2019) brought strategy competitions to MARL.Source: §3 · p. 3

4.1. Definitions

• A substrate is a partially observable general-sum Markov game where agents receive individual rewards and must explore to discover rules that are not given to them.It is modeled with states, observations, actions, a stochastic transition function, and individual reward functions.Source: §4.1 · p. 3

4.2. Testing

• A test scenario reduces a substrate to its focal players by marginalizing over background-player policies sampled from a background population.Resident scenarios test robustness to unfamiliar minorities; visitor scenarios test adaptation to dominant-group conventions without retraining; universalization repeats one sampled policy across all players to diagnose failures.Source: §4.2 · pp. 3–4

4.3. Training

• MAPLA trains a population with unlimited access to the raw substrate, but not background populations, so test performance requires zero-shot generalization to unseen scenarios.Independent policy sampling also favors generalists and role division that self-organizes at test time.Source: §4.3 · p. 4

4.4. Secondary evaluation metrics

• The focal-population per-capita return is the primary metric for comparing learning algorithms in novel social situations.The protocol also supports practical assessments using secondary metrics on the background population.Source: §4.4 · p. 4
• Background-population per-capita returns can reveal negative externalities, while inequality in individual returns can show whether resulting benefits or harms are fairly shared.These secondary measures address practical impacts beyond the focal population's primary evaluation score.Source: §4.4 · p. 4

5. Description of the substrates

• Across the Matrix games, players collect choice-representing items that determine rewards when they encounter others.The items can represent choices such as cooperation or defection, or rock, paper, and scissors; encounters use a classic matrix-game payoff pattern.Source: §5 · p. 4

5.1. Conflicting greater than corresponding interests

• Other conflicting-interest substrates range from strategic contests and team objectives to dilemmas over territory and cooperation.Running with Scissors in the Matrix supports feints and countermeasures, and its Arena version has eight players. Capture the Flag and King of the Hill require team objectives, including ad-hoc teamwork scenarios; Prisoner’s Dilemma, Chicken, and Territory test tensions between individual and collective outcomes.Source: §5.1 · p. 5
• Resource-use scenarios test whether agents balance individual rewards against shared resources and collective restraint.In Clean Up, harvesting berries competes with cleaning a river whose pollution suppresses berry growth (Hughes et al., 2018). Commons Harvest varies the challenge from restraining harvest in open patches to defending rooms alone or in pairs.Source: §5.1 · p. 5

5.2. Corresponding greater than conflicting interests

• Coordination substrates vary whether success requires interdependence, convention matching, or choosing among unequal outcomes.In Collaborative Cooking: Impassable, an impassable counter prevents any player from completing recipes alone; Passable allows unilateral but less efficient completion (Carroll et al., 2019; Wang et al., 2020). Pure Coordination rewards matching any color, while Rationalizable Coordination assigns different values to choices.Source: §5.2 · p. 5
• Bach or Stravinsky and Stag Hunt examine coordination shaped by unfairness, stubbornness, or individual risk.The Stag Hunt in the Matrix engages concepts similar to Peysakhovich & Lerer (2017)’s Markov Stag Hunt, with added complexity including eight players rather than two.Source: §5.2 · pp. 5–6
• Allelopathic Harvest and Chemistry combine coordination with competing incentives or the risk of exclusion.Allelopathic Harvest pairs group tensions over berry colors with a free-rider problem (K¨oster et al., 2020). Chemistry scenarios require transporting molecules to sustain reactions, including branched-chain alternation or two cyclic networks, while efficient coordination can exclude players.Source: §5.2 · p. 6

6. Extending Melting Pot

• Melting Pot’s expansion strategy adds a modular contribution process alongside modular environment components, aiming to make the platform scalable over time.The long-term goal is a comprehensive platform for assessing most aspects of social intelligence.Source: §6 · p. 6
• A scenario combines a substrate built on DMLab2D with a background population of RL agents called bots; substrates emit interaction events that can depend on player identity or location.The substrate uses an entity-component system approach similar to modern game engines such as Unity (Unity Technologies, 2020); DMLab2D is cited as Beattie et al. (2020).Source: §6 · p. 6
• The process for creating background populations has three steps: specification, training, and quality control.The paper points to Fig. 3 for this process.Source: §6 · p. 6

Specification

• Privileged substrate events make it easier to specify reward functions for desired behavior than for focal agents learning from pixels and final reward.When one reward function is insufficient, designers train event-rewarded bots and chain them with Python rules; in Clean Up, a bot cleans only when others are cleaning.Source: §1 · p. 7

Training

• Background populations must generalize to focal agents, so training includes bots likely to develop test-time-like behavior, such as rock, paper, and scissors specialists and free-reward bots.The Running With Scissors in the Matrix example trains the specialists alongside free bots receiving the true substrate reward.Source: §1 · p. 7

Quality control

• Candidate bots undergo 10–30 interaction episodes with fixed familiar and unfamiliar bots, and are rejected if they fail to optimize for their target event.Source: §1 · p. 7

Experimental setup

• The experiments trained A3C, V-MPO, and OPRE, plus prosocial reward-sharing variants, across 21 substrate runs and evaluated test returns on a normalized 0–1 scale.Testing used a uniform distribution over each substrate’s trained agents; random agents and scenario-specific exploiters provided rough lower and upper performance bounds, though exploiters did not always score highest.Source: §7 · pp. 7–8

Generalization

• High self-play scores did not ensure test success: prosocial OPRE and A3C became exploitable after learning cooperation strategies during self-play.This pattern appeared in Bach or Stravinsky in the Matrix and Stag Hunt in the Matrix, where their test-scenario scores were closer to random despite self-play scores similar to regular counterparts.Source: Figure 4 · p. 8, §7 · p. 9

Performance limits

• Collaborative Cooking proved unlearnable for these agents, while Arena Running with Scissors and Capture The Flag agents fell far short of exploiter performance.Exploiters also failed in some scenarios: strong scenario bots were hard to exploit, while selfish reward maximization failed where cooperation was required.Source: §7 · pp. 8–9

Overall results

• V-MPO ranked first on average, followed by OPRE and A3C; all three performed similarly in mixed-motivation games, while V-MPO outperformed in competitive games.The authors caution that agent rankings depend on how test scenarios contribute to the overall score.Source: §7 · p. 8, Figure 5 · p. 9

Prosocial performance

• Prosocial agents generally underperformed selfish counterparts, yet only prosocial architectures learned policies significantly better than random in Clean Up.Per-capita optimization can complicate credit assignment and produce lazy-agent problems; the authors suggest strong performance may require contingently balancing selfishness and prosociality.Source: §7 · p. 9

Universalization

• In Chemistry: Metabolic Cycles, OPRE’s low universalization scores despite strong self-play and other-scenario performance indicated specialized policies that reduced population robustness.Such division of labour can improve efficiency while making populations less robust.Source: §7 · p. 9

8. Conclusion

• Melting Pot evaluates generalization to novel social situations and engages with concepts long neglected by AI research.Source: §8 · p. 9
• The suite poses problems that seem to require agents to understand trust, generosity, forgiveness, reciprocity, stubbornness, and deception.Source: §8 · p. 9
• Reinforcement learning reduces human labor in environment design, enabling rapid creation of a diverse set of approximately 85 scenarios.Source: §8 · p. 9
• Because Melting Pot is openly released, interested researchers can extend it; the open-source project is available at https://github.com/deepmind/meltingpot.Source: §8 · p. 9
• Melting Pot can improve over time by incorporating the latest agent technology into new background populations and test scenarios.The paper says test-scenario bots' effectiveness also advances with improvements in learning systems.Source: §8 · p. 9

A. Secondary evaluation metrics

• Background per-capita returns are higher on average with prosocial variants, which usually do not reduce positive-income equality; exceptions include poor collaborative-substrate performance and mildly increased inequality in Chicken in the Matrix.In Chicken in the Matrix, prosocial variants greatly improved background per-capita return, likely by increasing cooperators, but not every background player could partner with or exploit a prosocial player.Source: A. Secondary evaluation metrics · p. 10, Figure A.1 · p. 10, Figure A.2 · p. 10

B.1. Common to all substrates

• Substrates share 1000-step episodes, 11 × 11 sprite observations represented as 88 × 88 × 3 RGB inputs, and six movement actions.The observation window is offset to show nine rows ahead, one behind, and five columns to each side.Source: §B.1 · p. 11

B.2.1. * IN THE MATRIX

• In * IN THE MATRIX, agents collect resources that determine mixed strategies in pairwise matrix-game interactions, requiring them to choose both what to collect and whom to interact with.Most substrates have eight simultaneous players; losing interactions typically remove the lower-reward player for 200 steps, with ties favoring the zapping agent.Source: §B.2.1 · p. 11

B.2.2. CHEMISTRY

• In Chemistry, agents carry one molecule and earn rewards when reactions involving it occur; reaction rates can vary with whether molecules are in an agent’s inventory.Some reactions, such as metabolizing food, occur faster in inventory, where they generate reward, than outside it.Source: §B.2.2 · p. 11

B.2.3. COMMONS HARVEST

• In Commons Harvest, local apple regrowth depends on nearby apples, so consuming every apple in an isolated patch can permanently destroy it and creates a tragedy of the commons.Regrowth probability is zero when no apples remain within the specified neighborhood, while multi-agent competition encourages consuming the last apple before another agent does.Source: §B.2.3 · p. 12

B.2.4. THE PAINTING MECHANIC IN KING OF THE HILL AND CAPTURE THE FLAG

• In King of the Hill and Capture the Flag, painting ground in a team’s color restores health up to three, while opposing-color ground lowers health to one and blocks movement.Health recovers at a fixed stochastic rate of 0.05 per frame.Source: §B.2.4 · p. 12

B.2.5. THE CLAIMING MECHANIC IN TERRITORY: OPEN AND TERRITORY: ROOMS

• In Territory: Open and Territory: Rooms, players claim resources by touch or beam; held claims activate after 100 timesteps and then reward their claimant stochastically.Zapped players are permanently removed, and their claimed resources return to the unclaimed state.Source: §B.2.5 · pp. 12–13

B.3. Details of specific substrates

• Specific substrates instantiate distinct dilemmas: Allelopathic Harvest rewards planting shared berry colors, Clean Up trades apple gathering against river cleaning, and Collaborative Cooking requires recipe coordination.Capture the Flag adds team territory expansion and flag capture, while Chemistry substrates require coordination to sustain reaction chains or cycles.Source: §B.3 · pp. 13–14
• The substrate description notes that preventing invasion can convert a difficult multi-agent problem into independent single-agent problems that are easier to solve.Source: §B.3 · p. 15
• Commons Harvest: Open requires restraint to preserve apple patches, while Commons Harvest: Partnership tests trust, joint defense, and sustainable harvesting of shared resources.Partnership requires two players to defend an apple room; the passage describes successful play as recognizing aligned interests with a partner.Source: §B.3 · p. 15
• King of the Hill uses Capture the Flag’s painting and zapping dynamics, but teams compete to color at least 80% of the central hill.Indicator tiles signal whether red or blue controls the hill, or whether neither team does.Source: §B.3 · p. 15
• Matrix substrates test dilemmas, coordination, and strategic adaptation through resource-collection encounters, including Prisoner’s Dilemma, Stag Hunt, coordination variants, and Running with Scissors.Pure Coordination requires all eight players to converge on one resource color; Rationalizable Coordination gives colors different intrinsic values, while Running with Scissors allows observation and countermeasures.Source: §B.3 · p. 15
• Territory: Rooms makes peaceful partition easy initially, but greedy invasion can prompt reciprocal incursions, conflict, and deadweight loss; Territory: Open also permits peaceful partition or irreversible destruction.In Territory: Open, zapping removes an agent for the episode and returns its claimed territory to neutral; in Rooms, walls can be broken and destroyed resources are lost forever.Source: §B.3 · pp. 15–16

C. Agent architecture details

• Agents follow authors’ configurations without hyperparameter tuning; Puppet combines event-trained goal policies using a scripted high-level controller and a learned goal-conditioned policy mixture.Puppet embeds one-hot goals for its LSTM and trains with goals switching at random intervals.Source: C. Agent architecture details · p. 16

D. Training setup

• Training uses a distributed learner on a GPU and CPU actors that generate experience with periodically cached learner parameters.Source: D. Training setup · p. 16

E.1. Allelopathic Harvest

• Allelopathic Harvest tests whether residents suppress a green-replanting visitor and whether red-preferring visitors adopt a green population’s convention.The self-play test also exposes agents that free-ride on others’ replanting.Source: §E.1 · pp. 16–17

E.2. Arena Running With Scissors in The Matrix

• Arena Running With Scissors tests whether agents identify partners’ pure or mixed rock-paper-scissors strategies and select the countering resource while avoiding other focal agents.A gullible-bot variant makes focal agents vulnerable to feinting.Source: §E.2 · p. 17

E.3. Bach or Stravinsky in The Matrix

• Bach or Stravinsky tests whether visitors coordinate by adopting the convention favored by a pure Bach or pure Stravinsky population, regardless of player role.Source: §E.3 · p. 17

E.4. Capture The Flag

• Capture The Flag tests competition against unfamiliar A3C or V-MPO teams and ad hoc teamwork with unfamiliar teammates against unfamiliar opponents.Focal teams have familiar teammates in the first two scenarios; visitor-mode scenarios instead require a lone focal agent to coordinate with unfamiliar teammates.Source: §E.4 · p. 18

E.5. Chemistry: Branched Chain Reaction / E.6. Chemistry: Metabolic Cycles

• The chemistry tests assess whether agents complement partners specialized in one reaction branch or cycle and whether specialized policies can adapt when the task requires another role.Branched Chain Reaction tests sustainable combinations of agents carrying suitable molecules; Metabolic Cycles tests balancing the two cycles or taking the needed task when visiting a population.Source: §E.5 · p. 18, §E.6 · p. 18

E.7. Chicken in the Matrix

• Dove-biased policies perform relatively well in Chicken in the Matrix, while hawk-based policies perform exceptionally badly.Source: §E.7 · p. 19

E.8. Clean Up

• Clean Up tests robustness to free riders and turn-taking partners, exposing brittle role assignments and grim-trigger retaliation; both free-riding and overly altruistic policies score poorly.With turn-taking bots, focal agents should alternate cleaning and eating out of phase rather than fix their behavior from the episode opening.Source: §E.8 · pp. 19–20
• Clean Up’s conditional cooperators require focal agents to maintain enough cleaners, sometimes accepting three cleaners when two would be optimal to prevent cooperation from collapsing.A lone suspicious reciprocator requires two focal cleaners to join; with two such bots, focal agents must sustain cleaning without gaps.Source: §E.8 · p. 20

E.9. Collaborative Cooking: Impassable

• Collaborative Cooking: Impassable tests whether a focal agent can join unfamiliar players’ coordination without obstructing them; the scenario penalizes overspecialization.A resident-mode variant replaces one familiar agent with an unfamiliar background agent who may not know the group’s conventions.Source: §E.9 · p. 20

E.10. Collaborative Cooking: Passable

• Collaborative Cooking: Passable tests whether agents act independently among uncoordinated generalists yet still cooperate despite a generalist who may get in the way.Both visitor- and resident-mode scenarios penalize overspecialization.Source: §E.10 · p. 20

E.11. Commons Harvest: Open

• Commons Harvest: Open tests sustainability under zappers: agents must deter unsustainable harvesting, revealing policies that exploit thresholds where individual greed seems harmless but widespread greed is catastrophic.The harder variant adds six unsustainable, zap-prone bots rather than two.Source: §E.11 · p. 21

E.12. Commons Harvest: Closed

• Commons Harvest: Closed tests whether agents secure separate rooms by controlling entrances, making harvesting less interdependent and incentivizing sustainable use of each agent’s territory.The scenarios include resident and visitor modes, with variants increasing the number of background zappers.Source: §E.12 · p. 21

E.13. Commons Harvest: Partnership

• Partnership bots learn to avoid invisible midline tiles, creating partners that guard one entrance and harvest one side so focal agents can be tested on trust and cooperation.The scenarios vary whether focal agents meet good partners or zappers and how much cooperation with unfamiliar agents is needed.Source: §E.13 · pp. 21–22

E.14. King of the Hill

• King of the Hill pits two four-player teams against each other, rewarding hill control by default; a training-only alternative rewards zapping opponents while in control.The alternative reward produces qualitatively different policies that still function under the default test-time reward.Source: §E.14 · p. 22
• King of the Hill tests full-team and ad hoc teamwork against default and shaped bots, including spawn campers whose counter is to capture the hill or guard escape routes.With default bots, an ad hoc focal agent should coordinate around the hill and guard a less-defended entrance; against shaped V-MPO bots, it should help guard opponents’ escape routes.Source: §E.14 · pp. 22–23

E.15. Prisoner’s Dilemma in the Matrix

• Prisoner’s Dilemma scenarios test whether agents exploit cooperators, preserve cooperation with peers, manipulate gullible partners, or time defection against reciprocators.Against cooperators, the suggested strategy is to defect selectively; against grim reciprocators, cooperate until near episode end before defecting.Source: §E.15 · pp. 23–24

E.16. Pure Coordination in the Matrix

• Pure Coordination scenarios ask agents to infer partners’ resource choices and coordinate with familiar players, while avoiding crowding that leaves some agents unable to collect a resource.The tests include unfamiliar specialists and uncoordinated strangers; choosing partners based on observed resource choices can support coordination.Source: §E.16 · p. 24

E.17. Rationalizable Coordination in the Matrix

• Rationalizable Coordination scenarios make resource C the rational coordination choice, testing whether agents avoid following specialists in A or B and select compatible partners.The scenarios include unfamiliar specialists and uncoordinated strangers; resource A and B are less rational choices than C.Source: §E.17 · pp. 24–25

E.18. Running With Scissors in the Matrix

• Running With Scissors tests whether agents can feint against a counter-strategy opponent or scout a pure-strategy opponent and choose its counter.Additional scenarios fix the opponent to rock, paper, or scissors.Source: §E.18 · p. 25

E.19. Stag Hunt in the Matrix

• Stag-biased agents do better than hare-biased agents if stag resources last; tests also probe matching stag or hare populations and avoiding reciprocator triggers.Stag resources can run out if agents collect more than they need; reciprocators switch to hare after a partner plays hare.Source: §E.19 · p. 25

E.20. Territory: Open

• Territory: Open includes a scenario where focal agents are residents and a shaped bot visits them.The supplied passage identifies this as SC 0 but provides no further scenario details.Source: §E.20 · p. 25
• In Territory: Open, resident agents may zap an active background bot early, while a visitor may defend territory and zap approaching bots.The resident bot claims territory but rarely zaps; early removal can let the focal population fairly divide resources according to its learned convention. Visitors may instead join background bots in claiming and reclaiming territory.Source: §E.20 · p. 25

• Overly aggressive policies perform very poorly in both Territory: Open and Territory: Rooms tests.Source: §E.20 · p. 25, §E.21 · p. 26

E.21. Territory: Rooms

• In Territory: Rooms, agents must defend against aggressive attackers; a visitor can claim territory left vacant after background bots fight.The background population is aggressive and destructive, and a focal population that retains self-defense is expected to do better than one that has forgotten it. A visitor must remain alert because attacks can permanently destroy resources.Source: §E.21 · pp. 25–26

F. Raw performance scores

• The raw performance scores are focal per-capita returns.Source: Table 1 · pp. 27–29