Source-linked AI summary
Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot
Joel Z. Leibo, Edgar Duéñez-Guzmán, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattie, Igor Mordatch, Thore Graepel
TL;DR
MARL lacks standardized evaluation focused on generalization to novel situations, and creating diverse test environments is labor-intensive. Melting Pot addresses this with a scalable suite and protocol that uses agents to create novel social test scenarios for zero-shot evaluation. Its benchmarks reveal that maximizing collective reward often yields less robust policies than maximizing individual reward in novel social situations.
Problem
MARL lacks standardized evaluation protocols that make like-for-like comparisons and prioritize generalization to novel situations.
Method
Melting Pot combines an evaluation methodology with test scenarios formed by preserving a substrate and substituting an unfamiliar background population.
Results
Maximizing collective reward often produces policies less robust to novel social situations than maximizing individual reward.
Takeaways & Limitations
Melting Pot provides a scalable evaluation suite for generalization to novel social situations and can reveal weaknesses absent from training performance alone.
Takeaways & Limitations
The benchmark omits communication/language, turn-based, and physics-based games from its represented research categories.
Abstract
from arXiv · showhide
Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised-learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap, and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent's behavior constitutes (part of) another agent's environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.
1. Introduction
MARL lacks standardized benchmarks that prioritize generalization, partly because creating diverse test environments is labor-intensive. Melting Pot addresses this by using agents’ behavior to generate novel social test situations and reveals robustness differences among training objectives.
- MARL lacks a broadly accepted benchmark protocol for like-for-like algorithm comparisons and explicit evaluation of generalization.
- Creating reinforcement-learning test environments is labor-intensive because each environment resembles designing and implementing a new computer game.
- Melting Pot uses a methodology and test-environment suite in which agents’ policies help constitute other agents’ evaluation environments.
- The suite spans competitive, common-interest, team-based, and mixed-motivation games, with two to sixteen simultaneous players.
- Maximizing collective reward often produces policies less robust to novel social situations than maximizing individual reward.
2. What does Melting Pot evaluate?
Melting Pot evaluates decentralized agent populations on social competence across varied interdependent situations. Its desiderata include broad performance, generalization to unfamiliar individuals, and universalization of behavior.
- Melting Pot evaluates decentralized populations of simultaneously interacting agents, called multi-agent population learning algorithms.
- Communication, language, turn-based games, and physics-based games are omitted but could be added in future work.
- The motivating application requires agents to function whether their population is a minority or majority among other drivers and agents.
- The evaluation asks agents to perform well across interdependent social situations, generalize to unfamiliar individuals, and pass a universalization test.
- The suite covers self-play and other population-based training approaches, including systems developed for games such as AlphaGo, AlphaStar, and DOTA.
3. Related work
Prior MARL benchmarks target specific domains without measuring generalization or using learned agents to create evaluation tasks. Melting Pot differs by combining held-out-agent evaluation with a broader benchmark methodology.
- Competitive-game research has evaluated generalization against held-out agents, and agent interaction has also been used to drive learning.
- Existing MARL benchmarks for specific domains generally do not measure generalization or use learning agents to produce evaluation tasks.
4. The Melting Pot protocol
The Melting Pot protocol trains populations on substrates, then evaluates them zero-shot in scenarios formed by substituting unfamiliar background populations. It measures focal-agent returns while supporting additional social-impact diagnostics.
- Definitions: A substrate is a partially observable general-sum Markov game with observations, actions, stochastic transitions, and individual rewards.
- Definitions: A population samples individual policies independently on a substrate, and performance is measured by expected per-capita return.
- Testing: A test scenario combines a substrate, focal/background configuration, and background population, reducing the focal perspective to an equivalent focal-player substrate.
- Testing: Resident-mode scenarios test robustness to unfamiliar minorities, whereas visitor-mode scenarios test whether an individual can follow dominant population conventions without retraining.
- Protocol: Melting Pot evaluates trained populations only at test time through zero-shot scenarios that preserve a substrate while replacing its background population.
- Evaluation metrics: The focal-population per-capita return is the primary metric for interpretable comparison in novel social situations, while background-population return can assess practical external impacts.
5. Description of the substrates
Melting Pot substrates cover diverse strategic and social situations, from matrix games and social dilemmas to coordination, competition, cooperation, and resource management.
- 5. Description of the substrates: The suite spans competitive, cooperative, coordination, social-dilemma, territorial, resource-sharing, and task-oriented environments.Its Matrix games use collected items to determine encounters’ rewards according to classic payoff matrices.
- 5. Description of the substrates: Other substrates introduce strategic adaptation through deception, territory control, chemical reaction chains, or tensions between group and individual preferences.Running With Scissors supports countermeasures and feinting, while Allelopathic Harvest combines group benefits with individual color preferences.
- 5.1. Conflicting greater than corresponding interests: Social-dilemma substrates make collective resources vulnerable to individual incentives, requiring restraint, cooperation, or public-good contributions.Examples include harvesting berries while cleaning a river and preserving apple patches from over-harvesting.
- 5.2. Corresponding greater than conflicting interests: Coordination substrates vary whether success depends on shared conventions, risk-sensitive cooperation, role division, or access to complementary tasks.Collaborative Cooking contrasts separated players who must collaborate with a passable kitchen where one player can complete tasks less efficiently.
6. Extending Melting Pot
Melting Pot is designed to scale through modular substrates and a modular process for contributing new scenarios, with scenarios combining physical worlds and agentic background populations.
- 6. Extending Melting Pot: The suite’s expansion strategy combines modular environment components with a modular process for contributing new scenarios.This design supports the long-term goal of assessing most aspects of social intelligence.
- 6. Extending Melting Pot: Each scenario combines a substrate describing the physical world with a background population of reinforcement-learning bots.Substrates emit interaction events that can depend on player identities or locations.
- 6. Extending Melting Pot: Background-population construction proceeds through specification, training, and quality control.The process is explicitly organized as three steps for creating new scenarios.
1. Specification: The designer typically starts with an idea
Background populations are designed from desired behaviors, trained to generalize to focal agents, and screened through interaction-based quality control.
- 1. Specification: The designer typically starts with an idea: Designers specify desired bot behaviors using substrate events and, when needed, hierarchical reinforcement-learning techniques such as reward shaping and option chaining.Event-conditioned behaviors can express rules such as cleaning only when other players are cleaning.
- 1. Specification: The designer typically starts with an idea: Training includes bots likely to develop behaviors resembling the focal agents encountered during testing.Running With Scissors uses specialist rock, paper, and scissors bots alongside free bots receiving the true substrate reward.
- 1. Specification: The designer typically starts with an idea: Quality control evaluates candidate bots over 10–30 episodes against mixtures of familiar and unfamiliar fixed bots.Bots are rejected when they fail to optimize the event they were trained to target.
7. Experiments
Melting Pot evaluates several MARL architectures on zero-shot test scenarios, revealing generalization weaknesses that training performance alone can miss. V-MPO performs best on average, while results expose overfitting, cooperation challenges, and brittleness from specialized policies.
- The benchmark trains A3C, V-MPO, OPRE, and prosocial variants across 21 substrates, using exploiters and random agents as rough performance bounds.
- V-MPO achieves the highest average test performance, followed by OPRE and A3C, with all three performing similarly.
- Performance remains weak on the hardest substrates, including Collaborative Cooking, where no agent exceeds random self-play performance.
- High self-play scores can coexist with low test-scenario scores, showing overfitting when prosocial agents learn cooperation strategies exploitable by defectors.
- Prosocial agents generally underperform selfish counterparts, yet only prosocial architectures learn policies significantly better than random in Clean Up.
- Universalization scenarios reveal that OPRE can perform well elsewhere yet poorly when all agents behave similarly, indicating efficient but non-robust specialization.
8. Conclusion
Melting Pot evaluates generalization to novel social situations and is designed to scale through reinforcement-learning-assisted environment design. Its scenarios target social capabilities including trust, reciprocity, generosity, forgiveness, and deception.
- Melting Pot evaluates multi-agent learning algorithms on generalization to novel social situations.
- The scenarios engage capabilities such as trust, generosity, forgiveness, reciprocity, stubbornness, and deception.
- Reinforcement learning reduced human labor, enabling creation of approximately 85 diverse scenarios.The suite can also be extended by interested researchers.
D. Training setup
The training setup defines common partially observable environments in which agents move, gather resources, and interact through substrate-specific mechanics. These mechanics include inventories, matrix-game rewards, stochastic reactions, resource regrowth, and territory or planting interactions.
- Agents use RGB observations of 88 × 88 × 3 pixels, partial observability, movement actions, and 1000-step episodes.The observation window spans 11×11 sprites and is offset toward the agent's front.
- Agents collect K resource types, and inventories determine mixed strategies in pairwise matrix-game interactions.The zapping agent is the row player and the zapped agent is the column player; rewards are assigned through the payoff matrix.
- Most substrates have eight simultaneous players, requiring agents to choose interaction partners as well as resource strategies.Losing players are removed temporarily after interactions, producing repeated interaction rounds within episodes.
- Substrates include stochastic molecular reactions, resource regrowth, territory painting, berry planting, and group-versus-individual reward tensions.Examples include reaction graphs, locally dependent apple regrowth, color-based territory mechanics, and berry-color incentives.
C. Agent architecture details
The agents retain author-proposed network configurations without hyperparameter tuning, using convolutional, recurrent, and hierarchical components across several algorithms. Evaluation scenarios vary background populations and strategic roles to test coordination, adaptation, and free-riding.
- A3C, V-MPO, and OPRE use convolutional networks, MLPs, and LSTMs, with V-MPO adding PopArt and OPRE adding options and a hierarchical critic.The implementation follows configurations proposed by the respective authors without hyperparameter tuning.
- Puppet combines event-trained basic policies through a scripted high-level policy and a goal-conditioned hierarchical network.Goals are embedded and supplied to an LSTM, which mixes multiple policies through learned coefficients.
- The distributed setup updates parameters on a GPU learner while CPU actors generate experience using periodically cached parameters.
- Scenario populations include residents, visitors, pure rock-paper-scissors bots, mixed strategy bots, and agents with shared Bach or Stravinsky conventions.These settings test recognition of population preferences, best responses, coordination conventions, and adaptation to transient partners.
- Visiting a green-preferring population exposes free riders because focal agents share incentives but some may rely on others to replant.
E.8. Clean Up
Clean Up scenarios test whether focal populations generalize their cooperation, task allocation, and resource-management strategies to unfamiliar background behaviors. They include free riding, turn-taking, conditional cooperation, uncoordinated generalists, and unsustainable harvesting.
- Clean Up: Visiting altruistic cleaners tests whether focal agents avoid redundant cleaning and instead consume apples while the background population maintains the public good.Both free riders and overly altruistic policies receive low scores in this setting.
- Clean Up: Clean Up scenarios test robustness to unfamiliar agents that free ride, change task roles, reciprocate conditionally, or harvest unsustainably.The suite includes both resident and visitor configurations, varying the proportion and behavior of background agents.
- Clean Up: Background populations that clean first or alternate cleaning and eating expose whether focal agents can adapt rather than commit to fixed roles.Turn-taking scenarios begin with different behaviors, and the recommended response is to alternate out of phase with the background population.
- Clean Up: Conditional-cooperation scenarios require focal agents to recognize behavioral thresholds and coordinate enough cleaning to sustain the river.Suspicious reciprocators may require two cleaners initially, while the least-bad solution can involve three agents cleaning together.
- Clean Up: Additional scenarios penalize overspecialized coordination and test whether agents prevent unsustainable background harvesters from depleting resource patches.The latter requires focal agents to zap background agents when they approach patches too closely, with difficulty increasing from two to six zappers.
- Clean Up: Greedy behavior may be individually harmless but catastrophic when widespread, so one test targets policies that exploit such threshold effects.The scenario includes two or six background zappers, making the larger-background condition harder.
E.14. King of the Hill
King of the Hill evaluates learned and ad hoc teamwork against unfamiliar teams whose behaviors depend on training algorithms and reward schemes. The scenarios require focal agents to adapt to opponents, teammates, and coordination conventions without test-time learning.
- King of the Hill: King of the Hill is an eight-player, two-team game where teams compete to control the central hill region.The default reward gives every member of the controlling team one reward per frame when that team controls more than 80% of the hill.
- King of the Hill: Focal teams face unfamiliar opponents trained with V-MPO or A3C under default or shaped rewards, testing whether learned teamwork generalizes across opponent policies.The shaped “zap while in control” reward produces qualitatively different background policies, including spawn camping.
- King of the Hill: Against spawn-camping opponents, focal agents should leave the spawn area, capture the hill, and force opponents into a more vulnerable defensive response.This counter-strategy uses the hill objective to disrupt the opponents’ spawn-location control.
- King of the Hill: Ad hoc teamwork scenarios place one focal agent with three unfamiliar teammates, so success depends on inferring and matching their coordination behavior.The focal agent must tip the balance against another four-agent team by working effectively with teammates from the background population.
- King of the Hill: Background teams trained with shaped rewards may spend most of their time near the hill or guard escape routes, requiring focal agents to follow their conventions and cover neglected sides.For spawn-camping teammates, the focal agent should help guard escape routes while selecting the less-covered side of the map.
- King of the Hill: The broader scenarios also vary reciprocity and player identifiability, testing whether agents exploit cooperators selectively, avoid defectors, and coordinate without persistent identities.Players may be indistinguishable, while background populations range from unconditional cooperators to defectors and grim reciprocators.