Source-linked AI summary
PettingZoo: Gym for Multi-Agent Reinforcement Learning
J. K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, Praveen Ravi
TL;DR
MARL lacks standardized APIs and clear game models that support diverse environments, limiting the interchangeability and accessibility of research. The paper introduces PettingZoo and its AEC games model, showing that AEC and POSG games are representationally equivalent and using case studies to connect clearer modeling with bug prevention. PettingZoo has also seen early adoption, while its sequential per-agent stepping creates performance issues for environments exceeding 10,000 agents.
Problem
MARL lacks a de facto standard API, while common POSG and EFG models can be conceptually unclear for implemented games and difficult to apply across environments.
Method
The paper introduces PettingZoo, a universal Python library and API based on the Agent Environment Cycle games model, and analyzes problems in popular MARL implementations.
Results
AEC games and POSGs are shown to be mutually representable, while case studies report bugs that an AEC-based API would have made impossible.
Takeaways & Limitations
PettingZoo is intended to make MARL research more accessible and is moving toward that goal through early adoption in MARL libraries and courses.
Takeaways & Limitations
Games with significantly more than 10,000 agents can face meaningful performance issues because PettingZoo steps each agent individually.
Abstract
from arXiv · showhide
This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle ("AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning ("MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of games commonly used in MARL and accordingly can promote confusing bugs that are hard to detect, and that the AEC games model addresses these problems.
1 Introduction
MARL research lacks a de facto standard API comparable to Gym, making code reuse, comparison, and library development difficult. PettingZoo addresses this gap with a universal API based on the AEC games model, which is presented as a clearer fit for implemented multi-agent games.
- Motivation: MARL lacks a de facto standard API, unlike single-agent reinforcement learning’s OpenAI Gym.This absence makes interfacing agents with environments less standardized across research implementations.
- Motivation: The lack of standardization makes reusing learning code for new purposes require substantial effort and prevents more thorough comparisons.The paper contrasts 669 Gym-dependent packages with only five MARL libraries having large user bases.
- Problem: Existing MARL APIs largely inherit POSG or EFG designs, which the authors found conceptually unclear for multi-agent games implemented in code.The paper argues these models cannot cleanly support all types of multi-agent environments.
- Contribution: PettingZoo introduces the AEC games model as the basis for a new formal model and API for general MARL environments.The authors argue that AEC games are a better conceptual fit for games implemented in code.
- Contribution: The paper proves equivalence between AEC games and POSGs and uses two case studies to show how confusing game models can permit long-undetected bugs.The reported bugs would have been made impossible by an AEC-based API.
2 Background and Related Works
MARL APIs commonly adapt simultaneous POSG stepping or complex EFG representations, but both approaches have important conceptual and practical limitations. These limitations include awkward handling of turn-taking, changing agent populations, rewards, complexity, and continuous actions.
- Gym: Gym is a simple, influential API based on the POMDP conceptualization of reinforcement learning.Its simplicity and conceptual clarity helped make it a de facto standard and a basis for later APIs.
- POSGs and RLlib: POSGs let all agents step, observe, and receive rewards together, translating naturally into Gym-like lists or dictionaries.This design made it easier to apply single-agent reinforcement learning methods to multi-agent settings.
- POSGs and RLlib: Simultaneous POSG APIs require dummy actions for strictly turn-based games and awkwardly handle agent death or creation.Learning code must cope with lists that suddenly change sizes when the agent population changes.
- EFGs and OpenSpiel: EFGs represent every possible action sequence as a tree and use a Nature player to model stochastic events.They have supported theory-of-mind problems through game-theoretic analysis and tree search.
- EFGs and OpenSpiel: EFG-based APIs are complex for beginners, formally place rewards at game ends, and OpenSpiel does not handle continuous actions.The paper identifies the continuous-action limitation as a choice not inherent to the EFG model.
- Alternative APIs: Using the single-agent Gym API for turn-based games does not reasonably scale beyond two agents because of changing order, death, and creation.The approach alternates which agent is controlled by the environment.
3 PettingZoo Design Goals
PettingZoo’s design goals combine Gym-like simplicity and familiarity with broad support for multi-agent environments. The API is intended to include diverse reference implementations while accommodating changing agent populations and future research needs.
- Gym compatibility: PettingZoo aims to make its API look and feel like Gym while remaining Pythonic and simple.The design reuses Gym’s established interface metaphors to help existing users understand PettingZoo quickly.
- Reference environments: The library includes numerous reference implementations of games alongside the main package.A broad environment collection is treated as important for attracting users and beginning standardization.
- Universal design: A universal MARL API must support large agent populations, agent death and creation, variable episode participants, and low-level features.The paper identifies these as technically difficult cases requiring careful consideration.
- Universal design: The design also prioritizes beginner accessibility and the ability to change the API if research directions shift substantially.These are described as softer goals for universal design.
4 Case Studies of Problems With The POSG Model in MARL
The paper examines how POSG-based MARL APIs obscure reward origins and execution order, producing confusing bugs in practical environments. Case studies show that sequential modeling makes these issues easier to identify and prevent.
- 4.1 POSGs Don’t Allow Access To Information You Should Have: POSG-based APIs combine rewards from multiple sources, obscuring their origins and making source attribution difficult for learning and debugging.Representing source-specific rewards would require a less convenient 2D array rather than the standard list.
- 4.1 POSGs Don’t Allow Access To Information You Should Have: In pursuit, evaders move after pursuers, so capture and rewards depend on the subsequent evader movement rather than the pursuers’ preceding actions alone.The environment awards each contributing pursuer 5 when an evader is surrounded, but the ordering makes expected captures nontrivial.
- 4.2 POSGs Based APIs Are Not Conceptually Clear For Games Implemented In Code: Simultaneous-action abstractions can create race conditions when conflicting actions require environment tie-breaking.The paper calls collision handling that resolves these conflicts “tie-breaking.”
- 4.2 POSGs Based APIs Are Not Conceptually Clear For Games Implemented In Code: If tie-breaking favors Alice, Bob may act on an outdated observation, making outcomes depend on the environment’s internal action-resolution order.The paper presents two agents contesting the same square as a concrete example.
- 4.2 POSGs Based APIs Are Not Conceptually Clear For Games Implemented In Code: Sequential APIs mitigate these race conditions by returning new observations after each agent acts.The paper also reports a related imperfect tie-breaking bug in the social sequential dilemma environments.
5 The Agent Environment Cycle Games Model
The Agent Environment Cycle model represents multi-agent interaction as sequential agent and environment steps. The paper argues that this model better matches implemented games while retaining the expressive power of POSGs.
- AEC games model: In AEC games, agents sequentially receive observations, take actions, emit rewards, and determine which agent acts next.The model is described as a sequentially stepping form of the POSG model.
- Benefits: Sequential modeling clarifies reward attribution, prevents confusing race conditions, and more closely matches how computer games execute in code.These are listed as direct benefits of modeling environments sequentially for APIs.
- Benefits: AEC games formally support rewards after every step, unlike the usual end-of-game reward structure of Extensive Form Games.Frequent rewards are required in reinforcement learning.
- Benefits: The model is intended to be a simple mental model for beginners and handles agent death or creation without constantly changing lists in learning code.The paper also contrasts sequential stepping with simultaneous stepping that requires difficult no-op actions.
- Expressive power: For every AEC game an equivalent POSG exists, and for every POSG an equivalent AEC game exists.The paper uses this equivalence to state that AEC is as powerful as the common POSG model.
- Environment agent: The AEC model includes an environment agent that represents environment updates and reactions to submitted agent actions.Chess transitions illustrate this agent-environment cycle.
6 API Design
PettingZoo preserves much of Gym’s API while adding iteration and state-management mechanisms for sequential multi-agent interaction. Its design exposes both simple workflows and lower-level information for varied MARL experiments.
- Gym compatibility: PettingZoo retains Gym-like action stepping, observations, rewards, termination flags, spaces, rendering, closing, and reset behavior.The API’s main differences are the last and agent_iter methods and their iteration logic.
- Iteration: The agent_iter generator returns the next acting agent and abstracts changing agent order, agent generation, and agent death.Combined with one-agent-at-a-time action, this supports diverse multi-agent environments.
- Iteration: The last method supplies an agent’s observation, reward, done, and info after other agents’ actions determine them.This reflects that other agents are part of the environment from one agent’s perspective.
- State access: Agent-keyed rewards, dones, and infos remain accessible throughout a trajectory, while action and observation spaces are retrieved per agent.The rewards are instantaneous values resulting from the most recent action.
- Extensibility: The API exposes lower-level information so researchers can run unusual experiments and access other agents’ rewards and observations.This supports universal use in an incompletely explored research field.
- Agent populations: PettingZoo supports variable agent populations through optional possible_agents metadata and runtime discovery when arbitrary agents are generated.After reset, agents lists those currently present.
7 Default Environments
PettingZoo includes a broad collection of default multi-agent environments intended to support adoption and research reuse. The collection combines cleaned existing benchmarks with additional Atari, Butterfly, board, and card-game environments.
- Environment collection: PettingZoo includes 63 environments, similar to Gym’s default environment collection.The included classes span several established and newly added environment families.
- Environment collection: MPE, MAgent, and SISL environments comprised half of the collection and required substantial maintenance because their prior code was unmaintained and difficult to install.The paper characterizes this earlier code as “research grade.”
- Environment collection: The library also includes multiplayer Atari games, original Butterfly environments, and popular classic board and card games.All default environments are surveyed in depth in Appendix B.
8 Adoption
PettingZoo has gained early adoption across numerous multi-agent learning libraries, including established and emerging frameworks.
- PettingZoo is supported by numerous learning libraries, including Autonomous Learning Library, AI-Traineree, PyMARL, RLlib, Stable Baselines, CleanRL, and Tianshou.
9 Conclusion
PettingZoo provides a unified Python library and the AEC games model for making multi-agent reinforcement learning research more accessible and reproducible. The authors report early adoption while acknowledging scalability limits for games with far more than 10,000 agents.
- PettingZoo packages diverse multi-agent reinforcement learning environments under one simple Python API and introduces the AEC games model.
- The library aims to make MARL research more interchangeable, accessible, and reproducible, similar to Gym's role in single-agent reinforcement learning.
- Early adoption by numerous MARL libraries and courses indicates progress toward making the field accessible to university-scale research and non-experts.
- Games with significantly more than 10,000 agents can face meaningful performance issues because the API steps each agent individually.The authors identify true parallel support, likely outside Python, as the relevant solution and regard this as practically acceptable.
- Future work includes adding environments, enabling agents from different researchers to compete, and developing procedurally generated environments for testing generalization.
A.1 Race Conditions in Sequential Social Dilemma Games
The case study shows how sequential action resolution can create agent-order-dependent outcomes and incorrect observations in multi-agent environments. The paper also reports that reward pruning improved pursuit training across two algorithms.
- Race conditions: Sequentially resolving agents' actions in a fixed order can make identical placements and actions produce different outcomes when agent labels are switched.In the cleaning example, beam resolution order determines whether the far river tile is reached.
- Race conditions: The second agent may act on an inherently wrong observation because the first agent's action has already changed the environment state.
- Race conditions: AEC games-based APIs prevent this class of race-condition bug, which is easy to introduce with parallel action-based APIs and had remained unnoticed for years in this instance.
- Reward pruning: 22.03% more total reward on average was achieved with reward pruning for parameter-shared Ape-X DQN in pursuit.The comparison used four training runs with and without reward pruning.
- Reward pruning: 16.12% more total reward on average was achieved with reward pruning for PPO in pursuit.
- Environment evaluation: Baseline experiments learned optimal policies in Pistonball and Cooperative Pong, while learning reasonably in Knights Archers Zombies and Prospector without reaching optimal policies.The reported plots summarize 10 training runs using tuned hyperparameters.
D.1 POSGs are Equivalent to AEC Games
The paper shows that AEC games can represent simultaneous actions while preserving the POSG’s state distributions, observations, and rewards. It also establishes equivalence in the reverse direction for AEC games, with deterministic rewards handled explicitly and stochastic rewards requiring additional construction.
- AEC games capture complex turn orders through a stochastic next-agent function and can also theoretically represent simultaneous-action games.The construction queues each agent’s action one at a time, then resolves the joint action through an environment step.
- Every N + 1 AEC steps matches one POSG step in state-distribution probability, observations, and agent rewards.The proof establishes identical state distributions, matching observation distributions, and equal total rewards when agents select the corresponding actions.
- The equivalence construction resolves joint actions simultaneously through the environment step after agents lock in their actions sequentially.This separates action production from the stochastic state transition and reward generation without changing the resulting joint outcome.
- Every AEC game has an equivalent POSG, while the presented reverse construction explicitly converts deterministic-reward AEC games and notes that stochastic rewards require a non-obvious extension.The POSG state can encode the acting agent, and the general theorem’s construction also stores reward information in the state.