PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks

Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V. Albrecht

arXiv:2006.07869v4cs.LGcs.AIcs.MAstat.ML

TL;DR

MARL lacks established benchmark tasks, protocols, and metrics for comparing modern algorithms. The paper evaluates three algorithm classes across 25 cooperative tasks, releases EPyMARL and two environments, and finds that algorithm effectiveness depends on coordination, observability, and reward structure.

  • Problem

    MARL lacks established benchmark tasks, evaluation protocols, and metrics for comparing recent algorithms.

  • Method

    The study systematically compares independent learning, centralised policy gradient, and value decomposition across 25 cooperative tasks using EPyMARL.

  • Results

    Centralised critics help with substantial coordination under partial observability, while independent learning is competitive in fully observable tasks and state-action critics perform poorly in sparse-reward RWARE.

  • Takeaways & Limitations

    The benchmark identifies task properties associated with different learning approaches and provides reusable evaluation code and sparse-reward environments.

  • Takeaways & Limitations

    The benchmark is limited to cooperative tasks, with some evaluated settings using pretrained policies for adversary and prey agents.

Abstract

from arXiv · show

Multi-agent deep reinforcement learning (MARL) suffers from a lack of commonly-used evaluation tasks and criteria, making comparisons between approaches difficult. In this work, we provide a systematic evaluation and comparison of three different classes of MARL algorithms (independent learning, centralised multi-agent policy gradient, value decomposition) in a diverse range of cooperative multi-agent learning tasks. Our experiments serve as a reference for the expected performance of algorithms across different learning tasks, and we provide insights regarding the effectiveness of different learning approaches. We open-source EPyMARL, which extends the PyMARL codebase to include additional algorithms and allow for flexible configuration of algorithm implementation details such as parameter sharing. Finally, we open-source two environments for multi-agent research which focus on coordination under sparse rewards.

1 Introduction

The paper addresses difficult MARL comparisons by benchmarking nine algorithms across cooperative tasks, while releasing EPyMARL and two sparse-reward environments to support consistent evaluation.

  • MARL comparison is difficult because established benchmark tasks, evaluation protocols, and metrics are lacking.
  • The study compares independent learning, centralised multi-agent policy gradient, and value decomposition algorithms.
  • Nine MARL algorithms are evaluated across 25 cooperative learning tasks spanning two matrix games and four multi-agent environments.Hyperparameters are separately grid-searched, and maximum and average evaluation returns are reported.
  • EPyMARL extends PyMARL with additional algorithms and configurable implementation details such as parameter sharing.
  • LBF and RWARE are open-sourced to test coordination under sparse rewards and partial observability.

2 Algorithms

The benchmark covers independent learning, centralised training with decentralised execution, and value decomposition, differing in information sharing, critic design, and joint-value representation.

  • Independent Learning (IL): Independent learning trains each agent separately while treating other agents as part of the environment.
  • Independent Learning (IL): IQL conditions each decentralised state-action value function on an agent’s local observation-action history.
  • Independent Learning (IL): IA2C uses separate actor and critic networks conditioned on each agent’s local observations, actions, and rewards.
  • Independent Learning (IL): IPPO extends PPO for decentralised multi-agent training and constrains policy changes, enabling multiple update epochs per trajectory batch.
  • Centralised Training Decentralised Execution (CTDE): CTDE shares information during training but conditions execution policies only on agents’ local observations.
  • Centralised Training Decentralised Execution (CTDE): Centralised policy-gradient methods pair decentralised actors with critics optimised from shared agent information.
  • Centralised Training Decentralised Execution (CTDE): MADDPG trains critics on joint observations and actions, while requiring continuous differentiable actions unless discrete actions use Gumbel-Softmax.
  • Centralised Training Decentralised Execution (CTDE): COMA modifies actor advantages with counterfactual baselines for cooperative credit assignment.

3 Multi-Agent Environments

The benchmark defines 25 fully cooperative tasks across varied observability, reward density, and team size, including matrix games, navigation, combat, foraging, and warehouse coordination.

  • The evaluation comprises two repeated matrix games and four multi-agent environments, yielding 25 fully cooperative tasks.Tasks vary in observability, reward density, and number of agents.
  • Matrix Games: The matrix games test coordination under constant observations and local minima, with penalty-game difficulty increasing as k becomes smaller.
  • Multi-Agent Particle Environments: MPE provides two-dimensional coordination tasks with feature-vector observations; Speaker-Listener additionally requires binary communication.
  • StarCraft Multi-Agent Challenge: SMAC simulates cooperative battles where agents observe nearby units and vary in controlled unit numbers and types.
  • Level-Based Foraging: LBF requires groups of agents to collect items when their summed levels meet or exceed an item’s level.
  • Multi-Robot Warehouse: RWARE models partially observable warehouse coordination with sparse rewards for fully delivering requested shelves.Agents observe a 3 × 3 surrounding grid and must deliver shelves to workstations and return them.
  • Open-Sourced Environments: LBF and RWARE were open-sourced as maintained environments focused on sparse-reward exploration and coordination.

4 Evaluation

The evaluation standardizes MARL comparisons through controlled training, evaluation, and implementation configurations, while EPyMARL broadens reproducible algorithm coverage and flexibility.

  • Evaluation protocol: On-policy algorithms receive ten times more training samples than off-policy algorithms to account for differences in sample efficiency.Training budgets vary by environment, with 20 million versus two million timesteps in MPE and LBF.
  • Evaluation protocol: Each algorithm is evaluated at constant timestep intervals, using 100 episodes per evaluation point and 41 evaluations during training.Matrix games use separate timestep schedules and 100 evaluations.
  • Implementation configurations: Training configurations include both independent and shared network parameters, with shared agents receiving one-hot identity inputs.Parameter sharing lets agents use common networks while conditioning behavior on agent identity.
  • Evaluation protocol: Hyperparameters are optimized separately for each algorithm and environment using selected representative tasks.Optimization tasks include Speaker-Listener, “3s5z”, “15x15-3p-5f”, and “Tiny 4p”.
  • Metrics: Maximum returns select each algorithm’s best evaluation timestep, whereas average returns incorporate learning speed across all evaluations.Maximum-return estimates include 95% confidence intervals across five random seeds.
  • Implementation configurations: EPyMARL extends PyMARL with five additional policy-gradient algorithms and configurable parameter sharing, observation dimensions, normalization, regularization, and network recurrence.The codebase is publicly available under the Apache License.
  • Reported results: Figure 2 averages normalized evaluation returns across tasks in all environments except matrix games, with shading denoting 95% confidence intervals.Table 3 reports maximum returns for nine algorithms across 25 parameter-sharing tasks, marking the highest mean and statistically indistinguishable alternatives.

5 Results

Across cooperative tasks, algorithm performance depends on observability, coordination demands, reward sparsity, and value-function structure. Independent learning remains competitive in simpler settings, while CTDE and value decomposition offer advantages in many harder tasks, with notable failures in sparse-reward RWARE.

  • Independent Learning: Independent learning performs adequately across tasks but is limited in partially observable SMAC and RWARE compared with CTDE.The limitation is attributed to independent learners’ inability to reason over joint agent information.
  • Independent Learning: IA2C and IPPO are broadly competitive, while IPPO outperforms IA2C in MPE, SMAC, and RWARE but not LBF on average.IPPO also outperforms MAA2C in partially observable RWARE, yet generally trails MAPPO.
  • Centralised Training Decentralised Execution: Centralised critics are valuable for substantial coordination under partial observability, whereas independent learning remains competitive in fully observable MPE and LBF.In RWARE, MAA2C and MAPPO improve returns over their independent counterparts, while COMA, VDN, and QMIX perform very poorly.
  • Centralised Training Decentralised Execution: MAPPO achieves high returns in most tasks and exceeds all other algorithms in RWARE, although not always significantly.Its surrogate objective combines on-policy optimisation with improved sample efficiency compared with MAA2C.
  • Value Decomposition: Value decomposition performs effectively in most environments, with VDN and QMIX usually matching or outperforming the highest returns except in RWARE.VDN and QMIX show no learning in RWARE, while QMIX can outperform VDN when linear value decomposition is insufficient in more complicated tasks.
  • Parameter Sharing: Parameter sharing improves returns in every environment except matrix games, with larger gains appearing where rewards are sparser.The reported improvements are especially associated with harder MPE and SMAC tasks and larger or lower-agent-count LBF and RWARE settings.

6 Analysis

The analysis identifies when independent learning, centralised training, and value decomposition are effective across cooperative MARL tasks. Parameter sharing generally improves returns outside matrix games, while coordination demands and reward structure shape algorithm performance.

  • Independent learning: Independent learning performs well when agents can use local observation histories without extensive coordination.IQL is limited in tasks requiring agents to track changing behaviour through a replay buffer, but performs relatively well in smaller SMAC and most LBF tasks.
  • Centralised training: Centralised information benefits individual-policy optimisation when partial observability hides other agents or parts of the environment.The advantage is especially apparent in four-agent RWARE tasks, whereas IA2C and MAA2C perform similarly in fully observable tasks and the two-agent RWARE task.
  • Value decomposition: VDN and QMIX improve over IQL across most tasks, but QMIX’s flexible mixing mainly helps harder MPE tasks and adds expense in simpler LBF settings.Both value-decomposition methods perform significantly worse than policy-gradient methods except COMA in sparse-reward RWARE tasks.

7 Conclusion

The paper benchmarks nine MARL algorithms across 25 cooperative tasks and releases tools and environments for consistent evaluation. Its conclusions are limited to cooperative settings and commonly used algorithms, leaving several MARL challenges for future study.

  • Evaluation: The study evaluates nine MARL algorithms across 25 cooperative tasks spanning observability, reward sparsity, and two to ten agents.Performance is compared using maximum and average returns.
  • Resources: EPyMARL provides an open-source codebase for consistent evaluation of MARL algorithms in cooperative tasks.The work also implements and releases the LBF and RWARE environments for sparse-reward exploration.
  • Scope: The conclusions are limited to cooperative environments and commonly used MARL algorithms.Competitive environments and challenges including exploration, communication, and opponent modelling require additional studies.

Funding Disclosure

The research received financial support from two academic sources.

  • Funding: The work was partly funded by the UK EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems.The named authors associated with this support are G.P. and F.C.
  • Funding: The work was also partly supported by the Edinburgh University Principal’s Career Development Scholarship.The named recipient associated with this support is L.S.
  • Funding: The disclosure identifies partial financial support rather than a single funding source.Both funding sources are listed in the paper’s acknowledgement.

Responsibility Statement

The supplementary material documents two open-source cooperative MARL environments and their Python/Gym interfaces, configurations, observations, actions, and rendering. It also states that the environments and benchmark use MIT licensing.

  • Environment resources: The authors provide details and open-source access for Level-Based Foraging and Multi-Robot Warehouse, designed around cooperation and sparse rewards.The supplementary material covers accessibility, licensing, installation, interfaces, observations, rewards, and dynamics.
  • Licensing and responsibility: The environments are licensed under MIT, and the authors state that they will maintain both environments as needed.The authors also bear responsibility for rights violations involving the proposed environments and benchmark.
  • Environment interface: Both environments follow OpenAI’s Gym interface and support creating registered configurations, starting episodes, stepping with one action per agent, and rendering.Environment interactions return next observations, rewards, termination flags, and transition metadata.
  • Configuration: Level-Based Foraging configurations specify observability, map dimensions, agent count, and food count, while RWARE configurations specify warehouse size, agent count, and difficulty.Default LBF maps range from 5 to 20 units and RWARE supports tiny, small, medium, and large warehouse sizes.
  • Level-Based Foraging: Agents receive full-state or partial observations in LBF, with observations represented as arrays containing coordinate and level triplets for food items and agents.The described example is the Foraging-8x8-2p-3f-v1 task, and each agent has six discrete actions.

A.8 Environments Simulation Speed Comparison

The benchmark compares simulation speed across all 25 tasks using 10,000 random-action steps without rendering. Results show substantial variation by environment, while EPyMARL supports broader environment coverage and flexible implementation tuning.

  • Benchmark setup: 10,000 environment steps were simulated with random actions and without rendering to approximate training or evaluation conditions.Total simulation time and time per step were reported for all tasks.
  • Results: Level-Based Foraging was faster to simulate than all environments except matrix games and the simplest MPE tasks.RWARE was only marginally more expensive to simulate than MPE despite its greater apparent complexity.
  • Reproducibility: EPyMARL extends PyMARL with additional algorithms, more environments, and flexible implementation-detail tuning.The codebase is intended to support consistent comparisons across MARL experiments.

C Task Specifications

The benchmark spans 25 fully cooperative tasks across matrix games, MPE, SMAC, LBF, and RWARE, varying observability, reward density, team structure, and agent count. LBF and RWARE expose particularly difficult sparse-reward coordination settings.

  • Benchmark scope: The evaluation contains 25 fully cooperative tasks across two matrix games and four multi-agent environments.Tasks vary in observability, reward density, and number of agents.
  • MPE: MPE tasks use continuous two-dimensional particle-and-landmark worlds with dense rewards and discrete movement actions.The tasks include Speaker-Listener, Spread, Adversary, and Predator-Prey.
  • SMAC: SMAC consists of partially observable cooperative combat scenarios requiring precise action sequences and coordination against a game-controlled army.Scenarios vary unit types, team sizes, symmetry, and terrain use.
  • Level-Based Foraging: LBF tasks require agents to navigate grids and jointly collect items when their combined levels meet item-level requirements.Task names encode grid size, agent count, food count, partial observability, and cooperative-only variants.
  • Level-Based Foraging: No algorithm learned cooperative-only LBF variants with three or more agents because sparse rewards require many agents to coordinate before any reward is obtained.The authors identify these tasks as challenges for multi-agent intrinsic exploration.
  • Multi-Robot Warehouse: RWARE benchmark tasks are limited by algorithms’ inability to learn and decompose rewards under sparse-reward settings.Larger-scale warehouse scenarios would additionally require scaling, communication, and coordination among many agents.

E Additional Results

Additional results report average and maximum returns for nine algorithms across 25 tasks under both parameter-sharing settings. The tables use five-seed estimates with 95% confidence intervals.

  • Parameter sharing: Table 6 reports average returns and 95% confidence intervals for nine algorithms with parameter sharing across 25 tasks.The results are averaged over five seeds.
  • No parameter sharing: Tables 7 and 8 report maximum and average returns, respectively, for nine algorithms without parameter sharing across the same 25 tasks.These tables provide complementary performance summaries under independent network parameters.

F Visualisation of the Evaluation Returns During Training

Training-return visualisations compare nine algorithms with parameter sharing across 25 tasks, using means and 95% confidence intervals over five seeds. Hyperparameters and implementation choices are selected separately by environment and algorithm.

  • Evaluation visualisation: Figure 11 presents training-time evaluation returns for nine algorithms with parameter sharing in 25 tasks.The figure reports means and 95% confidence intervals over five random seeds.
  • Hyperparameter selection: Hyperparameters are optimised separately for each algorithm in one task per environment and then held constant across the remaining tasks in that environment.Each hyperparameter combination is evaluated with three seeds, and the best average evaluation is selected.
  • Training details: The evaluation uses experience replay for off-policy algorithms and parallel synchronous workers for on-policy algorithms.Q-based exploration uses epsilon-greedy schedules, while stochastic policies explore by sampling their categorical policy.
  • Implementation configurations: The appendix provides separate hyperparameter tables for each algorithm with and without parameter sharing.The listed algorithms include IQL, IA2C, IPPO, MADDPG, COMA, MAA2C, MAPPO, VDN, and QMIX.
Loading 2006.07869v4…