Source-linked AI summary
The StarCraft Multi-Agent Challenge
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, Shimon Whiteson
TL;DR
Cooperative MARL lacks challenging standard benchmarks for agents coordinating under partial observability. The paper introduces SMAC and PyMARL, then reports benchmark results across diverse decentralised StarCraft II micromanagement scenarios. QMIX achieves the highest test win percentage overall, while harder scenarios expose distinct coordination challenges; the benchmark is intended to support systematic progress, with planned extensions to more diverse and difficult scenarios.
Problem
Cooperative MARL lacks challenging, standardised testbeds for evaluating coordination under partial observability.
Method
The paper introduces SMAC, 14 decentralised StarCraft II micromanagement scenarios, evaluation recommendations, and the open-source PyMARL framework.
Results
QMIX achieves the highest test win percentage overall and is the best performer on up to eight scenarios during training.
Takeaways & Limitations
SMAC provides a benchmark for systematic evaluation of cooperative MARL under partial observability and decentralised execution.
Takeaways & Limitations
The authors plan to extend SMAC with more diverse units, terrain features, and harder coordination scenarios.
Abstract
from arXiv · showhide
In the last few years, deep multi-agent reinforcement learning (RL) has become a highly active area of research. A particularly challenging class of problems in this area is partially observable, cooperative, multi-agent learning, in which teams of agents must learn to coordinate their behaviour while conditioning only on their private observations. This is an attractive research area since such problems are relevant to a large number of real-world systems and are also more amenable to evaluation than general-sum problems. Standardised environments such as the ALE and MuJoCo have allowed single-agent RL to move beyond toy domains, such as grid worlds. However, there is no comparable benchmark for cooperative multi-agent RL. As a result, most papers in this field use one-off toy problems, making it difficult to measure real progress. In this paper, we propose the StarCraft Multi-Agent Challenge (SMAC) as a benchmark problem to fill this gap. SMAC is based on the popular real-time strategy game StarCraft II and focuses on micromanagement challenges where each unit is controlled by an independent agent that must act based on local observations. We offer a diverse set of challenge maps and recommendations for best practices in benchmarking and evaluations. We also open-source a deep multi-agent RL learning framework including state-of-the-art algorithms. We believe that SMAC can provide a standard benchmark environment for years to come. Videos of our best agents for several SMAC scenarios are available at: https://youtu.be/VZ7zmQ_obZ0.
1 Introduction
Deep cooperative MARL needs agents to coordinate under partial observability, but research lacks challenging, standardised benchmarks. SMAC addresses this gap with decentralised StarCraft II micromanagement scenarios, evaluation guidance, and PyMARL.
- Cooperative MARL is relevant to decentralised systems where agents must coordinate using limited observations and potentially limited communication.
- The field lacks standardised benchmarks, with researchers often relying on one-off environments that may be overly simple or algorithm-specific.
- SMAC introduces decentralised StarCraft II micromanagement scenarios in which independent agents act from local observations against the built-in game AI.
- PyMARL is an open-source, modular PyTorch framework implementing several MARL algorithms and supporting research on practical multi-agent learning challenges.
- The paper provides evaluation guidelines covering standardised performance metrics, sample efficiency, and computational requirements.
- SMAC is intended to support systematic and robust progress in deep MARL as a standard benchmark.
2 Related Work
Prior multi-agent research spans simplified cooperative, competitive, and full-game environments, but challenging testbeds combining partial observability, dynamics, and high-dimensional observations remain limited. SMAC targets that gap through decentralised StarCraft II micromanagement.
- Many prior environments do not jointly provide qualitative challenge, partial observability, challenging dynamics, and high-dimensional observations.
- Existing multi-agent testbeds include Poker, Pong, Keepaway Soccer, and gridworld-like environments across competitive and cooperative settings.
- StarCraft has supported AI research from evolutionary and tabular methods to deep learning, often in single-agent or centrally controlled settings.
- SMAC differs from full-game StarCraft environments by imposing decentralisation and local observability for rich multi-agent problems.
3 Multi-Agent Reinforcement Learning
SMAC is framed as a fully cooperative Dec-POMDP problem in which agents share rewards but observe the environment locally. Centralised training can use additional state information while execution remains decentralised.
- SMAC studies fully cooperative multi-agent tasks in which a team works together toward a common goal, formalised as Dec-POMDPs.
- A Dec-POMDP models a true environment state, individual actions, joint actions, and state transitions governed by a transition function.
- Agents share a team reward while receiving individual observations and conditioning stochastic policies on their action-observation histories.
- Centralised training with decentralised execution allows learning to use additional state information available in simulation while policies execute from local information.
- COMA uses a multi-agent critic baseline, whereas QMIX belongs to the Q-learning family and uses centralised-training information.
4 SMAC
SMAC converts StarCraft II micromanagement into decentralised cooperative combat scenarios with local observations and varied unit configurations. The benchmark tests coordination under partial observability, constrained actions, terrain, and centralised training state information.
- SMAC: SMAC focuses on fine-grained unit control rather than macromanagement, assigning each unit an independent agent restricted to local observations.
- Scenarios: Scenarios vary army positions, unit counts and types, and terrain, and require agents to learn one or more micromanagement techniques to win.
- State and Observations: Agents receive observations within a circular sight range, creating partial observability and preventing them from distinguishing distant teammates from dead ones.
- State and Observations: The centralised-training state contains all units’ map-relative coordinates and additional unit features, energy, cooldowns, and agents’ last actions.
- Actions and Rewards: Agents can move, attack, stop, or do nothing, while Medivacs heal; the maximum action count ranges from 7 to 70 by scenario.
- Actions and Rewards: Attacks are restricted to enemies within shooting range 6, while sight range exceeds shooting range and forces movement before firing.
- Actions and Rewards: The default shaped reward combines damage dealt, enemy kills, and a winning bonus, with configurable event values and scales.
5 PyMARL
PyMARL is an open-source, modular PyTorch framework for developing and benchmarking deep MARL algorithms. It provides implementations of several state-of-the-art methods and baselines.
- PyMARL is designed for out-of-the-box experimentation and development in deep multi-agent reinforcement learning.
- Its modular organization supports rapid algorithm development, benchmarking, extension, and component-level testing.
- PyMARL encapsulates replay-buffer data to simplify handling between components while preserving access to underlying PyTorch tensors.
- The framework maximizes inference and learning batch processing to speed execution compared with more naive implementations.
- PyMARL implements QMIX, QTRAN, and COMA as state-of-the-art methods, alongside VDN and IQL as baselines.
6 Results
The SMAC evaluation compares deep MARL algorithms across 14 scenarios using decentralised greedy test episodes and test win rate. QMIX performs best overall, while scenario difficulty exposes substantial variation and unresolved challenges.
- Test win rate is the percentage of 32 greedy decentralised test episodes in which agents defeat all enemies within the time limit.
- Figure 3 compares median test win percentage across all 14 scenarios and counts scenarios where an algorithm leads by at least 1/32.
- QMIX achieves the highest overall test win percentage and is best on up to eight scenarios during training.
- IQL, VDN, and QMIX significantly outperform COMA, demonstrating greater sample efficiency for off-policy value-based methods than on-policy policy gradients.
- Easy scenarios: QMIX exceeds 95% test win rate on all five Easy scenarios, which function as sanity checks for new algorithm implementations.
- Hard scenarios: Hard scenarios isolate distinct challenges, including enlarged action spaces, high variance, asymmetric control, and delayed rewards from prolonged kiting.
- Super Hard scenarios: Only QMIX makes meaningful progress on two of five Super Hard scenarios, with exploration hypothesised as a bottleneck.
- QTRAN: QTRAN fails to achieve good performance on 3s5z and takes far longer to reach the performance of
7 Conclusion and Future Work
SMAC provides diverse cooperative MARL benchmark scenarios centered on decentralised micromanagement, and the authors plan harder coordination challenges to probe existing methods.
- SMAC comprises 14 diverse combat scenarios for cooperative MARL, testing partial observability and high-dimensional inputs.
- Future work: The authors plan scenarios with more diverse units, stronger coordination demands, and terrain-dependent strategies to expose gaps in MARL methods.They specifically identify multi-agent exploration and coordination as areas for further research.
- The authors aim for SMAC to become a standard benchmark for measuring progress in cooperative MARL.
- Scenario diversity: The scenarios include symmetric battles involving countering, asymmetric battles requiring health-aware targeting, and micro-trick challenges using terrain.In corridor, six Zealots must collectively wall off a choke point against 24 Zerglings.
- Scope: SMAC uses StarCraft II’s engine but targets decentralised micromanagement rather than training agents for full-game StarCraft II.The environment uses raw API observations and individual unit commands, while modifying unit behavior to encourage learned micro strategies.
B Evaluation Methodology
The evaluation methodology fixes environment and execution conditions to make SMAC results fair and comparable while allowing centralised training.
- SMAC evaluations should leave observation, state, action, mechanics, map files, AI difficulty, and other environment settings unchanged.The recommendation includes frame-skipping rate and scenario episode limits.
- Testing must be decentralised, with each agent using only its own action-observation history rather than global state or other agents’ observations.Centralised training remains allowed to exchange observations, parameters, gradients, and global state.
B.1 Evaluation Metrics
SMAC measures learning progress through win percentage over environment steps, using repeated evaluation runs and robust summary statistics.
- The main metric is mean win percentage across evaluation episodes as a function of environment steps during training.
- Results use five independent training runs, reporting median performance with 25–75% percentiles to reduce sensitivity to outliers.Evaluation uses 32 episodes with exploratory behavior disabled, and the paper recommends the median instead of the mean.
- SMAC supports replay saving and video inspection to help researchers examine interesting learned behaviors alongside quantitative metrics.
C.1 Architecture and Training
The experiments use recurrent, parameter-shared agent networks with algorithm-specific mixing or critic components, fixed exploration schedules, and shaped rewards.
- Agent networks are DRQNs with 64-dimensional GRU hidden states and independent ϵ-greedy exploration annealed from 1.0 to 0.05 over 50k steps.
- Agent parameters are shared across units, with one-hot agent_id appended to observations to distinguish agents.
- QMIX: QMIX uses a 32-unit mixing network and 64-unit ReLU hypernetworks whose outputs are made non-negative before reshaping.
- COMA: COMA uses a fully connected critic with two 128-unit layers and λ = 0.8, while its policies use a separate annealing schedule.COMA rolls out eight episodes, updates the critic backward through each episode, then updates policies once on all eight episodes.
- QTran: QTran experiments compare architectural variants, network depths, and λ settings through hyperparameter searches across three tested maps.
- Rewards and observations: All experiments use shaped rewards for damage, kills, and winning, scaled so the maximum cumulative reward is around 20.Observations omit allied units’ last actions, terrain height, and walkability from the available features.
D Table of Results
Table 2 reports final median performance for the tested algorithms and includes the heuristic-based algorithm’s mean test win rate across 1000 episodes.
- Table 2 lists test win rate percentages for IQL, COMA, VDN, QMIX, and a heuristic-based algorithm.Final median performance is defined as the maximum median across testing intervals within the last 250k training steps; the heuristic-based algorithm also has a mean test win rate across 1000 episodes.