PaperlayerView exact revision on arXiv

Source-linked AI summary

SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning

Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, Shimon Whiteson

arXiv:2212.07489v2cs.LGcs.MA

TL;DR

SMAC can often be solved without complex observation-dependent policies, limiting its value as a cooperative MARL benchmark. The paper introduces procedurally generated SMACv2 and the EPO challenge, then finds that strong SMAC baselines struggle on the resulting scenarios. These changes alleviate SMAC’s lack of stochasticity and meaningful partial observability, while motivating evaluation across multiple benchmarks.

  • Problem

    SMAC exhibits near-perfect performance, open-loop success on many scenarios, and limited meaningful partial observability, weakening its ability to test complex CTDE policies.

  • Method

    The paper introduces SMACv2 with procedural variation in team compositions and positions, plus EPO mechanisms that create stochastic enemy visibility and harder target selection.

  • Results

    Strong SMAC baselines struggle on many SMACv2 scenarios, while ablations identify stochastic unit types and random start positions as contributors to task difficulty.

  • Takeaways & Limitations

    SMACv2 alleviates SMAC’s stochasticity and meaningful-partial-observability deficiencies and provides a more challenging benchmark for MARL methods.

  • Takeaways & Limitations

    SMACv2 remains confined to StarCraft II, so MARL evaluation should include a variety of tasks and benchmarks.

Abstract

from arXiv · show

The availability of challenging benchmarks has played a key role in the recent progress of machine learning. In cooperative multi-agent reinforcement learning, the StarCraft Multi-Agent Challenge (SMAC) has become a popular testbed for centralised training with decentralised execution. However, after years of sustained improvement on SMAC, algorithms now achieve near-perfect performance. In this work, we conduct new analysis demonstrating that SMAC lacks the stochasticity and partial observability to require complex *closed-loop* policies. In particular, we show that an *open-loop* policy conditioned only on the timestep can achieve non-trivial win rates for many SMAC scenarios. To address this limitation, we introduce SMACv2, a new version of the benchmark where scenarios are procedurally generated and require agents to generalise to previously unseen settings (from the same distribution) during evaluation. We also introduce the extended partial observability challenge (EPO), which augments SMACv2 to ensure meaningful partial observability. We show that these changes ensure the benchmark requires the use of *closed-loop* policies. We evaluate state-of-the-art algorithms on SMACv2 and show that it presents significant challenges not present in the original benchmark. Our analysis illustrates that SMACv2 addresses the discovered deficiencies of SMAC and can help benchmark the next generation of MARL methods. Videos of training are available at https://sites.google.com/view/smacv2.

1 Introduction

SMAC is widely used for CTDE but shows ceiling effects and can often be solved without observation-dependent policies. SMACv2 addresses these deficiencies through procedural generation, stronger partial observability, and new evaluation challenges.

  • CTDE trains with privileged joint observations or state but executes policies without that centralised information.
  • SMAC became a prominent cooperative MARL benchmark for decentralised micromanagement requiring joint action sequences such as focus fire and kiting.
  • Near-perfect win rates and strong independent-learning performance suggest SMAC has ceiling effects and may no longer reward algorithmic improvements.
  • An open-loop policy using only timestep information performs well on many SMAC scenarios, while QMIX reconstructs joint Q-values with less than 10% error in all but two scenarios.
  • SMACv2 procedurally randomises team compositions and start positions, requiring coordination across diverse scenarios instead of repeating fixed action sequences.
  • The EPO challenge adds stochastic enemy visibility, removes available-action masking, and changes sight and attack ranges to require information sharing and harder target selection.
  • State-of-the-art algorithms struggle on many SMACv2 scenarios, while ablations identify stochastic unit types and random start positions as contributors to difficulty.

2 Related Work

Related benchmarks cover diverse cooperative MARL settings, while SMACv2 extends SMAC with procedural variation and meaningful partial observability. Its design targets generalisation beyond fixed unit compositions and positions.

  • Hanabi is stochastic and partially observable but supports only teams of 2 to 5 players, fewer than most SMACv2 scenarios.
  • Multi-agent MuJoCo uses complex continuous actions, whereas MPE focuses on simple communication-oriented continuous-action challenges.
  • SMAC focuses on decentralised micromanagement of StarCraft units, contrasting with full-game benchmarks using centralised control over macromanagement and micromanagement.
  • SMAC’s fixed unit types and starting positions enable memorised action sequences and limit generalisation to new test settings.
  • SMACv2 uses procedural content generation to vary episodes and additionally changes start positions, observability, sight ranges, and attack ranges.

3 Background

A Dec-POMDP formalises cooperative partially observable MARL through agents, states, actions, transitions, observations, rewards, and discounting. SMAC instantiates this setting as decentralised StarCraft micromanagement.

  • A Dec-POMDP is represented by agents, states, individual actions, transitions, joint observations, observation dynamics, rewards, and a discount factor.
  • The transition function maps a state and joint action to a distribution over next states, while the reward function maps them to a reward distribution.
  • At each timestep, agents choose actions, the environment transitions and produces reward, and each agent receives an observation sampled from the observation function.
  • SMAC controls individual StarCraft military units with separate learning agents operating under limited field of view and no explicit test-time communication.

4 Partial Observability and Stochasticity in CTDE

Open-loop policies ignore observations and suffice in deterministic settings, so stochasticity is needed for observation-dependent control to matter. Partial observability is meaningful only when hidden information affects decisions and cannot be inferred locally.

  • If initial states and transition dynamics are deterministic, an optimal joint policy can ignore observations and replay actions based only on timestep and agent ID.
  • The open-loop versus closed-loop comparison tests whether SMAC policies need environment observations to handle stochasticity.
  • Partial observability is meaningful when hidden information is task-relevant, unknown to another agent, and not inferable from that agent’s own observation.

5 Limitations of SMAC

SMAC often permits successful open-loop control, indicating insufficient stochasticity and meaningful partial observability for complex closed-loop policies. Q-value regression likewise shows that timestep information can explain much of the learned policy.

  • 5.1 SMAC Stochasticity: Open-loop policies, conditioned only on agent ID and timestep, perform well across many SMAC scenarios.They fail to learn good policies on only four maps: 3s5z_vs_3s6z, corridor, 6h_vs_8z and 5m_vs_6m.
  • 5.1 SMAC Stochasticity: Open-loop performance varies widely across scenarios and algorithms, with some policies matching closed-loop results and others failing completely.MAPPO open-loop policies can match closed-loop performance on several scenarios, while corridor and 2c_vs_64zg show contrasting outcomes.
  • 5.1 SMAC Stochasticity: SMAC’s limited stochasticity means agents can often repeat a fixed action sequence instead of adapting to diverse situations.The analysis links this deficiency to the absence of meaningful partial observability on many scenarios.
  • 5.2 SMAC Feature Inferrability & Relevance: Masking all features still yields low QMIX Q-value regression error, showing that timestep information can largely predict the joint Q-function.The error is below 0.12 for all scenarios except 5m_vs_6m, and is typically 5–10% of the mean Q-value during most of an episode.
  • 5.2 SMAC Feature Inferrability & Relevance: Because observation features add little predictive information, SMAC may not require decentralised micromanagement reasoning in many scenarios.The feature-masking analysis compares subsets of observations to assess how strongly the learned policy conditions on them.

6 SMACv2

SMACv2 addresses SMAC’s deficiencies by randomising teams, positions, and unit ranges, while EPO adds stochastic enemy visibility and removes action masks. These changes diversify scenarios and require agents to use observations and coordination more meaningfully.

  • 6 SMACv2: SMACv2 randomises team compositions, start positions, and unit attack and sight ranges to increase scenario stochasticity.The benchmark uses procedural generation so agents cannot know their unit types or initial positions before an episode.
  • 6 SMACv2: Procedural team generation samples unit types with fixed probabilities shared between training and test time.Scenarios are separated by Protoss, Terran, and Zerg races, with symmetric and asymmetric team configurations.
  • 6 SMACv2: Reflect scenarios mirror randomly spawned allied positions across the map, whereas surround scenarios place allies centrally among enemies on four diagonals.These two start-position distributions create different spatial configurations across episodes.
  • 6 SMACv2: Using true StarCraft II attack and sight ranges increases diversity among unit types compared with SMAC’s fixed ranges.A minimum attack range of 2 is imposed because true melee ranges make attacking difficult.
  • 6.1 Extended Partial Observability Challenge: EPO stochastically masks enemy observations after an initial sighting, with suggested success probability p = 0 for subsequent observers.The first observing agent retains visibility, while other agents receive independent persistent visibility outcomes.
  • 6.1 Extended Partial Observability Challenge: EPO removes the available-action mask, forcing agents to infer or communicate which enemies are attackable.Invalid actions become no-ops, preventing action availability from revealing shooting-range information.

7 SMACv2 Experiments

SMACv2 experiments show that procedural variation and extended partial observability create challenges that strong SMAC baselines do not consistently solve. Open-loop policies fail across maps, while feature-masking results indicate that environment features matter for SMACv2 performance.

  • 7.1 SMACv2 Baseline Comparisons: QMIX generally outperforms MAPPO across most SMACv2 scenarios, with stronger performance on two Protoss maps and greater sample efficiency.MAPPO may improve with more computation, while its similarity to IPPO suggests limited benefit from the centralised state in these experiments.
  • 7.1 SMACv2 Baseline Comparisons: Open-loop policies cannot learn good policies on any SMACv2 map, including maps where QMIX and MAPPO achieve high win rates.This result supports the conclusion that SMACv2 addresses SMAC’s lack of stochasticity.
  • 7.1 SMACv2 Baseline Comparisons: Win rates are roughly even on symmetric maps but very low on asymmetric maps, particularly the 20_vs_23 scenarios.All algorithms struggle more on Zerg scenarios, while difficulty does not substantially change as the number of agents scales.
  • 7.2 EPO Baselines: On EPO, p = 0 struggles to attain modest performance, whereas p = 1 comes close to solving the task in several instances.The p = 0.5 setting resembles p = 0, indicating that effective communication would need to be reasonably robust.
  • 7.3 Feature Quality Experiments: SMACv2 scenarios have higher feature-masking errors than SMAC scenarios, indicating greater stochasticity and the importance of attending to ally and enemy features.For example, the lowest SMACv2 difference is 0.12 for 5_gen_protoss, compared with 0.07 for the highest SMAC value, 5m_vs_6m.

8 Conclusion

The paper concludes that SMAC lacks sufficient stochasticity and meaningful partial observability, and presents SMACv2 with EPO as a more demanding evaluation setting. Its experiments show that strong SMAC baselines struggle on SMACv2, while stochasticity and partial observability contribute to difficulty.

  • 8 Conclusion: SMAC suffers from insufficient stochasticity and meaningful partial observability, as shown by the paper’s experimental evaluations.
  • 8 Conclusion: Strong SMAC baselines struggle to achieve high win rates on SMACv2 scenarios.
  • 8 Conclusion: Unit-type stochasticity and random start positions jointly explain the difficulty of SMACv2 tasks in the reported ablations.
  • 8 Conclusion: Meaningful partial observability significantly contributes to the challenge in the SMACv2 EPO environment.
  • 8 Conclusion: The paper recommends evaluating MARL methods on multiple benchmarks where possible because single-benchmark evaluation can remain vulnerable to environment flaws and community-wide overfitting.

C Experimental Details

The experimental-details section introduces the supplementary description of the paper’s experiments and points to implementation details for the closed-loop and open-loop procedures.

  • C Experimental Details: The section provides additional details about the experiments reported in the paper.
  • C Experimental Details: The closed-loop and open-loop algorithms used the MAPPO implementation, shared its neural-network architecture, and differed in the information available to the actor.
  • C Experimental Details: Both procedures received the environment’s available-actions mask and conditioned the critic on the state.

C.2 SMAC Feature Inferrability & Relevance

The supplementary experiments detail how feature relevance, baseline training, EPO, and SMACv2 team generation were implemented. They describe masked-Q regression procedures, training settings, and environment design choices.

  • C.2 SMAC Feature Inferrability & Relevance: Feature-quality experiments train regression QMIX networks to predict expert-policy Q-values from trajectories with selected observations and states masked.
  • C.2 SMAC Feature Inferrability & Relevance: The nothing mask leaves observations and states unchanged, while feature masks zero selected features across agents’ observations and the QMIX state.
  • C.2 SMAC Feature Inferrability & Relevance: Regression networks match expert-network architectures and use MSE loss with Adam, batch size 512 episodes, and learning rate 0.005.
  • C.2 SMAC Feature Inferrability & Relevance: Hyperparameters were tuned on separate SMAC and SMACv2 scenarios and expert-policy datasets to minimise validation MSE.
  • D.1 SMACv2 Additional Details: SMACv2 team generation uses three unit types per race, with special units such as colossi, medivacs, and banelings spawned less frequently.Each special unit is spawned with probability 10%, while other units use probability 45%.
  • D.1 SMACv2 Additional Details: SMACv2 adds each agent’s field-of-view direction and normalized x- and y-position to observations, and adds field-of-view direction to the state.
  • D.1 SMACv2 Additional Details: SMACv2 fixes a reward bug involving enemy healing and retains an API identical to the original SMAC.

E Analysis of Changes in SMACv2

SMACv2’s ablations show that randomised starting positions and diverse unit types drive much of its difficulty, while varying field of view has a smaller effect. Recurrent policies outperform feed-forward policies, but generalising across micro-management scenarios remains the primary challenge.

  • Random start positions: Without random start positions, MAPPO achieves win rates close to 100%, showing that stochastic starting positions substantially increase SMACv2’s difficulty.Surround and reflect scenarios have similar win rates, and combining them does not add apparent difficulty because start positions disambiguate the scenarios early.
  • Unit type diversity: Diverse unit compositions significantly affect task difficulty across all three races and make generalisation across unit types a central SMACv2 challenge.Melee-only Zerg and Protoss scenarios are easier than ranged-unit ablations, while Terran marine scenarios are slightly easier.
  • Field of view: Varying unit sight and attack ranges increases difficulty, but only slightly compared with the effects of start-position and unit-type diversity.The ablation compares fixed SMAC ranges with SMACv2’s true unit ranges.
  • Partial observability: Feed-forward policies perform significantly worse than RNN baselines, although they can still learn reasonable policies in SMACv2.The performance gap is larger in the Zerg scenario, where baneling splash damage encourages allies to spread out.
  • Overall challenge: State-of-the-art MARL methods struggle with many SMACv2 scenarios, indicating that map randomisation requires inference from observations about relevant state or joint features.The analysis identifies more effective sequence models, including specialised transformers, as a possible research direction.

F Limitations and Broader Impact

SMACv2 is limited by its confinement to StarCraft II and by allowing reinforcement-learning agents to control only one army. The authors therefore recommend evaluating MARL algorithms across a variety of tasks and leave two-sided control for future work.

  • Scope: SMACv2 is confined to StarCraft II, whose dynamics cannot represent all multi-agent tasks.The authors recommend that MARL evaluation target a range of tasks rather than relying on one benchmark.
  • Control setting: Only one side of each two-army battle can be controlled by reinforcement-learning agents; two-sided control and self-play remain future work.Controlling both armies would require two StarCraft II clients communicating over a LAN.
  • Broader impact: Improved cooperative-AI capabilities may have unethical uses, although the authors judge the potential benefits to outweigh the potential risks.The paper cites automated driving as one potential benefit of better cooperative AI.
Loading 2212.07489v2…