Source-linked AI summary
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, Shimon Whiteson
TL;DR
Cooperative MARL commonly relies on centralized joint value functions, but their necessity and independent learning's practical viability remain open questions. The paper evaluates IPPO, which learns decentralized policies with local value functions, against joint-learning approaches and variants. IPPO matches or outperforms state-of-the-art methods on hard SMAC maps, while ablations identify policy clipping as crucial and suggest robustness to some environment non-stationarity.
Problem
Independent learning has theoretical limitations, while centralized and factored joint-value methods introduce their own decentralization, credit-assignment, and representational concerns.
Method
The paper empirically evaluates IPPO, using decentralized policies and local observation-based critics with PPO's individual policy clipping.
Results
IPPO matches or outperforms centralized-state and independent-learning baselines on multiple hard SMAC maps; ablations find policy clipping crucial to performance.
Takeaways & Limitations
Independent learning can achieve the best performance on some hard SMAC maps without joint or centralized value functions, motivating further study and improvement of approaches such as IPPO.
Takeaways & Limitations
The value of central state information in SMAC remains unclear, despite evidence that it can accelerate training in value-factorization algorithms.
Abstract
from arXiv · showhide
Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.
1 Introduction
Cooperative MARL commonly uses centralized training to address independent learning's theoretical limitations, but this paper finds that IPPO can match or exceed joint-learning methods on SMAC. The authors further investigate whether PPO's policy clipping improves robustness to environment non-stationarity.
- Independent learning: Independent learning decomposes an n-agent problem into decentralized single-agent problems, treating other agents as part of the environment and conditioning policies on local histories.This makes learning easy to distribute and decentralizable by construction.
- Independent learning: Other learning agents make the environment non-stationary from each independent learner's perspective, removing convergence guarantees and potentially causing instability or suboptimal performance.Independent learners may also struggle to distinguish environmental stochasticity from other agents' exploration.
- Centralized training: Centralized training can provide agents with others' observations and extra state information, while decentralized execution preserves locally conditioned policies.Centralized joint learning can reduce partial-observability and non-stationarity issues, but vanilla joint policies are not inherently decentralizable.
- Value factorization: Value factorization reduces the joint action-space burden but can suffer relative overgeneralization, converging to suboptimal joint actions when successful coordination is rare.Monotonic factorizations cannot represent some nonmonotonic team-reward functions.
- Paper contribution: IPPO matches or outperforms state-of-the-art CTDE algorithms such as QMIX and MAVEN on multiple hard SMAC maps.The paper tests whether PPO choices, especially policy clipping, help mitigate some forms of environment non-stationarity.
2 Related Work
Related work spans independent learning, centralized critics, value decomposition, and trust-region policy optimization. Recent decentralized cooperative applications of PPO include MAPPO, which differs from IPPO by using a centralized value function.
- Centralized critics and value decomposition: Centralized joint critics can support decentralized policies because critics are unnecessary during execution, while COMA adds a counterfactual baseline for multi-agent credit assignment.Value Decomposition Networks instead decompose joint state-action values into sums of decentralized utility functions.
- Independent learning: Independent learning predates neural MARL, and earlier studies found policy or experience sharing generally advantageous while extra sensory information could sometimes interfere with learning.This work extends the longstanding comparison between independent and more fully observing agents.
- Policy optimization: TRPO constrains policy updates within a trust region, whereas PPO trades theoretical guarantees for computational speed while retaining many empirical benefits.PPO uses clipped probability ratios to regularize policy updates.
- Multi-agent PPO: MAPPO uses a centralized value function conditioned on full state or concatenated observations, unlike IPPO's independent-learning setup.Both methods employ weight sharing between each agent's critic, but their value-function information differs.
3 Background
A Dec-POMDP models cooperative agents acting under partial observability and stochastic transitions while optimizing a shared discounted reward. CTDE provides centralized information during training while retaining decentralizable policies, and TRPO/PPO regulate policy updates through trust-region or clipping mechanisms.
- Dec-POMDPs: A Dec-POMDP represents cooperative agents choosing sequential actions under partial observability and environment stochasticity.Its formal tuple includes agents, states, actions, transitions, rewards, observations, initial-state distribution, and discount factor.
- Dec-POMDPs: Agents jointly choose actions, transition to a next state through the environment kernel, and receive a scalar team reward.The joint action combines the simultaneous actions of all agents.
- Decentralized execution: Each agent observes only a local observation and bases its decentralized action policy on its individual observation-action history.The collection of individual histories forms the joint history used to describe the team policy.
- Value functions: The joint action-value function estimates expected discounted return for joint actions in states with agent histories.A joint policy maps the team's histories to joint actions.
- Centralized training: CTDE allows agents to access full state and shared policies or experiences during training while formalizing policies that remain decentralizable at execution.Centralized information is used to facilitate training rather than to require centralized execution.
- Policy optimization: TRPO restricts updates using a KL-divergence trust-region constraint, while PPO approximates that constraint with policy-ratio clipping.PPO's approximation avoids the computational expense of repeatedly computing Hessian-vector products for nonlinear conjugate gradients.
4 Independent PPO
IPPO learns decentralized policies with local critics, policy and value clipping, shared network parameters, and entropy regularization. Its loss combines clipped policy objectives with critic and entropy terms.
- IPPO learns decentralized policies πa using individual policy clipping and local observation-based critics for each agent.Each local critic is parameterized separately by φ and supports the independent-learning advantage estimate.
- Generalized Advantage Estimation uses γ = 0.99 and λ = 0.95 for the local critics.The policy and critic networks share parameters across agents, while the team reward approximates each agent’s local reward.
- Value clipping restricts each critic update to within a trust region, helping avoid overfitting to the most recent data batch.The method applies value clipping in addition to policy clipping.
- The overall loss sums per-agent policy losses with weighted critic and entropy-regularization terms.The entropy coefficient and critic-loss coefficient vary according to Table 1.
- The networks use variance-scaled truncated-normal initialization and stacked recent observations processed by three Conv1D layers.The architecture uses fixed kernel size 3 and map-dependent filter settings.
5 Empirical Results
Across SMAC experiments, IPPO performs strongly against independent and centralized-training baselines, including on difficult maps. Ablations indicate that policy clipping is central to this performance, while centralized critics can hurt results.
- 5.1 IPPO Performance: IPPO is evaluated on 16 SMAC maps with only mild per-map hyperparameter tuning.SMAC contains cooperative StarCraft II unit-micromanagement tasks with varying difficulty.
- 5.1 IPPO Performance: Figure 1 compares IPPO with QMIX and IQL using median test win rates and [0.25, 0.75] confidence intervals.The figure covers selected SMAC maps.
- 5.1 IPPO Performance: Figure 2 compares IPPO with MAPPO at 10M steps using median test win rates on selected SMAC maps.
- 5.1 IPPO Performance: IPPO significantly outperforms MAPPO and QMIX on several difficult SMAC maps, exceeds MAVEN on corridor, and remains competitive on additional maps.It also learns difficult maps such as 3s5z vs 3s6z and 6h vs 8z, while outperforming and generally exceeding the stability of IQL.
- 5.2 Role of Centralised Value Functions: Replacing IPPO’s local critics with parameter-sharing critics conditioned on full state information performs substantially worse on selected hard maps.The comparison calibrates the implementation against centralized value-function results reported for MAPPO.
- 5.3 Ablation Studies: Removing both policy and value clipping produces a poor-performing IAC variant across all six studied SMAC maps, with policy clipping essential to performance.
- 5.3 Ablation Studies: Value clipping improves performance on some maps, including corridor and MMM2, when combined with policy clipping.Its selective usefulness is consistent with observations that value clipping helps when critic estimates have high variance.
- 5.3 Ablation Studies: Reducing IAC’s learning rate does not provide the expected gains relative to IPPO, so policy clipping’s effect is not emulated by lowering learning rate alone.
6 Discussion
IPPO outperforms several centralized-state and independent-learning baselines across hard and easy SMAC maps, while policy clipping appears crucial to its performance. These results suggest that some theoretical obstacles to independent learning may not arise in SMAC, although the value of central state information remains unclear.
- IPPO outperforms QMIX, MAPPO, and MAVEN, as well as IAC and IQL, on several hard and easy SMAC maps.The comparison challenges the emphasis on exploiting centralized state during training.
- PPO’s optimization objective, particularly policy clipping, is crucial to cooperative deep multi-agent performance on SMAC.Ablations indicate that this effect cannot be explained by reducing the effective learning rate alone.
- IPPO’s improved stability over IAC and IQL may reflect mitigation of environment non-stationarity through its surrogate objective.The proposed mechanism includes suppressing updates that are catastrophic to performance.
- On some hard SMAC maps, the best performance comes from an algorithm requiring neither joint nor centralized value functions nor other central state information.The authors suggest SMAC’s sequential structure may decompose difficult simultaneous coordination into easier temporal coordination.
- The role of central state information in SMAC remains unclear despite its ability to accelerate training in value-factorization algorithms.IPPO’s strong performance and outperformance of MAPPO raise questions about why central state helps QMIX.
- The discussion draws on results from SMAC experiments with fixed and map-specific IPPO hyperparameters.Table 1 reports the hyperparameter settings used across SMAC maps.
7 Conclusion
The paper finds that IPPO performs competitively across state-of-the-art cooperative MARL benchmarks and can outperform value-factorization methods on some maps. Its results motivate further study of independent learning and whether relative overgeneralization matters in practice.
- IPPO performs competitively across state-of-the-art benchmark tasks and outperforms value-factorization methods on some maps.The conclusion also reports that policy clipping is crucial and that the value of central state information in SMAC is unclear.
- The authors suggest revisiting whether relative overgeneralization matters in practice and improving independent learning approaches such as IPPO.They recommend not focusing exclusively on joint value-function factorization.
9 Appendix
The appendix includes figures comparing IPPO and IPPO-C with value-factorization and independent-learning baselines, and examining relative overgeneralization on Stag Hunt.
- Figure 5 compares IPPO and IPPO-C with QMIX, VDN, and IQL on selected SMAC maps.
- Figure 6 examines relative overgeneralization experienced by selected MARL algorithms on Stag Hunt with penalty −2.The caption notes that DCG is not fully decentralized during execution.