PaperlayerView exact revision on arXiv

Source-linked AI summary

The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, Yi Wu

arXiv:2103.01955v4cs.LGcs.AIcs.MA

TL;DR

PPO is under-used in cooperative multi-agent reinforcement learning because it is widely believed to be less sample-efficient than off-policy methods and sensitive to transferred tuning practices. The paper empirically studies PPO across four cooperative benchmarks, analyzes implementation and hyperparameter factors, and finds that properly configured PPO often matches or exceeds competitive off-policy methods in final returns and sample efficiency. It concludes that simple PPO-based methods can provide strong cooperative MARL baselines.

  • Problem

    PPO is under-used in multi-agent reinforcement learning because it is believed to be less sample-efficient than off-policy methods and because single-agent tuning practices may not transfer well.

  • Method

    The paper conducts a comprehensive empirical study of PPO across four cooperative benchmarks and analyzes five implementation and hyperparameter factors without domain-specific algorithmic changes or architectures.

  • Results

    PPO achieves competitive or superior final returns and sample efficiency relative to state-of-the-art methods across a variety of cooperative multi-agent challenges.

  • Takeaways & Limitations

    Properly configured PPO can serve as a competitive baseline for cooperative multi-agent reinforcement learning tasks.

Abstract

from arXiv · show

Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO's empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at \url{https://github.com/marlbenchmark/on-policy}.

1 Introduction

PPO is widely used in single-agent reinforcement learning but under-used in cooperative multi-agent settings because it is believed to be less sample-efficient and difficult to configure effectively. This study evaluates PPO across four cooperative benchmarks and finds strong performance with minimal tuning and no domain-specific algorithmic changes or architectures.

  • PPO is under-utilized in multi-agent reinforcement learning partly because it is believed to be less sample-efficient than off-policy methods.
  • Default single-agent PPO hyperparameter practices often fail to transfer effectively to multi-agent settings.
  • The study evaluates PPO on MPE, SMAC, Google Research Football, and Hanabi, comparing performance and sample efficiency with off-policy baselines.
  • PPO achieves competitive final performance and comparable sample usage to many off-policy methods without domain-specific algorithmic changes or architectures and with minimal tuning.
  • The paper analyzes five implementation and hyperparameter factors and provides concrete best-practice suggestions for configuring PPO in cooperative multi-agent settings.

2 Related Works

Prior MARL research commonly contrasts centralized, decentralized, and centralized-training-with-decentralized-execution approaches, while off-policy methods have often been viewed as more sample-efficient than policy-gradient methods. Existing PPO studies provide narrower or less focused evidence, motivating a broader PPO-centered empirical analysis.

  • MARL methods span centralized learning, decentralized learning, and centralized training with decentralized execution.
  • Prior comparisons reported multi-agent policy-gradient methods as less effective than MADDPG and QMix in particle-world and StarCraft benchmarks.
  • An earlier IPPO study found high success rates on difficult SMAC maps but overall performance remained worse than QMix and was limited to SMAC.
  • This paper differs by focusing on PPO and analyzing implementation and hyperparameter factors that are understudied or unique to multi-agent settings.

3 PPO in Multi-Agent Settings

The paper studies cooperative DEC-POMDPs with shared rewards, where agents act from local observations, and implements PPO using separate policy and value networks. MAPPO supplies centralized value-function inputs during training, whereas IPPO uses local inputs for both networks.

  • The setting is a decentralized partially observable Markov decision process with shared rewards, joint transitions, and agent-specific observations.
  • Each agent’s policy maps its local observation to an action, while agents jointly optimize the discounted accumulated reward.
  • PPO uses separate neural networks for the policy and value function, with the value function used during training for variance reduction.
  • MAPPO uses centralized value-function inputs under centralized training and decentralized execution, whereas IPPO uses local inputs for both policy and value function.
  • Parameter sharing is used for homogeneous agents, while the heterogeneous MPE communication task does not use parameter sharing.

4 Main Results

Across four cooperative multi-agent benchmarks, PPO-based methods achieve strong performance and competitive sample efficiency against off-policy baselines. The experiments also examine implementation choices, value-function inputs, and training budgets that shape these comparisons.

  • Testbeds and baselines: The study evaluates MAPPO and IPPO against off-policy baselines on MPE, SMAC, GRF, and Hanabi.Baselines include QMix, MADDPG, QPlex, CWQMix, AIQMix, RODE, CDS, TiKick, SAD, and VDN.
  • Overall findings: PPO achieves better or comparable results than off-policy methods in most environments with comparable sample efficiency.This is the paper’s overall empirical finding across the benchmark suite.
  • MPE: In MPE, MAPPO performs similarly to QMix across tasks and exceeds MADDPG on Comm while using a comparable number of environment steps.IPPO also matches or exceeds centralized off-policy methods, though it converges to slightly lower returns than MAPPO in Comm and Reference.
  • SMAC: In SMAC, MAPPO and IPPO perform at least as well as QMix on most maps using the same number of samples.MAPPO matches or exceeds RODE on 10 of 14 maps at equal training samples and approaches or surpasses it on nearly every map with more samples.
  • Google Research Football: In GRF, MAPPO clearly outperforms QMix in every scenario and outperforms TiKick on 4/5 scenarios.MAPPO is comparable or superior to other off-policy methods despite not using CDS’s intrinsic reward; TiKick uses pretrained models.
  • Hanabi: In Hanabi, MAPPO matches or exceeds SAD and VDN in nearly every setting, while its advantage over IPPO grows as the number of players increases.The results suggest that centralized critic inputs can become crucial in larger-player settings.

5 Factors Influential to PPO’s Performance

Five implementation and hyperparameter factors strongly influence MAPPO’s performance: value normalization, value inputs, training-data reuse, clipping, and batch size. The study translates observed trends into practical configuration guidance.

  • Value normalization: Value normalization stabilizes value learning and can improve final performance or reduce training variance.It is critical in MPE Spread, where episode returns range below -200 to 0.
  • Input representation to value function: Effective value inputs combine agent-specific local features with global information while avoiding redundant dimensions.The AS state concatenates EP and local observations; FP removes repeated features from AS.
  • Training data usage: MAPPO performance degrades when collected samples are reused too often, especially through excessive epochs or mini-batches.The study recommends at most 10 epochs on difficult environments, 15 on easy environments, and avoiding mini-batch splitting.
  • PPO clipping: Clipping strength trades off learning speed against stability, with large ϵ values often producing suboptimal performance.The recommended clipping ratio is below 0.2; smaller values can slow learning while improving stability.
  • Batch size: Batch size has a task-dependent critical threshold: undersized batches harm final performance, while oversized batches can reduce sample-efficiency.Beyond the critical setting, increasing batch size may not improve final performance and can worsen sample-efficiency.

6 Conclusion

The paper finds that properly configured PPO can match state-of-the-art performance and sample efficiency across cooperative multi-agent challenges. Its scope is limited to mostly homogeneous agents, cooperative tasks, discrete actions, and empirical analysis.

  • Conclusion: PPO achieves final returns and sample efficiency comparable to state-of-the-art methods across diverse cooperative multi-agent challenges.The authors present properly configured PPO as a competitive baseline for cooperative MARL tasks.
  • Conclusion: The study identifies five influential implementation and hyperparameter factors and provides concrete best-practice suggestions for them.These suggestions are intended to assist practitioners in achieving competitive PPO results.
  • Limitations: The benchmarks use discrete action spaces, cooperative settings, and mostly homogeneous agents, while the analysis is primarily empirical rather than theoretical.Future work is proposed for competitive games, continuous actions, heterogeneous agents, and theoretical analysis.

A MAPPO Details

MAPPO uses separate actor and critic networks, with the critic trained on global state information and the actor producing actions from observations. Recurrent versions extend the losses across time and train with backpropagation through time.

  • Recurrent-MAPPO: Recurrent-MAPPO is the algorithmic procedure used for recurrent actor and critic networks.The appendix provides pseudocode for recurrent-MAPPO.
  • Training procedure: MAPPO initializes policy and critic parameters, collects trajectories, computes GAE advantages and normalized reward-to-go values, then updates both networks with Adam.The procedure stores trajectory chunks and samples mini-batches containing all agent data.
  • Actor and critic: The critic maps global state information to a scalar value, while the actor maps observations to an action distribution.The actor outputs a categorical distribution for discrete actions or Gaussian parameters for continuous actions.
  • Optimization objectives: The actor maximizes a PPO objective using policy-ratio clipping and entropy regularization, while the critic minimizes a clipped value-loss objective.The batch size B and agent count n appear in the loss definitions; the reward-to-go is discounted.
  • Recurrent training: Recurrent losses sum across time, and recurrent networks are trained with Backpropagation Through Time.The appendix states that pseudocode for this recurrent procedure is given in Algorithm 1.

B Testing domains

The paper evaluates cooperative multi-agent tasks spanning particle-world navigation, StarCraft combat, Google Research Football, and Hanabi. Each domain requires agents to coordinate under distinct observation, action, reward, or turn-taking structures.

  • MPE: MPE comprises three fully cooperative 2D particle-world tasks: Spread, Comm, and Reference.Speaker-listener is excluded from shared-policy settings because its agents have different observation and action spaces.
  • SMAC: SMAC requires 2–27 decentralized agents to cooperate against adversarial bots across scenarios with varied team sizes.Centralized critics or Q-functions receive the global game state during training.
  • SMAC: SMAC’s agent-specific global state augments environment state with features relevant to each agent.The paper combines the environment-provided global state with each agent’s local observation for MAPPO’s critic input.
  • Hanabi: Hanabi assigns each player a forward-accumulated reward over the subsequent players’ turns.This converts rewards from sequential actions into one turn-level reward for the acting player.
  • GRF: GRF scenarios task teams of agents with scoring goals against scripted opponents, using full-state local observations and shared dense rewards.Because each local observation fully describes the environment state, MAPPO and IPPO are equivalent in these scenarios.

C Training details

Training uses shared network parameters across agents and standardized algorithm configurations, while comparing MAPPO with tuned off-policy baselines. The appendix reports that parameter sharing materially improves MAPPO on selected SMAC maps.

  • Parameter sharing: All algorithms share network parameters across agents in every environment except MPE’s Comm scenario.The Comm agents have different observation and action spaces, so separate policies are trained there.
  • Baseline configuration: Baseline MADDPG and QMix hyperparameters are tuned so their performance matches or exceeds results from the original implementations.This supports a competitive comparison rather than relying on untuned baseline settings.
  • Batching: Parameter sharing combines agents’ data, causing actual batch sizes to increase with the number of agents.MAPPO, QMix, and MADDPG use common hyperparameter settings listed separately in the training tables.
  • Hyperparameters: The training tables define recurrent chunk length, value clipping, discounting, Huber loss delta, exploration annealing, and feature normalization.Recurrent chunks are used for BPTT, while epsilon is linearly annealed between starting and ending values for epsilon-greedy exploration.
  • Parameter-sharing ablation: On selected SMAC maps, MAPPO with parameter sharing outperforms MAPPO without parameter sharing by a clear margin over 6 random seeds.Table 4 reports median evaluation win rates with standard deviations.

C.3 Death Masking

Death masking replaces a dead agent’s critic input with a fixed zero vector carrying its agent ID. The ablations favor this treatment because it improves performance and value prediction relative to alternative handling of dead agents.

  • Definition: Death masking replaces a dead agent’s value-state input with a zero state containing the agent ID.Alternatives retain the nonzero global state, drop post-death transitions, or use a pure zero-state without the agent ID.
  • Results: Death masking significantly outperforms retaining global states or dropping dead-agent transitions and consistently achieves strong performance.The comparison evaluates different dead-agent treatments using median win rate.
  • Definition: The appended agent ID makes the masked critic input agent-specific, which can reflect an agent’s type or role.This feature has been empirically useful when environments contain heterogeneous agents.
  • Motivation: When an agent dies, agent-specific global-state features become zero while agent-agnostic features remain nonzero, creating a critic-input distribution shift.Dead agents occupy about 20% of timesteps in most SMAC maps.
  • Motivation: A fixed post-death vector gives the critic one input representing the dead-agent condition, helping it fit the average post-death reward.The authors present this as an explanation supported by value-function fitting-error ablations.
  • Alternative: Skipping value learning after death is theoretically correct but can increase variance by turning a one-step estimate into a full post-death return estimate.This may reduce the benefit of GAE’s trajectory truncation and dampen its advantage after death.
  • Results: Death masking lowers value loss on the vast majority of SMAC maps, indicating more accurate value predictions.The paper notes that theoretically characterizing this effect remains valuable.

D.1 Additional SMAC Results

Additional SMAC experiments compare MAPPO variants with decentralized PPO and several off-policy baselines across maps and scenarios. The reported analyses include a delayed-convergence case and modified state inputs for comparison.

  • Results: All SMAC algorithm results are reported in Tables 5 and 6.
  • Convergence: MAPPO does not converge within 10M environment steps on the 3s5z versus 3s6z map, so its performance is shown until convergence.Figure 11 reports the median win rate after 40M environment steps.
  • State inputs: Figure 12 compares MAPPO value-input variants, IPPO, QMix, and QMix with a modified mixer-state input.QMix(MG) concatenates the default global state with all agents’ local observations.
  • Off-policy comparisons: Figure 13 compares MAPPO(FP) with QMix(MG), RODE, QPLEX, CWQMix, and AIQMix across many SMAC maps.The compared methods use agent-agnostic global state and agent-specific local observations in different parts of their architectures.
  • Cross-benchmark comparison: Figure 14 compares MAPPO with QMix, CDS, and TiKick across six Google Research Football academy scenarios.

E Ablation Studies

The ablation studies examine how value normalization, state information, training settings, death masking, and agent identity affect MAPPO performance. Additional SMAC figures report median evaluation win rates across maps and algorithms.

  • Ablation Studies: The ablations evaluate value normalization, global state information, training epochs, clipping, death masks, and agent identity in MAPPO.The studies include MAPPO(FP) and MAPPO(AS) for death-mask analyses.
  • SMAC Evaluation: The SMAC evaluation includes median win rates across 23 maps.Figure 12 reports the median evaluation win rate for the SMAC domain.
  • Ablation Studies: Separate analyses compare death-mask choices during GAE computation and examine the death mask’s effect on MAPPO’s value loss.The studies also assess whether agent identity is included in the agent-specific global state.
  • SMAC Evaluation: The SMAC comparison reports median evaluation win rates for MAPPO(FP), QMix(MG), RODE, QPlEX, CWQMix, and AIQMix.Figure 13 compares these six algorithms in the SMAC domain.
Loading 2103.01955v4…