Source-linked AI summary
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch
TL;DR
Multi-agent reinforcement learning faces non-stationarity and high variance, making coordination difficult across cooperative and competitive settings. The paper extends actor-critic methods with centralized training and decentralized execution, and reports improved performance with policies that discover complex physical and communicative coordination.
Problem
Multi-agent reinforcement learning lacks stable, low-variance methods for learning coordinated behavior as agents’ policies change during training.
Method
The method trains decentralized actors with a centralized critic that uses other agents’ observations, actions, and policies, supplemented by policy ensembles for robustness.
Results
The approach outperforms traditional reinforcement learning across cooperative and competitive environments, with agents discovering complex physical and communicative coordination strategies.
Takeaways & Limitations
Centralized training with decentralized execution supports robust multi-agent policies across mixed cooperative-competitive environments.
Takeaways & Limitations
The critic’s input space grows linearly with the number of agents, motivating future investigation of modular critics that consider only nearby agents.
Abstract
from arXiv · showhide
We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.
1 Introduction
The section motivates multi-agent reinforcement learning as a setting where co-evolving agents create non-stationarity and coordination challenges, then introduces a centralized-training actor-critic approach with decentralized execution and policy ensembles.
- Motivation: Multi-agent applications span multi-robot control, communication and language discovery, multiplayer games, social dilemmas, and related hierarchical reinforcement-learning variants.These domains involve interaction among multiple agents whose co-evolution produces emergent behavior and complexity.
- Challenges: Changing policies make the environment non-stationary for each agent, creating learning-stability challenges and preventing straightforward experience replay.This undermines a mechanism crucial for stabilizing deep Q-learning.
- Challenges: Policy-gradient methods exhibit high variance when multi-agent coordination is required, while model-based optimization requires differentiable dynamics and interaction assumptions.Competitive environments are also challenging for these approaches from an optimization perspective.
- Contributions: The proposed algorithm learns policies using only local observations at execution, requires no differentiable environment model or fixed communication structure, and supports cooperative, competitive, and mixed interactions.It is designed for both physical and communicative behavior.
- Method: Centralized training with decentralized execution lets critics use extra training information while policies remain decentralized at test time, motivating an actor-critic extension.The section contrasts this flexibility with Q-learning’s difficulty in using different information during training and testing.
- Contributions: The method models other agents’ policies online and trains policy ensembles to improve stability through robust interaction with varied collaborators and competitors.The paper reports empirical success for this approach.
2 Related Work
Prior multi-agent learning approaches struggle with non-stationarity, limited interaction assumptions, or restricted communication settings. The paper instead emphasizes agent-specific centralized critics, explicit communication, and robustness to other agents’ decision-making processes.
- Independent learning: Independent Q-learning and policy-gradient agents perform poorly because changing policies create a non-stationary environment and prevent naïve experience replay.The passage identifies independently learning agents as a simple approach that does not perform well in practice.
- Interaction settings: Many algorithms target cooperative, competitive, or mixed interactions; cooperative methods include optimistic or hysteretic Q-updates and policy-parameter sharing.Optimistic and hysteretic updates assume other agents improve collective reward, while parameter sharing requires homogeneous agents.
- Centralized critics: Compared with concurrent centralized-critic work, the paper learns one centralized critic per agent, supporting differing reward functions and competitive scenarios.The cited approach learns a single centralized critic for all agents, whereas this paper uses agent-specific critics.
- Communication: Grounded communication methods generally apply only when agents communicate through a dedicated, differentiable communication channel.Recent work studies cooperative communication protocols for solving various tasks.
- Robustness to other agents: The paper models other agents’ decision-making and seeks robustness by requiring successful interaction with an ensemble of possible policies.This modeling emphasis is connected to prior reinforcement-learning and cognitive-science work on decision-making and robustness.
3 Background
This section formalizes partially observable Markov games for multi-agent reinforcement learning and reviews key value-based, policy-gradient, and deterministic policy-gradient methods. It emphasizes that independently learning agents create non-stationarity for Q-learning, while policy-gradient estimates suffer increased variance in multi-agent settings.
- Markov Games: Partially observable Markov games model N agents with states, individual action and observation sets, stochastic policies, and state transitions.Each agent i uses a stochastic policy πθi over its observations and actions.
- Q-Learning and Deep Q-Networks (DQN): Q-learning and DQN estimate action values, with DQN stabilizing learning through target networks and experience replay.DQN minimizes a squared temporal-difference loss and uses periodically updated target parameters.
- Q-Learning and Deep Q-Networks (DQN): Independent multi-agent Q-learning violates the Markov assumptions needed for convergence because agents continually update their policies, making the environment non-stationary.The changing policies also prevent straightforward use of an experience replay buffer in general.
- Policy Gradient (PG) Algorithms: Policy-gradient methods directly optimize policy parameters, while actor-critic algorithms estimate action values with a learned critic.High gradient variance is amplified in multi-agent settings because rewards depend on many agents’ actions.
- Deterministic Policy Gradient (DPG) Algorithms: Deterministic policy gradients require continuous action spaces, and DDPG implements them with deep actor and critic networks, replay buffers, and target networks.DDPG is an off-policy algorithm that samples stored trajectories during training.
4 Methods
The methods develop MADDPG through centralized training with decentralized execution, using centralized critics that condition on all agents’ actions while policies rely only on local observations at execution. The section also introduces learned approximations of other agents’ policies and policy ensembles to address unavailable policy information and brittleness under changing competitors.
- Centralized training, decentralized execution: The approach uses centralized training with decentralized execution: critics receive extra information during training, while policies use only local observations at test time.This extra training information is permitted only if it is not used at test time.
- Centralized critics: Each agent has a centralized action-value function that takes all agents’ actions and state information, supporting arbitrary reward structures, including conflicting rewards.The state input may be all agents’ observations or additional available state information.
- MADDPG: The deterministic-policy extension is called multi-agent deep deterministic policy gradient (MADDPG), and the centralized critic with deterministic policies works well in practice.The full algorithm is provided in the Appendix.
- MADDPG: Conditioning on all agents’ actions makes the environment stationary as policies change, unlike traditional methods that do not explicitly condition on other agents’ actions.The motivation is that transition dynamics depend on the joint actions rather than directly on the changing policies once those actions are known.
- Policy approximation: Agents can learn approximations of other agents’ policies from observed actions, using entropy-regularized log-probability maximization to remove the assumption that those policies are known.The approximate policies can be updated online from replay-buffer samples, and their action log probabilities are input directly into the centralized Q function.
- Policy ensembles: To improve robustness against changing competitors, the method trains K sub-policies and randomly selects one sub-policy per agent at each episode.This addresses brittle policies that overfit competitors’ behavior and may fail when competitors alter strategies.
5 Experiments1
Experiments evaluate MADDPG in grounded communication, cooperative, competitive, and adversarial multi-agent tasks with heterogeneous observations, action spaces, and policies. Across these settings, MADDPG learns coordination that traditional methods often fail to discover and improves performance against DDPG.
- Experimental environments: The experiments use a continuous-space, discrete-time grounded communication environment where agents can take physical and broadcast communication actions without requiring identical spaces or policies.The benchmark includes cooperative communication, predator-prey, cooperative navigation, physical deception, keep-away, and covert communication tasks.
- Cooperative communication: Traditional RL methods fail at cooperative communication, with the listener ignoring the speaker and moving toward the middle of the observed landmarks.Critic, first-order TRPO, and DDPG fail to learn the correct behavior, measured by whether the listener reaches the target landmark.
- Cooperative communication: MADDPG reliably learns coordinated speaker-listener policies, with the listener navigating to the target 84.0% of the time.The centralized critic provides a coordination signal that makes the task easier to learn.
- Physical deception: MADDPG cooperating agents deceive adversaries by covering all landmarks around 94% of the time when L = 2, while DDPG-trained adversaries succeed only 16.4%.Cooperating DDPG agents cannot deceive MADDPG adversaries as effectively.
- Cooperative navigation and predator-prey: MADDPG outperforms DDPG in cooperative navigation and predator-prey, including almost half as many collisions in navigation and 16.1 versus 10.3 collisions per episode against prey.In cooperative navigation, MADDPG also achieves a slightly smaller average distance to each landmark.
- Covert communication: In covert communication, Bob’s relative success rate is 52.4% with MADDPG versus 25.1% with DDPG, and only MADDPG-trained Alice makes Eve’s reconstruction near-random.Both MADDPG- and DDPG-trained Bob outperform Eve at reconstructing Alice’s message.
6 Conclusions and Future Work
The paper proposes a multi-agent policy-gradient algorithm with a centralized critic using all agents’ observations and actions, outperforming traditional reinforcement-learning algorithms across cooperative and competitive environments. Policy ensembles can further improve performance, while the critic’s input space grows linearly with the number of agents and motivates modular-Q future work.
- Conclusions: The proposed algorithm trains agents with a centralized critic based on all agents’ observations and actions, outperforming traditional reinforcement-learning algorithms across cooperative and competitive environments.The empirical comparison covers a variety of multi-agent environments.
- Conclusions and Future Work: Training agents with an ensemble of policies can further improve the method’s performance and may apply generally to multi-agent algorithms.The paper presents policy ensembles as a broadly applicable improvement strategy.
- Future Work: The critic’s Q-function input space grows linearly with the number of agents N, depending on the information contained in x.A modular Q function that considers only nearby agents is proposed as a practical remedy, left for future work.
Appendix · Multi-Agent Deep Deterministic Policy Gradient Algorithm
The appendix presents the Multi-Agent Deep Deterministic Policy Gradient algorithm for N agents, covering exploration, replay-buffer training, actor updates, and target-network updates. Its pseudocode proceeds from state initialization through action execution, minibatch sampling, gradient updates, and target-parameter updates.
- Appendix: The appendix provides the Multi-Agent Deep Deterministic Policy Gradient algorithm for N agents.The algorithm is identified as Algorithm 1 and is provided for completeness.
- Multi-Agent Deep Deterministic Policy Gradient Algorithm: The procedure initializes a random process N for action exploration, receives state x, and iterates to max-episode-length.
- Multi-Agent Deep Deterministic Policy Gradient Algorithm: Each agent i selects ai = µθi(oi) + Nt, executes the joint action, observes r and x′, and stores (x, a, r, x′) in replay buffer D.The current state is then replaced with x′ before agent-specific updates.
- Multi-Agent Deep Deterministic Policy Gradient Algorithm: The algorithm samples a random minibatch of S samples (xj, aj, rj, x′j) from D and sets yj = rj.
- Multi-Agent Deep Deterministic Policy Gradient Algorithm: The pseudocode continues with next-action and sampled-action expressions before updating the actor using the sampled policy gradient.These expressions appear as continuation lines in the algorithm listing.
- Multi-Agent Deep Deterministic Policy Gradient Algorithm: After the agent loop, the procedure updates target-network parameters for each agent i.
Experimental Results
The experiments use specified optimization and training settings, with results reported across cooperative communication, navigation, predator-prey, physical deception, and covert communication tasks. The tables evaluate target-reaching, distance, collisions, prey touches, and communication success under the described conditions.
- Experimental setup: Networks are updated after batches of 1024 episodes, except TRPO, which uses batches of 50 episodes for more updates relative to MADDPG.The passage attributes TRPO’s smaller batch size to better performance.
- Cooperative communication: Table 1 evaluates target-reaching percentages and average distance from the target in cooperative communication after 25000 episodes.Target-reaching percentage differs from policy-learning success rate because agents may hover slightly outside the landmark even after learning the correct behavior.
- Cooperative navigation: Table 2 reports average collisions per episode and average agent distance from a landmark in cooperative navigation using 2-layer 128-unit MLP policies.The table’s metrics jointly assess collision avoidance and landmark proximity.
- Predator-prey: Table 3 measures average predator prey touches per episode in PP1 and PP2 predator-prey environments with N = L = 3 and differing prey speeds.Prey are 30% faster in PP1 and 100% faster in PP2; all policies are 2-layer 128-unit MLPs.
- Deception and communication: Tables 4 and 5 report success rates for physical deception and covert communication, respectively, including agent-adversary outcomes and message reconstruction.Physical deception uses N = 2 and 4 cooperative agents/landmarks; covert communication uses two 4-dimensional one-hot input messages.
Variance of Policy Gradient Algorithms in a Simple Multi-Agent Setting
In a simple cooperative binary-action setting, policy-gradient learning becomes exponentially less likely to move in the reward-improving direction as the number of agents grows. Although gradient variance can decrease, the shrinking expected gradient lowers the signal-to-noise ratio, while conditioning a critic on other agents’ actions removes uncertainty.
- Setting: The scenario has N cooperative agents with binary actions, rewarding 1 only when all actions are identical and 0 otherwise.Agents must learn to always output either 1 or 0 at each time step, with no temporal component.
- Gradient-direction probability: The probability of taking a policy-gradient step in the correct direction is (0.5)^N, so it decreases exponentially as the agent count grows.This difficulty is especially relevant to sparse-reward environments.
- Gradient quality: As N grows, policy-gradient variance can decrease, but the expected gradient also decreases, lowering the signal-to-noise ratio.The reduced signal-to-noise ratio corresponds to the declining probability of selecting a correct gradient direction.
- Centralized critic: A centralized critic reduces gradient uncertainty by conditioning on other agents’ actions, which removes variability caused by unobserved actions.Conditioning only on an agent’s own actions leaves substantial variability associated with the other agents.