PaperlayerView exact revision on arXiv

Source-linked AI summary

FACMAC: Factored Multi-Agent Centralised Policy Gradients

Bei Peng, Tabish Rashid, Christian A. Schroeder de Witt, Pierre-Alexandre Kamienny, Philip H. S. Torr, Wendelin Böhmer, Shimon Whiteson

arXiv:2003.06709v5cs.LGcs.AIstat.ML

TL;DR

Cooperative MARL actor-critic methods can lag behind value-based methods on challenging tasks, motivating FACMAC’s factored critic and joint-action policy-gradient design. Across continuous and discrete benchmarks, FACMAC outperforms MADDPG and other baselines, while nonmonotonic factorisation solves some otherwise difficult tasks but can increase learning difficulty.

  • Problem

    Actor-critic methods can significantly underperform value-based methods such as QMIX on challenging SMAC tasks, motivating improved cooperative MARL methods.

  • Method

    FACMAC learns decentralised policies with a centralised factored critic and a centralised gradient estimator that optimises over the joint action space, including nonmonotonic factorisations.

  • Results

    FACMAC outperforms MADDPG and other baselines across continuous particle and MAMuJoCo tasks and discrete SMAC tasks.

  • Takeaways & Limitations

    Nonmonotonic factored critics can solve tasks that monolithic or monotonically factored critics cannot, while factoring and centralised gradient estimation provide demonstrated advantages.

  • Takeaways & Limitations

    On harder SMAC maps, FACMAC-nonmonotonic performs significantly worse than FACMAC, indicating that unconstrained factorisation can increase learning difficulty.

Abstract

from arXiv · show

We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent Q-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

1 Introduction

FACMAC extends cooperative multi-agent actor-critic learning with a centralised factored critic and joint-action policy-gradient estimation for discrete and continuous tasks. It is evaluated across continuous particle and MuJoCo domains and discrete SMAC tasks, where it outperforms the cited baselines.

  • Actor-critic methods can significantly underperform value-based methods such as QMIX on the challenging SMAC benchmark.
  • Its centralised factored critic combines per-agent utilities through a non-linear monotonic function, while allowing unconstrained nonmonotonic factorisations.Unlike QMIX, FACMAC has no inherent constraints on factoring the critic.
  • FACMAC’s centralised gradient estimator optimises the entire joint action space instead of each agent’s action space separately, enabling more coordinated policy changes.
  • MAMuJoCo broadens continuous MARL evaluation beyond simple particle environments with cooperative robotic-control tasks involving multiple agents within one robot.
  • FACMAC outperforms MADDPG and other baselines across particle, MAMuJoCo, and SMAC domains, with better scaling as agent count, action count, or task complexity increases.

2 Background

The paper formulates cooperative MARL as a Dec-POMDP under CTDE: training may use global information, but execution must rely on each agent’s local history. It contrasts centralised actor-critic critics with factored value-based critics and their decentralisability constraints.

  • A cooperative Dec-POMDP models agents sharing a goal, jointly selecting actions that induce state transitions and team rewards under partial observability.
  • Agents may use stochastic policies πa(ua|τa) or deterministic policies µa(τa), each conditioned only on its local action-observation history.
  • Under CTDE, policy training can use global information and shared training information, while execution restricts each agent to its own action-observation history.
  • VDN and QMIX: VDN and QMIX factor centralised action values for cooperative discrete-action tasks, assuming additivity and monotonicity respectively.
  • VDN and QMIX: QMIX’s monotonic mixing function uses non-negative weights so global greedy action selection agrees with independent per-agent greedy selections.
  • MADDPG: MADDPG learns separate deterministic actors and centralised monolithic critics for agents, supporting cooperative, competitive, and mixed reward settings.
  • Centralised critics: Centralised critics estimate joint action values using global state and all agents’ actions during training; if unavailable, joint observations or histories may substitute for the global state.

3 FACMAC

FACMAC combines a centralised but factored critic with a centralised policy-gradient estimator for coordinated cooperative MARL in discrete and continuous action spaces. It also supports nonmonotonic critic factorisation and differentiable discrete-action training.

  • Centralised but Factored Critic: FACMAC learns decentralised policies with a shared centralised critic that factors Qtot into per-agent utilities combined by a nonlinear monotonic mixing function.The canonical FACMAC critic uses a QMIX-style mixing network.
  • Centralised but Factored Critic: FACMAC-nonmonotonic removes monotonicity constraints from the mixing network, providing full representational capacity for the joint action-value function.The method uses unconstrained mixing-network weights rather than non-negative weights.
  • Centralised but Factored Critic: FACMAC also includes VDN-style sum factorisations and a state-dependent-bias variant, and its factorisation technique can be applied to centralised monolithic critics in other actor-critic algorithms.The approach is presented as general beyond the canonical mixing function.
  • Centralised Policy Gradients: FACMAC’s centralised gradient estimator optimises the entire joint action space and samples all agents’ actions from their current policies to improve coordination and avoid relative overgeneralisation.This differs from per-agent gradients that hold other actions fixed or sample them from replay.
  • Centralised Policy Gradients: On the continuous matrix game, MADDPG reaches a local optimum within 200k timesteps, whereas MADDPG with the centralised policy gradient robustly converges to the optimal policy.The comparison is designed to isolate the benefit of the centralised policy-gradient estimator.
  • Discrete Actions: For discrete actions, FACMAC uses Straight-Through Gumbel-Softmax, sampling categorical actions in the forward pass and differentiable relaxed samples in backpropagation.This keeps training and evaluation action dynamics aligned while enabling gradient-based updates.

4 Multi-Agent MuJoCo

MAMuJoCo is a benchmark for continuous cooperative multi-agent robotic control that partitions single-robot bodies into controllable agent subgraphs. It includes partial-observability motivations and scenarios with configurable numbers of agents.

  • Benchmark Motivation: MAMuJoCo extends single-agent MuJoCo control tasks into diverse continuous cooperative multi-agent robotic-control scenarios.The benchmark is intended to address the limited diversity of continuous MARL evaluations.
  • Agent Partitioning: Figure 4 maps agent partitions across swimmer, hopper, HalfCheetah, humanoid, walker, reacher, and ant environments using colors and agent-by-joint dimensions.Square brackets encode the number of agents multiplied by joints per agent.
  • Agent Partitioning: Each MAMuJoCo scenario partitions a robot’s body graph into disjoint subgraphs, assigning one or more controllable joints to each agent.The body graph represents joints as vertices connected by adjacent body segments.
  • Agent Partitioning: MAMuJoCo introduces multiple agents within one robot to model partial observability from latency, bandwidth limits, noisy sensors, and communication failures.The benchmark emphasizes local decision rules when the exact full state is unavailable.
  • Scalable Scenarios: ManyAgent Swimmer and ManyAgent Ant allow an arbitrarily large number of agents within memory limits, drawing on modular-robotics settings.Modular robotics motivates configurable and scalable degrees of freedom.

5 Experimental Results

FACMAC is evaluated across continuous particle and MAMuJoCo tasks and discrete SMAC maps, where it generally outperforms MADDPG and other baselines. Ablations and nonmonotonic-factorisation experiments show that critic factorisation and centralised policy gradients matter especially as coordination problems grow more complex.

  • Overall performance: FACMAC significantly outperforms MADDPG on continuous cooperative tasks and beats or matches major baselines across discrete SMAC maps.On SMAC, it outperforms MADDPG on 4 of 6 maps, all tested actor-critic baselines, and is competitive with QMIX and QPLEX.
  • Scalability: FACMAC scales better than MADDPG as agent count, action count, and task complexity increase.The largest gaps occur in high-agent Continuous Predator-Prey, 10-agent ManyAgent Swimmer, and difficult SMAC maps.
  • Ablations: Critic factorisation improves performance and learning speed on challenging coordination tasks compared with a monolithic critic.FACMAC without centralised policy gradients still outperforms MADDPG on both tested SMAC maps, while FACMAC performs best on MMM2.
  • Ablations: Centralised policy gradients provide additional gains when a factored critic is used with many agents.The estimator has little effect with 2 ManyAgent Swimmer agents but significantly improves learning with 10 agents.
  • Nonmonotonic factorisation: Nonmonotonically factored critics solve the continuous matrix game and nonmonotonic Predator-Prey variant when monolithic or monotonic critics fail.FACMAC-nonmonotonic robustly learns the optimal matrix-game policy and successfully captures the prey in the variant task.
  • Nonmonotonic factorisation: The relative performance of FACMAC and FACMAC-nonmonotonic is task dependent, with the unconstrained variant performing worse on harder SMAC maps.They perform similarly on the original Continuous Predator-Prey task and easy SMAC maps, but FACMAC-nonmonotonic is significantly worse on harder maps.

6 Related Work

Prior work factors joint action-values to support decentralised execution, but actor-critic methods can struggle with coordination and complex tasks. FACMAC addresses these issues with an unconstrained factored centralised critic and a centralised gradient estimator.

  • VDN and QMIX factor the joint action-value function into per-agent utilities using summation or monotonic mixing to preserve decentralised arg max consistency.
  • Monotonicity limits the representational capacity of factored value functions, while QTRAN and QPLEX pursue less restricted alternatives with scalability or task-solving limitations.
  • FACMAC uses a centralised critic that can be factored without constraints, providing a direct way to handle nonmonotonic tasks and continuous actions.
  • Centralised critics alone do not guarantee coordinated behaviour because per-agent policy gradients can leave agents in mutually suboptimal solutions.
  • Unlike related centralised-critic approaches, FACMAC factorises the critic similarly to QMIX for more efficient learning on complex tasks.

7 Conclusion

The paper presents FACMAC as a cooperative multi-agent actor-critic method for discrete and continuous tasks, combining decentralised policies with a centralised factored critic. Across three domains, the reported results show superior performance over existing MARL algorithms, while future work targets richer nonmonotonic factorisations.

  • FACMAC learns decentralised policies with a centralised but factored critic for discrete and continuous cooperative tasks.
  • The paper reports advantages from both factoring the critic and using the centralised gradient estimator.
  • MAMuJoCo is introduced as a benchmark for demonstrating scalability to more complex continuous tasks.
  • Results across three domains show superior performance over existing MARL algorithms.
  • Future work will explore additional nonmonotonic factorisations for tasks with nonmonotonic value functions.

A Multi-Agent MuJoCo

MAMuJoCo is introduced to address the limited diversity and decentralisability of existing continuous MARL benchmarks. It creates cooperative multi-agent robotic-control scenarios from MuJoCo, with configurable observations and agent-specific motor actions.

  • A Multi-Agent MuJoCo: Existing continuous MARL benchmarks are rarely diverse, fully cooperative, decentralisable, and partially observable at the same time.
  • A Multi-Agent MuJoCo: MAMuJoCo develops diverse continuous cooperative robotic-control tasks by placing multiple agents within a single robot.
  • A Multi-Agent MuJoCo: MAMuJoCo extends single-agent MuJoCo into a broad suite of multi-agent robotic scenarios, unlike narrower related environments.
  • A Multi-Agent MuJoCo: Each agent controls the motors assigned to it, so its action is a vector whose dimensions equal the number of controllable joints.
  • A Multi-Agent MuJoCo: Observations are built from joints and body segments within the agent’s own subgraph and a configurable graph-distance radius.
  • A Multi-Agent MuJoCo: Restricting observation distance and element categories creates partial observability while preserving collaborative versions of the original task goals.
  • A Multi-Agent MuJoCo: Continuous joint-action maximisation is difficult because joint action spaces grow exponentially with agent count, making enumeration and direct Qtot training impractical.
  • A Multi-Agent MuJoCo: COVDN and COMIX address continuous actions by factoring Qtot into per-agent utilities and using CEM for approximate greedy action selection.

C.1 Continuous Predator-Prey

Continuous Predator-Prey uses cooperating predators to catch a faster prey in a toroidal environment with obstacles. Variants introduce limited visibility and a nonmonotonic reward that requires coordinated captures.

  • C.1 Continuous Predator-Prey: Three slower cooperating predators with continuous two-dimensional movement must catch a faster prey on a toroidal plane containing obstacles.
  • C.1 Continuous Predator-Prey: The prey follows a heuristic that selects the sampled position farthest from the closest predator, and a collision yields a team reward of +10.
  • C.1 Continuous Predator-Prey: The original task provides each agent with an almost complete representation of the environment through relative entity positions, velocities, and landmark locations.
  • C.1 Continuous Predator-Prey: A view-radius restriction makes other agents observable only roughly 60% of the time, introducing partial observability.
  • C.1 Continuous Predator-Prey: The nonmonotonic variant rewards a coordinated capture with +10 but penalises an isolated capture with −1.

C.2 Multi-Agent MuJoCo

The Multi-Agent MuJoCo experiments use default partial-observation configurations and include a 10-agent, 2-segment-per-agent ManyAgent Swimmer variant.

  • MAMuJoCo agents observe positions of their own body parts and body parts at graph distances greater than zero, but not velocities.
  • ManyAgent Swimmer uses 10 agents, each controlling a consecutive segment of length 2, denoted ManyAgent Swimmer [10x2].
  • SMAC evaluates decentralised multi-agent control through allied agents fighting enemy units controlled by built-in heuristic AI, with win rate as the objective.

D Experimental Details

Experiments use fixed evaluation schedules, recurrent or multilayer perceptron agent architectures, and method-specific training configurations across the benchmark domains.

  • Evaluation: Methods are evaluated periodically using independent test episodes with greedy decentralised action selection.Evaluation pauses occur every 2000, 4000, or 10000 timesteps depending on the environment, using 10 or 32 episodes.
  • Network architectures: Actor-critic methods use shared DRQN agent networks with a 64-dimensional GRU hidden state, while their final output layer is method-specific.
  • Training configuration: Episodes last 25 steps, training lasts 2 million timesteps, exploration uses Gaussian noise with σ = 0.1, and γ = 0.85.The replay buffer stores the most recent 10^6 transitions, batches contain 1024 transitions, and target updates use τ = 0.001.
  • Network architectures: Actor-critic shared agent and critic networks use multilayer perceptrons with hidden layers of 400 and 300 units, and actor outputs use tanh bounding.
  • Training configuration: Some experiments use 1000-step episodes and 2 million or 4 million timesteps, with Gaussian noise and 10000 initial uniformly random-action steps for exploration.
  • Method-specific configurations: FACMAC and MADDPG use shared recurrent actors, 64-unit two-layer critics, and COMA-like exploration, while baseline methods use author-provided SMAC training setups.
  • Method-specific configurations: FACMAC’s mixing network has one 32-unit ELU hidden layer, with nonnegative weights generated by 64-unit ReLU hypernetworks and absolute activation.

E Additional Results on Different Critic Factorisations

The additional results examine critic-factorisation choices using continuous predator-prey returns and SMAC win rates across maps of varying difficulty.

  • Continuous Predator-Prey: Figure 12 reports mean episode return for the Continuous Predator-Prey task with 3 agents and 1 prey.
  • SMAC: Figure 13 compares FACMAC critic-factorisation variants using median test win percentage across six SMAC maps.The maps range from easy to super hard and include 2s3z, MMM, 2c_vs_64zg, bane_vs_bane, MMM2, and 27m_vs_30m.
Loading 2003.06709v5…