PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning

Xueguang Lyu, Yuchen Xiao, Brett Daley, Christopher Amato

arXiv:2102.04402v2cs.LGcs.AI

TL;DR

The implications of centralized critics in CTDE remain insufficiently understood, despite their popularity and the perception that they offer benefits without compromise. The paper analyzes centralized and decentralized critics theoretically and empirically, finding that neither is generally dominant and that decentralized critics often perform more robustly in the tested domains.

  • Problem

    Although centralized critics are popular in CTDE, their implications remain poorly understood and their presumed benefits have not been formally proven or empirically tested.

    Source: §1 · p. 1

  • Method

    The paper theoretically analyzes centralized and decentralized critics and empirically compares them across environments to examine gradient, variance, and practical performance differences.

    Source: §1 · p. 1, §6 · p. 5

  • Results

    In theory, both critic types give decentralized policies the same expected gradient, while centralized critics yield higher-variance policy updates; in the tested domains, decentralized critics regularly provide more robust performance.

    Source: §1 · p. 1, §8 · p. 8

  • Takeaways & Limitations

    Neither critic type is expected to dominate performance in general, so their practical bias-variance and scalability trade-offs matter when choosing a design.

    Source: §6 · p. 5, §6.3 · p. 6

    The theoretical analysis assumes critics converge to true on-policy values, an assumption the authors describe as unrealistic.

    Source: §4.1 · p. 3

Abstract

from arXiv · show

Centralized Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learning community. In particular, actor-critic methods with a centralized critic and decentralized actors are a common instance of this idea. However, the implications of using a centralized critic in this context are not fully discussed and understood even though it is the standard choice of many algorithms. We therefore formally analyze centralized and decentralized critic approaches, providing a deeper understanding of the implications of critic choice. Because our theory makes unrealistic assumptions, we also empirically compare the centralized and decentralized critic methods over a wide set of environments to validate our theories and to provide practical advice. We show that there exist misconceptions regarding centralized critics in the current literature and show that the centralized critic design is not strictly beneficial, but rather both centralized and decentralized critics have different pros and cons that should be taken into account by algorithm designers.

1 INTRODUCTION

The paper challenges the assumption that centralized critics are an uncompromised default in CTDE and examines their effects on cooperation and policy learning. Its theory and empirical comparisons show that critic choice involves trade-offs, with decentralized critics often outperforming centralized critics in the tested domains.

1 INTRODUCTION

  • Prior work often treated centralized critics as an uncompromised CTDE default, while claims of improved cooperation, stability, or speed lacked formal proof or empirical tests.The paper also notes that related work commonly focused on other problems rather than comparing centralized and decentralized critics.Source: §1 · p. 1
  • Centralized critics and decentralized critics provide decentralized policies with the same expected gradients, so centralization does not necessarily improve cooperation.The paper reports validation on classical cooperation games and more realistic domains.Source: §1 · p. 1
  • Assuming converged on-policy value functions, centralized critics produce higher-variance actor updates and can make policy learning less stable.The paper distinguishes stability of value-function learning from variance in policy learning and identifies a practical bias-variance trade-off.Source: §1 · p. 1
  • Decentralized critics often outperform centralized critics in standard implementations across popular domains, motivating practical comparison rather than assuming one design dominates.The comparison uses history-based critics for both approaches to keep the information type consistent.Source: §1 · p. 1
  • The authors analyze performance differences to identify room for improvement in current methods and lay theoretical groundwork for future work.The introduction also describes testing standard implementations over a wide range of popular domains.Source: §1 · pp. 1–2

2 RELATED WORK

Prior multi-agent actor-critic work has used centralized critics for varied objectives, but has not specifically investigated whether centralizing the critic is effective. This paper extends earlier convergence analyses with a comparison of critic choices and their bias-variance properties.

2 RELATED WORK

  • COMA [11] related centralized-critic policy gradients to single-agent actor-critic convergence, while this paper extends convergence analysis to both critic types and their policies.The paper also provides a detailed bias-variance analysis.Source: §2 · p. 2
  • Prior work used centralized critics for credit assignment, competition, communication, exploration, transfer, and emergent tool use, but did not specifically investigate their effectiveness.Examples include COMA [11], MADDPG [21], M3DDPG, and MAAC [13].Source: §2 · p. 2

3 BACKGROUND

The background frames cooperative MARL with decentralized execution and partial observability, then introduces actor-critic methods and three multi-agent extensions. These extensions differ in whether actors and critics use joint or agent-local information.

3 BACKGROUND

  • The background introduces cooperative multi-agent reinforcement learning with decentralized execution and partial observability, alongside single-agent actor-critic and its multi-agent extensions.The section presents this material as the formal problem setting and method background.Source: §3 · p. 2

3.1 Dec-POMDPs

  • A Dec-POMDP models decentralized agents with partial observability using agents, states, actions, transitions, rewards, observations, and observation probabilities.Agents receive local observations while the joint return is determined by the shared environment.Source: §3.1 · p. 2

3.2 Actor Critic method (AC)

  • Actor-critic methods learn a value function, or critic, and use its return estimates to guide policy-gradient updates to the actor.The policy-gradient direction is the gradient of expected return.Source: §3.2 · p. 2

3.3 Multi-Agent Actor Critic methods

  • The multi-agent extensions include joint actor and critic learning (JAC), local actors and critics (IAC), and decentralized actors updated with a joint critic (IACC).IACC’s joint return estimate updates each decentralized policy using joint action-history information, unlike the local critic’s expectation.Source: §3.3 · pp. 2–3

4 BIAS ANALYSIS

With converged critics, centralized and decentralized critics yield equal expected policy gradients. The Climb Game shows that this equivalence does not remove action shadowing.

4.1 Equivalence in Expected Gradient

  • After critic convergence, centralized and decentralized critics produce equal expected policy gradients under the theorem’s assumptions.The proof relates the decentralized critic’s fixed point to the centralized critic’s marginal expectation.Source: §4.1 · p. 3
  • The theory assumes critics converge to their true on-policy values, an assumption the authors call unrealistic.They note that convergence assumptions often correspond to infinitesimal actor step sizes.Source: §4.1 · p. 3

4.2 Climb Game as an Intuitive Example

  • In the Climb Game, both methods converge to the suboptimal ⟨a3,a3⟩ solution despite the centralized critic learning the optimal joint-action value.Under uniform policies, action u1 has an expected return of about −6.3 for IAC, while IACC’s sampled gradients also leave it shadowed.Source: §4.2 · p. 4

5 VARIANCE ANALYSIS

The variance analysis shows that centralized critics can increase policy-gradient variance, with added stochasticity arising from teammates’ actions and, under partial observability, their histories.

5 VARIANCE ANALYSIS

  • With converged value functions and shared policy parameterization, centralized-critic policy-gradient variance is at least decentralized-critic variance along every dimension.The theorem excludes variance from value-function learning.Source: §5 · p. 4

5.1 Multi-Action Variance (MAV)

  • Multi-Action Variance arises because a centralized critic’s value for an agent’s action varies with teammates’ sampled actions, while a decentralized critic averages over them.The centralized critic transfers this teammate-action stochasticity into policy updates.Source: §5.1 · p. 5

5.1.1 The Morning Game and the MAV.

  • In the Morning Game, the centralized critic has zero value-estimate variance, but the decentralized critic makes agent 1’s update direction less variable toward cereal.The centralized critic distinguishes cereal with milk (return 3) from cereal with vodka (return 0), while the decentralized critic averages these outcomes.Source: §5.1.1 · p. 5

5.2 Multi-Observation Variance (MOV)

  • In the partially observable thought experiment, the decentralized critic estimates Q(h_i,a_i) = 0 with zero variance, while the centralized critic’s return estimates have variance r^2.The difference arises because the centralized critic distinguishes teammate histories that the decentralized agent averages over.Source: §5.2 · p. 5

6 DISCUSSION

The discussion frames critic choice as a practical trade-off: both methods share expected policy gradients, but differ in practical bias, variance handling, and scalability, so neither is expected to dominate generally.

6.1 Value Function Learning

  • Although both critics learn on-policy values subject to non-stationarity, centralized value learning has zero variance in update targets and may yield more stable bootstrapping.The authors expect this stability may make centralized critics less biased in cooperative environments with a moderate number of agents.Source: §6.1 · p. 6

6.2 Handling MAV and MOV

  • Decentralized critics average over other agents’ information during critic learning, whereas centralized critics leave that averaging to policy updates, shifting bias and variance between stages.The authors expect decentralized critics to have more bias and less variance, and centralized critics less bias and more variance.Source: §6.2 · p. 6

6.3 Scalability

  • Centralized critic representations scale at least linearly with agent count, while decentralized critics can keep feature counts constant and may share parameters in homogeneous systems.When agents’ decisions rarely depend on others’ trajectories, joint value learning may add little; decentralized critics are expected to perform better with better sample efficiency.Source: §6.3 · p. 6
  • As agent count grows, centralized policy-gradient variance can rise while decentralized critic bootstrapping targets may become less stable, making scalability domain- and approximation-dependent.The authors suggest IAC as a better starting point because its policy updates are more stable and its parameters may be shared.Source: §6.3 · p. 6

6.4 The Overall Trade-off

  • Critic centralization trades lower value-function bias for higher policy-update variance, and its best choice depends on the environment.The discussion also identifies more severe scalability issues for centralized critics, so neither method is expected to dominate overall.Source: §6.4 · p. 6

7 EXPERIMENTS AND ANALYSIS

The experiments compare centralized and decentralized critics across varied MARL domains, using grid-searched hyperparameters and typically aggregating 20 runs per method. Overall performance is similar in most tested domains, but centralized-critic runs are less stable in some settings and lose ground as observation spaces grow.

7 EXPERIMENTS AND ANALYSIS

  • The evaluation spans popular MARL domains, uses grid search for hyperparameter tuning, and aggregates 20 runs per method unless otherwise specified.Tested domains include classical matrix games, StarCraft Multi-Agent Challenge, Particle Environments, and the MARL Environments Compilation.Source: §7 · p. 6

7.1 Variance in the Morning Game

  • IAC has zero per-rollout policy-gradient variance in the Morning Game, while IACC has non-zero variance; both critics converge to the correct optimal-action value of 3.IACC’s extra Q-value variance comes from incorporating teammates’ actions; IAC’s variance instead reflects critic biases across trials.Source: §7.1 · p. 6, Figure 2 · p. 7

7.3 Shadowed Equilibrium

  • In Move Box, both methods select the safe +10 destination over the joint +100 optimum, which remains shadowed even with IACC’s centralized critic.The joint optimum is rarely sampled when an uncooperative teammate makes its on-policy return extremely low.Source: Figure 3 · p. 7, §7.3 · pp. 7–8

7.2.1 Dec-Tiger.

  • In Dec-Tiger, high-variance IACC updates lead agents to avoid punishment but forgo the +20 coordinated return, while IAC performs better.IACC settles on opening an agreed-upon door immediately rather than listening and coordinating on the treasure door.Source: §7.2.1 · p. 7, Figure 3 · p. 7

7.2.2 Cleaner.

  • In Cleaner, IACC can send both agents along the longer path and converge on a suboptimal solution, whereas high-variance updates make policy coordination volatile.The optimal behavior is for agents to split up and cover the maze; IACC’s safer option can lead both agents to ignore the other path.Source: §7.2.2 · p. 7, Figure 3 · p. 7

7.4 Robustness in Performance

  • Across most tested domains IAC and IACC perform similarly, but IACC is less stable in a few; continuous-navigation tasks and larger Capture Target grids expose greater IACC difficulty.In Capture Target, increasing the grid from 4 × 4 to 12 × 12 produces a notable comparative drop in IACC performance.Source: §7.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

8 CONCLUSION

The paper concludes that centralized and decentralized critics have equal expected policy gradients in theory, but centralized critics increase update variance and decentralized critics often perform more robustly in the tested domains.

8 CONCLUSION

  • In practice, decentralized critics regularly give more robust performance, while centralized critics have higher policy-update variance and decentralized critics may have greater bias with limited samples.The authors report that stable policy gradients appear more crucial than stable value functions in their domains.Source: §8 · p. 8

A.1 Proposition 1: Existence of Steady-State History Distribution

  • Under stationary observation and transition models, irreducible and aperiodic chains, and a fixed policy, a stationary distribution over state-history pairs can be obtained.The history-conditioned state distribution is normalized from expected time spent in each state-history pair.Source: §A.1 · p. 11

A.2 Lemma 1: Convergence of Centralized Critic

  • With infinite visits and a suitably annealed learning rate, the centralized critic update converges to its Bellman fixed point, Qπ.The update operator is a contraction when γ < 1, yielding a unique fixed point identified as Qπ.Source: §A.2 · pp. 11–12

A.3 Lemma 2: Convergence of Decentralized Critic

  • The decentralized critic update is also a contraction when γ < 1, implying a unique fixed point for its value update.The proof considers the two-agent case and states that it generalizes to n agents.Source: §A.3 · p. 12

A.4 Theorem 1: Unbiased Policy Gradient

After both critics converge, their expected policy gradients for an agent are equal, so either critic can be used without introducing bias.

A.4 Theorem 1: Unbiased Policy Gradient

  • The centralized and decentralized critics yield equal expected policy gradients for an agent after convergence, so they can be interchanged without introducing bias.The centralized expectation over joint histories and actions reduces to the decentralized expectation over the agent's history and action.Source: §A.4 · p. 13

A.5 Theorem 2: Policy Gradient Variance Inequality

The variance analysis shows that centralized critics produce policy-gradient covariance no smaller than decentralized critics; the Morning Game figure plots individual runs.

A.5 Theorem 2: Policy Gradient Variance Inequality

  • Every element of the centralized critic's policy-gradient covariance matrix is at least as large as the corresponding decentralized critic element.The comparison follows from subtracting the variances after establishing equal expected gradients.Source: §A.5 · p. 13

B MORNING GAME

  • Figure 6 plots individual-run Morning Game Q-values used to update π(a2).Source: Figure 6 · p. 14

C GUESS GAME

Across the supplied empirical passages, the Guess Game illustrates instability with centralized critics, while the remaining passages describe the grid-world, cooperative-navigation, and partially observable predator-and-prey tasks.

C GUESS GAME

  • Decentralized critics robustly reach the Guess Game's optimal action a3 for return 5, while centralized critics are less stable as joint-observation variance grows.Figure 7 identifies the simple case where a centralized critic cannot bias actors toward the global optimum; larger batches can mitigate the variance effect.Source: C GUESS GAME · p. 14, Figure 7 · p. 14, Figure 8 · p. 15

D GRID WORLD ENVIRONMENTS

D.1 Go Together

  • In Go Together, two fully observed agents seek the same goal for reward while avoiding penalties for being too close or too far apart.Source: §D.1 · p. 15

D.2 Find Treasure

  • In Find Treasure, one agent must operate a switch while the other passes through a door to reach a treasure worth +100.Source: §D.2 · p. 15

D.3 Cleaner

  • In Cleaner, agents earn +1 for untouched ground and should cover cells while avoiding their teammate's path.The passage describes an optimal policy as navigating to as many cells as possible.Source: §D.3 · p. 15, Figure 9 · p. 15

D.4 Move Box

  • In Move Box, two agents must carry a heavy box together, choosing between a nearer +10 destination and a farther +100 destination.Source: §D.4 · p. 15, Figure 10 · p. 16

E COOPERATIVE NAVIGATION

F PARTIALLY OBSERVABLE PREDATOR AND PREY

G CAPTURE TARGET

These subsections describe Capture Target, Small Box Pushing, its three-agent variant, and selected SMAC scenarios, with figures identifying domain configurations and performance comparisons.

G CAPTURE TARGET

  • Capture Target has two agents pursuing a moving target on an m×m toroidal grid, with capture rewarded only when both agents reach its cell.The target moves right deterministically; agents have five movement options with 0.1 probability of accidentally reaching an adjacent cell, and the target is blurred with probability 0.3. Episodes last 60 steps, and Figures 15–16 show domain configurations and comparisons across grid sizes.Source: G CAPTURE TARGET · p. 18, Figure 15 · p. 18, Figure 16 · p. 18

H SMALL BOX PUSHING

  • In Small Box Pushing, two agents push two boxes toward the goal, ideally concurrently to maximize discounted return.A box moves forward when an agent facing it acts forward; reaching the goal gives +100 and ends the episode. Agents observe only the status of the front cell, and γ=0.98. Figures 17–18 show configurations and performance comparisons across grid sizes.Source: H SMALL BOX PUSHING · p. 19, Figure 17 · p. 19, Figure 18 · p. 19

I SMALL BOX PUSHING (3 AGENTS)

J SMAC

  • The SMAC evaluation uses three default-configuration scenarios: 2 Stalkers vs 1 Spine Crawler, 3 Marines, and 2 Stalkers and 3 Zealots.The authors selected simpler tasks because harder tasks pose a huge challenge for both methods in their vanilla formulations; Figure 21 compares performance on easy SMAC maps.Source: J SMAC · p. 21, Figure 21 · p. 21
Loading 2102.04402v2…