PaperlayerView exact revision on arXiv

Source-linked AI summary

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, Chongjie Zhang

arXiv:2106.02195v2cs.LG

TL;DR

Shared parameters can make cooperative agents behave too similarly, while removing sharing wastes common knowledge. The paper combines identity–trajectory mutual-information regularization with partially shared representations and L1 regularization, achieving state-of-the-art results on GRF and SMAC super hard tasks.

  • Problem

    Parameter sharing supports cooperation but can make agents behave similarly and limit coordination, creating a need to balance diversity with shared learning.

  • Method

    The method maximizes mutual information between agent identities and trajectories, adds an intrinsic diversity reward, and combines shared and non-shared Q-functions with L1 regularization.

  • Results

    The approach achieves state-of-the-art performance on challenging GRF tasks and SMAC super hard maps while producing diverse, coordinated behaviors.

  • Takeaways & Limitations

    A partially shared network with controlled diversity supports behaviors that remain individualized when needed for complex cooperation.

Abstract

from arXiv · show

Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks.

1 Introduction

Cooperative MARL must scale to exponentially growing joint action-observation spaces, yet shared parameters can produce homogeneous behaviors that hinder exploration and coordination. The paper proposes balancing diversity with sharing through information-theoretic optimization and partially shared representations, reporting strong performance on GRF and SMAC.

  • The joint action-observation space grows exponentially with the number of agents, creating a central scalability challenge for cooperative MARL.
  • Parameter sharing improves learning efficiency by reducing policy parameters and allowing agents to reuse one another’s experiences and gradients.
  • Shared parameters can induce homogeneous behaviors under similar observations, limiting exploration and the emergence of sophisticated cooperative policies.
  • Removing parameter sharing is also unfavorable because agents may waste effort searching a larger parameter space when differentiated behavior is unnecessary.
  • The method maximizes mutual information between agent identities and trajectories, combines intrinsic and environmental rewards, and decomposes local Q-functions into shared and non-shared components.
  • The approach achieves state-of-the-art performance on SMAC super hard maps and challenging GRF multi-agent tasks.

2 Background

Cooperative multi-agent tasks are modeled as Dec-POMDPs, where agents act from individual action-observation histories under partial observability. CTDE addresses scalability by centralizing training with global information while decentralizing execution, with value factorization supporting coordination.

  • A fully cooperative task is represented as a Dec-POMDP with agents, states, actions, transitions, observations, rewards, histories, and a discount factor.
  • Under partial observability, each agent conditions its policy on its own action-observation history.
  • CTDE decentralizes control policies while using centralized training with global information to learn cooperation despite the exponentially growing joint action space.
  • Value function factorization implements CTDE by relating individual utility functions to the global action value through the IGM principle.

3 Method

The framework combines identity-aware trajectory diversity with shared and agent-specific action-value components, allowing agents to balance individualized behavior and experience sharing. Intrinsic diversity rewards are integrated with environmental rewards in centralized training.

  • The framework balances agent individuality with group coordination and can combine with existing CTDE value-factorization methods.
  • Identity-Aware Diversity: Mutual information between each agent’s trajectory and identity encourages distinguishable actions and observations through intrinsic rewards.The objective uses trajectory and identity variables to promote behavioral diversity.
  • Identity-Aware Diversity: Action-aware diversity is optimized using a Boltzmann softmax distribution over local Q values rather than the ϵ-greedy action distribution.
  • Identity-Aware Diversity: Observation-aware diversity is optimized through a variational posterior estimator whose KL-based lower bound encourages identity-distinguishable observations.
  • Action-Value Learning for Balancing Diversity and Sharing: Each individual local Q-function combines a shared Q-function with an agent-specific Q-function, while L1 regularization promotes knowledge sharing and limits excessive diversity.The paper reports that L1 regularization is critical for balancing diversity and cooperation.
  • Overall Learning Objective: Intrinsic rewards are added to environmental rewards in a TD loss, with QPLEX mixing local Q-functions and β weighting intrinsic rewards.

4 Case study: outperforming by being diverse only when necessary

The Pac-Men case study shows agents using diversity when navigating toward different rooms and shared knowledge when searching within them. This balance supports sophisticated strategies and outperforms the cited baselines.

  • The Pac-Men environment uses four partially observed agents, randomly initialized edge-room dots, unequal path lengths, and a 100-step limit.The paths have relative lengths down:left:up:right = 4:8:12:8, with three paths outside the agents’ observation scope.
  • Agents scatter into different rooms to eat dots, illustrating identity-aware diversity in their learned strategies.
  • A higher independent-to-shared Q-value standard-deviation ratio occurs in the central room and paths, while shared Q-functions dominate in edge rooms.The ratio is used to indicate whether independent or shared Q-functions play the leading role.
  • Independent Q-networks help agents reach different rooms, whereas the shared Q-network supports dot searching within rooms.The authors interpret this division as a balance between diversity and knowledge sharing.
  • The approach outperforms the baselines, while MAVEN and EOI learn optimal strategies more slowly.

5 Related Work

Prior MARL work develops value decomposition, policy-gradient, communication, knowledge-sharing, and diversity methods. This paper positions its approach as combining identity-conditioned trajectory diversity with a network structure that preserves knowledge sharing and independent action values.

  • Value-based MARL decomposes joint value functions into individual utilities to support efficient optimization and decentralized execution.
  • Knowledge-sharing methods improve coordination through shared experience, representation alignment, policy prediction, or grouped policies, but specifying what agents should share remains challenging.
  • Existing MARL diversity methods use latent exploration, pairwise mutual influence, or intrinsic value functions while accounting for coordination.
  • This paper optimizes mutual information between agent identities and trajectories and combines shared knowledge with independent action-value functions.

6 Experiments

Experiments evaluate the approach on GRF and SMAC, comparing benchmark performance with ablations and visualizations of learned cooperation. Results show strong performance on challenging settings, while ablations indicate that both controlled diversity and the partially shared structure matter.

  • Experimental setup: The experiments compare the approach with QMIX, QPLEX, MAVEN, and EOI on GRF and SMAC tasks.GRF includes three offensive scenarios; SMAC includes four super hard and two hard maps.
  • 6.1 Performance on Google Research Football (GRF): The approach outperforms all baselines on the evaluated GRF scenarios through diverse, coordinated roles such as passing, scoring, and threatening runs.The reported strategy combines partially shared representations with individualized behaviors.
  • 6.2 Performance on StarCraft II: On four super hard SMAC maps, the approach outperforms all baselines with acceptable variance across random seeds.On two hard maps, it performs similarly to baselines, suggesting the balancing process may not improve settings requiring pure homogeneity.
  • 6.3 Ablations and Visualization: Removing intrinsic-reward components decreases performance, while removing L1 regularization produces performance similar to MAVEN on academy_counterattack_hard.The ablations test identity-aware, action-aware, and observation-aware diversity together with the partially shared architecture.
  • 6.3 Ablations and Visualization: Removing the specialized network structure performs worse than QPLEX on academy_counterattack_hard, indicating identity-aware diversity is difficult to emerge without it.
  • 6.3 Ablations and Visualization: Visualized GRF strategies use off-the-ball movement and distinct positioning to create scoring opportunities, while corridor strategies use sacrificial distraction to enable teammates’ attacks.The examples illustrate diversity being used selectively alongside coordinated group behavior.

7 Closing Remarks

The paper concludes that shared MARL should introduce behavioral diversity when needed rather than abandon parameter sharing. Its closing discussion presents this balance as supporting coordination on challenging benchmark tasks.

  • The CDS approach introduces diversity when necessary while retaining the benefits of parameter sharing.
  • The paper frames the balance between individual diversity and group coordination as advancing performance on challenging deep MARL benchmarks.
  • The intrinsic reward contains action-aware and observation-aware diversity components whose formulation is revisited for improved estimation.

A.1 Intrinsic Rewards for Action-Aware Diversity

The paper develops estimators for action- and observation-aware diversity rewards, comparing approximation strategies for identity distributions and forward or backward prediction models. Experiments favor the uniform identity approximation and forward prediction on the tested SMAC map.

  • Action-aware diversity: Approximating p(id|τt) with p(id) avoids estimating the identity posterior directly, but may be invalid when identity frequencies depend on trajectories.The paper notes that any approximation produces an upper bound, whereas optimization requires a lower bound.
  • Action-aware diversity: Monte Carlo estimates identity frequencies from visitation counts, but becomes impractical in GRF and SMAC because of long horizons and continuous spaces.Variational inference instead learns qξ(id|τt) by optimizing the evidence lower bound.
  • Action-aware diversity: On SMAC 6h_vs_8z, the uniform identity approximation outperforms variational inference in average performance and variance across random seeds.The authors hypothesize that estimation error makes variational inference unstable and therefore use Eq. 13.
  • Observation-aware diversity: Observation-aware diversity can be estimated with forward prediction of the next observation or backward prediction of identity in a smaller discrete space.The backward approach uses qη2(id|τt,at), while the forward approach estimates qφ2(o_{t+1}|τt,at).
  • Observation-aware diversity: On SMAC 6h_vs_8z, the forward prediction model noticeably outperforms the backward prediction model.The forward model may be harder to estimate because it operates over the continuous observation space.

B Experiment Details

The experiments compare CDS with several established multi-agent value-based, exploration, and individuality-emergence methods. Figure 9 examines forward and backward estimation for observation-aware diversity on a SMAC super hard map.

  • Figure 9 compares forward and backward estimation for observation-aware diversity on SMAC map 6h_vs_8z.
  • CDS is compared with QMIX, QPLEX, MAVEN, and EOI using author-provided implementations with fine-tuned hyperparameters.

B.2 Architecture and Hyperparameters

The implementation uses a QPLEX-style mixing network and shared optimization settings across CDS, baselines, and ablations. Hyperparameters are tuned for differences among SMAC environments and documented in a dedicated table.

  • Architecture: The QPLEX-style mixing network uses four 32-bit attentional heads with 64-unit middle layers in GRF and SMAC.
  • Hyperparameters: RMSprop uses a 5 × 10−4 learning rate, α=0.99, no momentum or weight decay, and linearly annealed ϵ-greedy exploration.ϵ decreases from 1.0 to 0.05 over 500K time steps and then remains constant.
  • Hyperparameters: SMAC hyperparameters are tuned separately for 6h_vs_8z and Corridor, while β is fine-tuned for other maps and GRF uses prioritized replay.
  • Hyperparameters: Table 1 lists the hyperparameters used by CDS.

B.3 GRF Scenarios

The GRF evaluation covers two official scenarios and a full-field scenario with more complex problem space. Figure 10 visualizes agent, opponent, and ball positions, and training finishes in under two days.

  • GRF Scenarios: Figure 10 shows initial positions in three GRF environments, with blue agents, red opponents, and a yellow ball.
  • GRF Scenarios: CDS achieves state-of-the-art performance on the tested GRF tasks, including academy_3_vs_1_with_keeper and academy_counterattack_hard.The evaluation also includes the full-field 3_vs_1_with_keeper scenario.
  • Training Setup: Training the approach on all environments finishes in less than two days on an NVIDIA GTX 2080 Ti GPU.
Loading 2106.02195v2…