PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Multi-Agent Deep Reinforcement Learning with Communication

Changxi Zhu, Mehdi Dastani, Shihan Wang

arXiv:2203.08975v2cs.MAcs.LG

TL;DR

Comm-MADRL lacks a systematic framework for distinguishing and comparing approaches despite growing research on learned communication. This survey reviews recent work through nine communication dimensions, identifies literature trends, and proposes future research directions. It concludes that communication research should expand toward non-cooperative settings, heterogeneous players, and larger agent populations.

  • Problem

    Existing Comm-MADRL surveys lack comprehensive recent coverage and fine-grained perspectives for distinguishing communication approaches.

  • Method

    The survey classifies Comm-MADRL literature across nine dimensions covering communication decisions, message use, integration, and learning objectives.

  • Results

    The multidimensional review identifies literature trends and gaps, including limited exploration of non-cooperative settings, heterogeneous players, and larger agent populations.

  • Takeaways & Limitations

    Communication in MADRL can be more fruitful and versatile when applied beyond predominantly cooperative settings and homogeneous, smaller-agent scenarios.

  • Takeaways & Limitations

    Comm-MADRL still faces challenges in neural architecture selection, hyperparameter tuning, and explaining communicated messages.

Abstract

from arXiv · show

Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

1 Introduction

MADRL addresses multi-agent settings with partial observability and non-stationarity, while communication can support agents’ learning and coordination. The survey responds to limited systematic coverage by proposing a structured framework for analyzing Comm-MADRL approaches.

  • Multi-agent systems span autonomous driving, sensor networks, robotics, and game-playing, with agents that may cooperate, compete, or combine both objectives.
  • Partial observability and non-stationarity are central MARL challenges because agents receive local observations and other agents’ changing policies alter the environment.
  • Existing Comm-MADRL surveys provide limited coverage and lack a fine-grained classification system for comparing recent learning-communication approaches.
  • The survey focuses on learnable protocols for deciding when, with whom, what, and how agents communicate, and which learning objectives communication can support.
  • It proposes nine dimensions to systematically differentiate and categorize Comm-MADRL approaches and guide the design of new systems.

2 Background

This section introduces MARL formalizations and learning paradigms, then explains how communication extends multi-agent deep reinforcement learning. It motivates fine-grained classification because existing surveys have limited coverage and communication spans multiple design processes.

  • Multi-agent reinforcement learning: A POSG models agents, states, actions, observations, transitions, and rewards; identical rewards reduce it to a Dec-POMDP.Agents receive local observations and individual rewards, while joint actions and observations describe the multi-agent interaction.
  • Multi-agent reinforcement learning: Decentralized learning decomposes an n-agent problem into single-agent problems, with each agent learning from its local observation.Other agents are treated as part of the environment in this setting.
  • Value-based methods: Value decomposition factors a joint Q-function into local Q-functions, enabling efficient training but requiring aggregation choices such as fixed or constrained learnable weights.These methods are widely used in Comm-MADRL, and advantage functions can reduce variance.
  • Policy-based methods: Policy-gradient methods optimize parameterized policies, while stochastic gradient ascent updates parameters using sampled actions and states.REINFORCE uses episode returns, whereas actor-critic methods use an estimated value function whose critic guides the actor.
  • Actor-critic methods: MADDPG combines a centralized critic that captures global information with local actors for decentralized execution.It assumes deterministic actors with continuous actions, allowing gradients to be backpropagated through the learning setup.
  • Extensions with communication: Comm-MADRL studies learnable protocols for sharing information, including when and with whom to communicate, while targeting reduced communication overhead.Existing surveys lack coverage of recent developments and do not fully address communication as a combinatorial problem across learning and decision-making.

3 Learning Tasks with Communication in MADRL

Comm-MADRL treats communication as a jointly learned part of multi-agent task solving, with agents learning both domain policies and dynamic protocols. The survey organizes these systems through a structured guideline covering goals, communication decisions, message processing, policy integration, and training.

  • Learning tasks with communication: Comm-MADRL jointly learns domain-specific action policies and dynamic communication protocols.Both message content and communication occurrences are subject to learning.
  • Survey scope: The survey reviews 41 Comm-MADRL models and classifies them across dimensions spanning problem, communication, and training processes.The classification is intended to make similarities and differences among recent works clear.
  • Guideline: Emergent-language studies require rearranging the procedure because messages emitted with domain actions become observable only at the next time step.Dimensions 6 and 7 therefore precede dimension 4 in that setting.
  • Survey approach: The survey examines recent Comm-MADRL literature by classifying systems according to how communication is utilized to improve MADRL performance.It presents an extensive survey organized by each dimension and a comprehensive table of recent works.

3.1 Controlled Goal

Controlled goals classify Comm-MADRL settings by whether agents learn cooperatively, competitively, or with mixed incentives. The surveyed literature is dominated by cooperative benchmarks, while competitive communication is comparatively limited and mixed settings produce more complex patterns.

  • Goal categories: Controlled goals produce cooperative, competitive, or mixed agent behaviors through different reward configurations and objectives.Communication and domain-task goals are aligned because agents communicate to obtain higher rewards.
  • Cooperative: Cooperative settings use shared global rewards or potentially different local rewards to promote team performance.Traffic Junction and SMAC are prominent cooperative benchmarks in the surveyed literature.
  • Competitive: Competitive settings can discourage communication in zero-sum games, although communication may support low-level cooperation toward long-term goals.IC3Net is identified as the only work testing competitive settings in the survey passage.
  • Mixed: Mixed settings combine cooperative and competitive behaviors through individual rewards, producing more complex communication patterns.DGN rewards agents for eating food while offering higher rewards for attacking, with punishment for being attacked.
  • Mixed: Agents in independent scenarios learn not to communicate, whereas agents with complementary abilities can collaborate through communication to obtain higher rewards.In IC, senders have greater vision while receivers can clear obstacles.

3.2 Communication Constraints

Communication constraints capture practical limits that can make Comm-MADRL systems less realistic than simulations. The survey distinguishes unconstrained communication from constrained communication involving bandwidth limits or corrupted messages.

  • Motivation: Communication cost and environmental noise are practical concerns that can impair the use of Comm-MADRL systems in realistic applications.The survey uses communication constraints to categorize which concerns a system addresses.
  • Unconstrained Communication: Unconstrained communication leaves channels, message content and transmission, and communication decisions unrestricted.Agents can, in principle, communicate as much information as they can without explicit overhead-prevention restrictions.
  • Constrained Communication: Constrained communication requires agents to use communication resources efficiently because processes are limited by cost or noise.The survey identifies limited bandwidth and corrupted messages as two practical concerns.
  • Limited Bandwidth: Limited bandwidth restricts channel capacity, communication frequency, or message size, motivating succinct or short messages.Some methods add reward penalties or use token limits to encourage efficient communication.
  • Corrupted Messages: Corrupted-message settings model transmission errors caused by environmental noise or malicious intentions.Examples include Gaussian channel noise and discrete communication channels with unknown noise.

3.3 Communicatee Type

Communicatee type specifies who receives messages and whether agents communicate directly or through a proxy. The survey distinguishes nearby agents, other learning agents, and virtual proxy-mediated communication.

  • Definition: Communicatee type classifies Comm-MADRL systems according to which agents receive messages and whether communication is direct.The categories are based on communication relationships among agents in the environment.
  • Agents in the MAS: Nearby-agent communication restricts messages to observable agents, distance-based neighbors, or neighboring agents on a graph.Examples include communication with observable agents or the three closest neighbors.
  • Agents in the MAS: Other-learning-agent communication uses other agents or opponents as communicatees when nearby agents are not specifically identified.Some systems also assign disjoint sender and receiver roles.
  • Proxy: A proxy is a virtual agent that mediates communication without directly affecting the environment.Proxies can collect, transform, aggregate, store, and selectively retrieve agents’ information.
  • Example: In the illustrated five-agent system, nearby communication connects Agent 1 to Agent 3 and Agent 2 to Agent 4, while proxy communication routes messages through a shared medium.Agent 5 is outside Agents 1 and 2’s view range in the example.

3.4 Communication Policy

Communication policy determines when and with whom agents communicate, forming links for message transmission. The survey distinguishes four categories by whether communication structure is predefined or learned and controlled individually or globally.

  • Communication policy specifies when and with which agents communication occurs, defining actions that form communication links.Actions may encode whether communication with each other agent is allowed at a given time step.
  • The survey identifies four communication-policy categories: Full Communication, Predefined Partial Structure, Individual Control, and Global Control.These categories organize predefined structures and learnable policies according to the location of control.
  • Full Communication: Full Communication connects every pair of agents, producing a fully connected graph in which messages are broadcast.This category was often used in early Comm-MADRL work.
  • (Predefined) Partial Structure: Predefined Partial Structure uses a predetermined partial graph so each agent communicates with only a limited number of agents.Examples include networks that maintain a fixed average number of connections per agent during learning.
  • Individual Control: Individual Control lets each agent independently decide whether to communicate, commonly through a learnable gate mechanism.The resulting communication graph is formed implicitly by agents’ individual decisions.
  • Global Control: Global Control learns a shared communication policy centrally, providing system-wide control over communication links.A global scheduler can limit how many agents broadcast their messages.

3.5 Communicated Messages

After links are established, agents must determine what information to communicate. The survey distinguishes messages based on whether they encode existing knowledge or imagined future information.

  • Communicated messages can encode historical experiences, intended actions, or future plans to enrich coordination.These messages can expand agents’ understanding of the environment and support coordinated behavior.
  • Existing Knowledge: Existing Knowledge messages share past observations, previous movements, policies, or other environment knowledge.Recurrent networks commonly encode accumulated history into lower-dimensional messages to reduce communication overhead.
  • Existing Knowledge: Emergent-language systems may communicate goal-related information such as a goal’s location.
  • Imagined Future Knowledge: Imagined Future Knowledge includes intended actions, policy fingerprints, and future plans.Recent work often combines intended actions with local observations, while future plans may use an approximated environment model.

3.6 Message Combination

Message combination determines how agents integrate multiple received messages before internal processing. The survey contrasts equal treatment with preference-based selection and then classifies where messages enter the learning model.

  • Message Combination integrates multiple received messages before they are processed by an agent’s internal model.When a proxy coordinates messages, agents receive combined messages and need no further combination step.
  • Equally Valued: Equally Valued combination treats received messages uniformly, using equal weights, no weights, or concatenation.Concatenation preserves all messages but can substantially expand the action policy’s input space.
  • Unequally Valued: Unequally Valued combination assigns distinct preferences, using rules or learned mechanisms to prune or prioritize messages.Examples include probabilistic dropout and expiration checks within a preset time window.
  • Inner Integration: Inner Integration classifies how combined messages enter an agent’s policy, value function, or both.Most literature treats messages as additional observations supplied to the learning model.
  • Inner Integration: Policy-level integration uses messages in policy learning, while value-level integration feeds them into value functions from which actions are selected.Policy-level studies use methods such as REINFORCE or actor-critic, whereas value-level studies commonly use DQN-like methods.
  • Inner Integration: Policy- and Value-level integration supplies messages to both actor and critic models or uses them to generate shared internal states.

3.8 Learning Methods

Learning methods determine how communication protocols are trained and what assumptions they require. The survey groups approaches into differentiable, supervised, reinforced, and regularized methods.

  • Learning methods specify the machine-learning techniques used to learn communication protocols within Comm-MADRL.Communication training may be integrated with overall learning when derivatives through message generation and communication policies are available.
  • The survey identifies differentiable, supervised, reinforced, and regularized communication-learning categories.These categories differ in their training signals and assumptions, summarized in the survey’s learning-method taxonomy.
  • Differentiable: Differentiable methods learn communication by backpropagating gradients between agents.They can learn message content under predefined communication structures and address nondifferentiable communication actions through specialized techniques.
  • Supervised: Supervised methods require true labels for when and what information to communicate.Labels can be derived from Q-value differences, causal effects, message status, or message-content targets.
  • Reinforced: Reinforced methods train communication alongside action policies using reinforcement-learning signals.Applications include learning message content, auxiliary rewards for goal prediction, and compositional communication languages.
  • Regularized: Regularized methods reduce redundant communication by constraining mutual information involving messages, observations, or receivers’ action selection.NDQ uses a lower bound related to received messages and action selection, while IMAC minimizes an upper bound involving messages and senders’ observations.

3.9 Training Schemes

Training schemes determine how agents’ experiences and model parameters are organized for learning and execution. The survey distinguishes centralized, decentralized, CTDE, and concurrent arrangements, including independent policies and parameter sharing.

  • Centralized Learning: Centralized learning gathers agents’ experiences into one central unit that learns to control all agents.Recent Comm-MADRL works usually do not assume a central controller.
  • Fully Decentralized Learning: Fully decentralized learning collects experiences individually, with each agent undergoing an independent training process.This scheme often uses actor-critic methods and is prominent in emergent-language tasks.
  • Centralized Training and Decentralized Execution: CTDE jointly optimizes agents’ local policies using collective experiences, then executes with only the learned policies and without gradients.The scheme uses centralized information during optimization while facilitating decentralized execution after training.
  • Independent Policies: Independent-policy CTDE assigns each local policy its own learning parameters while a central unit provides global information and guidance.Policy-gradient and actor-critic algorithms can train the system.
  • Parameter Sharing: Parameter sharing gives all local policies or value functions one shared parameter set, reducing the number of learning parameters.DQN-like, actor-critic, and REINFORCE-based algorithms are used in this category.
  • Concurrent: Concurrent approaches use replicated experience storage when central storage is infeasible, while agents can observe other agents’ actions and observations.In concurrent CTDE, each agent maintains individual policy parameters and receives guidance from a local unit collecting global information.

3.10 Possible Relations of Dimensions

The nine Comm-MADRL dimensions are classification axes rather than inherently dependent components. Nevertheless, concrete system implementations can create dependencies between dimensions, such as bandwidth constraints affecting communication design.

  • Possible Relations of Dimensions: The survey introduces nine dimensions and categories for analyzing Comm-MADRL approaches, while noting that implementations may link dimensions in practice.The classification criteria themselves do not make the dimensions inherently dependent.
  • Possible Relations of Dimensions: Table 12 records the notation used for the categories across the proposed dimensions.The table is presented as a notation reference for the categories.

4 Findings, Discussions, and Research Directions

The survey identifies recurring design patterns, unresolved challenges, and research directions across communication, training, message representation, and system structure. It uses the proposed dimensions to organize findings and motivate future combinations, including multimodal, structural, robust, and emergent-language communication.

  • Controlled Goals: Recent work mainly studies cooperative settings, while communication in non-cooperative environments remains limited and may involve deception or manipulation.Existing partial-competition results concern a simplified game without state transitions, leaving larger-state settings open.
  • Communicatee Type: Proxy-based communication can target only subsets of agents when independence among agents permits more selective coordination.This approach is discussed under the Communicatee Type dimension.
  • Communication Policy: Communication policies are often binary, but finer-grained actions can represent uncertainty, confidence, or budget thresholds.The survey suggests tailoring communication actions to human heuristics and system requirements.
  • Communicated Messages: Complex message formats such as graphs and logical expressions can convey knowledge concisely, but require more sophisticated encoding, decoding, and learning signals.Model-based RL may improve predictions about future situations before agents communicate intentions or plans.
  • Message Combination: Attention mechanisms commonly weight received messages according to their varying importance, potentially incorporating prior knowledge about other agents’ capabilities.This finding concerns the Message Combination dimension.
  • Inner Integration: Messages are often integrated into policy models, with different neural-network layers offering different levels of abstraction for decision-making.This observation concerns the Inner Integration dimension.
  • Findings and Discussions: Parameter sharing with CTDE is widely adopted to reduce learning parameters, but access to agents’ memories and parameters can raise privacy concerns.Fully decentralized learning faces limited environmental knowledge and increasing non-stationarity as the number of agents grows.
  • Findings and Discussions: Communication efficiency is underused as an evaluation metric, motivating further study of communication-resource usage in Comm-MADRL.Emergence Degree is mainly used in emergent-language studies but may also illuminate communication in other systems.

5 Conclusions

The survey classifies Comm-MADRL literature across nine dimensions, uses those dimensions to identify trends and research directions, and highlights several ways communication could become more versatile.

  • The survey classifies existing Comm-MADRL research using 9 dimensions.
  • The dimensions support comparisons across studies, trend analysis, and identification of gaps for future research.
  • Future Comm-MADRL systems could address non-cooperative settings, heterogeneous players, and larger numbers of agents.
  • Communication could incorporate diverse data sources, including voice and text, beyond raw images and handcrafted features.
  • Novel metrics could help evaluate communication’s contribution to the overall learning process.
Loading 2203.08975v2…