A Survey of Multi-Agent Deep Reinforcement Learning with Communication

1 Introduction

• Communication can address partial observability and non-stationarity in MARL by sharing observations, intentions, experiences, or derived features to support better-informed decisions.Agents may have only local observations, while other agents’ changing policies make their environment dynamic.Source: §1 · p. 1
• Prior surveys treated communication as predefined or offered limited coverage; Gronauer and Diepold classified recipient scope but omitted message types and training paradigms.The paper identifies the need for a systematic, fine-grained classification of recent Comm-MADRL work.Source: §1 · p. 2
• The survey includes task-oriented emergent-language studies that use learnable symbolic language to improve task performance, while distinguishing them from work primarily aimed at learning a symbolic language.Its focus is achieving domain-specific tasks rather than learning a symbolic language.Source: §1 · p. 2
• The survey proposes nine dimensions—Controlled Goals, Communication Constraints, Communicatee Type, Communication Policy, Communicated Messages, Message Combination, Inner Integration, Learning Methods, and Training Schemes—to classify Comm-MADRL systems.The framework is intended to reveal insights into current approaches and guide future system design.Source: §1 · p. 2

2.1 Multi-agent Reinforcement Learning

• A POSG models agents, states, actions, observations, transitions, and rewards; shared rewards yield a Dec-POMDP, full state observability yields a multi-agent MDP, and one agent yields single-agent forms.Under partial observability, methods often use each agent’s observation-action history to approximate the state.Source: §2.1 · p. 3
• Decentralized learning faces non-stationarity, while centralized learning mitigates it but scales poorly; MADDPG pairs a centralized critic with local actors for decentralized execution.MADDPG assumes deterministic actors with continuous actions.Source: §2.1 · pp. 3–4

2.2 Extensions with Communication

• The survey focuses on task-oriented Comm-MADRL, including its intersection with emergent language, and on explicit learnable messages that influence policies alongside observations.It excludes communication through domain-level actions and focuses on explicit message transfer rather than training-only gradient signals.Source: §2.2 · pp. 4–5

2.3 Communication in Recent Surveys

• Existing Comm-MADRL surveys cover limited communication aspects and lack fine-grained classification, motivating analysis across broader dimensions to distinguish approaches, including those in similar domains.The survey distinguishes task-oriented communication from emergent-language research whose primary goal is learning a human-like language.Source: §2.3 · p. 6

• Communication content and occurrence are learned dynamically alongside agents’ domain-specific action policies, making Comm-MADRL a joint learning challenge that optimizes environmental rewards and communication protocols.Agents use reinforcement learning to maximize environmental rewards while applying machine-learning techniques to develop effective communication protocols.Source: §3 · p. 7
• The survey reviews 41 Comm-MADRL models and organizes nine design dimensions around problem settings, communication processes, and training processes.These dimensions distinguish communication-specific and other settings, message decisions and procedures, and learning for both agents and communication.Source: §3 · p. 7
• The guideline moves from agent objectives and communication constraints to recipients, communication decisions, message generation and integration, environment actions, and training updates.The nine dimensions cover rewards and behavioral objectives, who communicates with whom and when, what is shared, how messages are combined and integrated, and updates to policies, value functions, and communication processes.Source: §3 · pp. 7–8
• For emergent language, messages are observed only at the next time step, so agents integrate prior messages before initiating new communication and then act using that information.Training uses experience from environmental interactions and inter-agent communication to update agents’ behavior and communication.Source: §3 · p. 8
• The following survey classifies literature by dimension and frames recent works in a comprehensive table using the nine dimensions.Source: §3 · p. 8

3.1 Controlled Goal

• Desired behaviors of communicating, learnable agents are classified as cooperative, competitive, or mixed according to their reward configurations.Fixed opponents and teammates are excluded from this classification.Source: §3.1 · p. 8

Cooperative

• Cooperative settings use shared global rewards or local rewards to encourage team performance through collective outcomes, collision avoidance, or reward sharing.Communication is used to promote cooperation, across environments ranging from small games to complex video games.Source: §3.1 · p. 8
• Cooperative communication has been studied across games, including Traffic Junction, StarCraft, SMAC, predator-prey, and emergent-language tasks.MAGIC outperformed CommNet [48], IC3Net [49], and GA-Comm on Traffic Junction with local rewards; FCMNet and MAIC surpassed multiple communication and value-decomposition methods on different SMAC maps. MAGIC is the only method reported on Google research football, while ATOC [39] was tested with many learning agents in predator-prey.Source: §3.1 · p. 9

Competitive

• Competitive agents may communicate for low-level cooperation toward long-term goals, and IC3Net [49] communicates when profitable, such as before catching prey.R-MACRL is classified as competitive because agents defend against malicious messages, despite the overall environment being cooperative.Source: §3.1 · p. 9

Mixed

• Mixed reward designs combine cooperative and competitive behaviors, yielding more complex communication patterns among self-interested agents.DGN agents learn to share resources rather than attack; IC3Net [49], TarMAC, and MAGIC communicate only when necessary in mixed predator-prey. NDQ agents learn not to communicate in independent search scenarios.Source: §3.1 · pp. 9–10
• In mixed tasks with asymmetric abilities, communication can enable agents to collaborate toward shared goals.IC [62] pairs senders with more vision but no ability to clear obstacles with receivers who have limited vision but can clear them; communication helps them earn higher rewards.Source: §3.1 · p. 10

3.2 Communication Constraints

• Communication Constraints distinguishes systems with explicit cost or noise limits from those without them, addressing practical barriers to real-world use.Constrained systems must use communication resources efficiently to promote learning.Source: §3.2 · p. 10
• Unconstrained communication imposes no explicit limit on channels, message content or transmission, or whether agents communicate; selective communication still qualifies if it is not cost-limited.Works that learn or predefine when to communicate are included when they differentiate useful information without explicitly assuming communication cost [39] [41] [49] [74].Source: §3.2 · p. 10
• Limited-bandwidth methods reduce message size or volume through succinct messages, pruning, shared memory, sender selection, learned gates, penalties, and constrained coding.RIAL and DIAL [73] send a binary value or real number per time step; MD-MADDPG [42] uses shared fixed-size memory. NDQ cuts 80% of messages, while MAIC prunes messages by importance. IMAC [66] constrains mutual information to encourage low-entropy messages; ETCNet and Variable-length Coding use penalties, and limited-token emergent languages [61] [62] [63] [65] are also classified here.Source: §3.2 · pp. 10–11
• Corrupted-message research addresses environmental noise and malicious messages using robust representations, noisy-channel learning, or defenses against adversarial policies.DIAL [73] finds that training with Gaussian channel noise can produce two message modes; Diff Discrete models an unknown noisy discrete channel and enables gradient backpropagation under centralized training. DCSS [64] proposes semantic token representations that are robust to noise, while R-MACRL learns defenses against malicious messaging policies.Source: §3.2 · p. 11

3.3 Communicatee Type

• Nearby-agent communication restricts recipients to observable, distance-defined, or graph neighbors; GA-Comm and MAGIC use a learned graph through a proxy instead.Neighbor relations may be predefined or learned; the survey classifies GA-Comm and MAGIC as proxy-mediated because their central unit coordinates messages rather than agents communicating directly.Source: §3.3 · p. 11
• When nearby agents are not identified, recipients may be other learning agents or explicitly assigned receivers, with agents communicating directly.IC3Net [49] allows communication with opponents, who learn not to communicate to avoid exploitation; receivers may be a separate set [62, 63, 64] or all other agents [61] [65].Source: §3.3 · p. 11
• A proxy is a virtual agent that mediates communication without affecting the environment, collecting agent information and returning common or private messages.MS-MARL-GCM sends a common message to each agent, while HAMMER sends private messages.Source: §3.3 · p. 11
• Some proxy-based systems encode and route messages without deciding whether to communicate, while others let agents decide whether to contact a coordinator.MD-MADDPG [42] uses shared memory and IMAC [66] uses a scheduler; ATOC [39] and Gated-ACML make local decisions, while GA-Comm and MAGIC use a global graph and GNN to aggregate and return messages.Source: §3.3 · p. 12
• With nearby-only communication, agents send to visible neighbors; with a proxy, all agents can send messages and receive coordinated messages.The illustrated system has five agents and one proxy; Agent 1 communicates with Agent 3 and Agent 2 with Agent 4 when communication is limited to nearby agents.Source: §3.3 · p. 12

Full Communication

• Full Communication connects every pair of agents for broadcast messaging, as in DIAL [73], RIAL [73], CommNet [48], and BiCNet.Some methods drop received messages without learning whether to send them: DCC-MD uses a fixed dropout probability, while R-MACRL learns to drop adversary messages.Source: §3.4 · p. 13

(Predefined) Partial Structure

• (Predefined) Partial Structure uses a predetermined partial graph to limit each agent’s connections, through random networks, proximity restrictions, learned relationships, thresholds, pruning, or sender-receiver assignments.Examples include NeurComm and IP for random networks, DGN for proximity, and VBC [31], NDQ, TMC [32], and MAIC for thresholds or pruning.Source: §3.4 · p. 14

Individual Control

• Individual Control lets each agent decide whether to communicate, commonly through learned gates, but also through unilateral impact evaluation or group leader election.IC3Net [49] uses deterministic gating, ATOC [39] probabilistic gating, and ETCNet limits overall message-sending probability; Gated-ACML gates communication with a proxy.Source: §3.4 · p. 14

Global Control

• Global Control learns a shared policy over communication links: SchedNet [43] schedules broadcasters, while FlowComm, GA-Comm, and MAGIC learn directed or undirected graphs.These systems add a message coordinator to coordinate and transform agents’ messages.Source: §3.4 · p. 14

Existing Knowledge

• Agents commonly encode accumulated observations, action-observation histories, or action-observation-message histories into low-dimensional messages; RNNs selectively retain or forget this history [48] [49] [42] [73] [62] [63].Low-dimensional encodings reduce communication overhead.Source: §3.5 · p. 14
• With a proxy, agents send local observations encoded [42] [66] or directly, and it returns unified or individualized global messages [42] [66].These messages relieve agents of combining multiple received messages.Source: §3.5 · p. 14
• Without a proxy, agents send messages directly: MADDPG-M shares raw local observations [41], while DIAL and RIAL encode past observations, actions, and current observations [73].Other systems encode local observations using feed-forward networks, MLPs, autoencoders, CNNs, RNNs, or GNNs [43] [40] [31] [32] [48] [49] [74]; BiCNet encodes local observations and a global view.Source: §3.5 · p. 14
• In NDQ, messages are drawn from a multivariate Gaussian distribution to maximize expressiveness by maximizing mutual information between messages and receivers’ action selection; emergent-language agents often communicate goal-related information, such as the goal’s location [61] [62] [63] [65] [64].Source: §3.5 · p. 15

Imagined Future Knowledge

• Imagined future knowledge includes intended actions [39], policy fingerprints—action probabilities in a state —and plans that model possible future observation-action sequences for receivers.Intentions are often combined with local observations; plans use an approximate environment model and other agents’ behavior models so receivers can consider senders’ potential future outcomes.Source: §3.5 · pp. 14–15

3.6 Message Combination

• With a proxy, agents receive coordinated, combined messages; without one, each agent combines messages independently according to their relative value.Message combination reduces the input to an action policy; the survey groups approaches by how they prioritize received messages.Source: §3.6 · p. 15
• Equal-weight schemes concatenate messages [73] [61] [62] [41] [43], average [48] [49] [31], or sum.Concatenation avoids information loss but can substantially expand policy inputs; messages may be single values or short vectors, and some two-agent approaches assign no explicit preferences [73] [62] [61] [41] [43] [63] [65] [64].Source: §3.6 · p. 15
• Unequal weighting ranges from rule-based pruning in DCC-MD and TMC [32] to learned gates in R-MACRL, attention [40] [74], or neural aggregation.Aggregators include feedforward networks [66], CNNs, LSTMs or RNNs [39] [42], and GNNs, which use learned agent graphs to weight neighbors.Source: §3.6 · p. 16

3.7 Inner Integration

• Inner integration classifies approaches by the learning model that incorporates combined messages, which most often serve as additional observations for a policy, value function, or both.The survey summarizes these categories in Table 8.Source: §3.7 · p. 16
• Policy-level integration learns policies with REINFORCE [48] [61] [49] or actor-critic methods [39] [40] [42] [43] [66].REINFORCE approaches collect episode rewards and train policies at episode end; actor-critic approaches use a critic Q-function to guide the actor policy.Source: §3.7 · p. 16
• Value-level approaches use messages as value-function inputs and select actions with the highest Q-value, most often using DQN-like methods [73] [31] [32].In cooperative global-reward settings, VBC [31], NDQ, TMC [32], and MAIC use value decomposition to learn a decomposed joint Q-function.Source: §3.7 · p. 16
• Both-level integration uses actor-critic methods, feeding messages to both actor and critic [74] [64].Alternatively, messages can be combined with local observations to form internal states shared with both models [62] [41] [63] [65].Source: §3.7 · p. 16

Differentiable

• Differentiable methods backpropagate gradients between agents to learn message content, and some also learn communication policies or handle discrete channels.Gumbel-softmax approximates non-differentiable communication actions but requires parameter tuning; GCL [61] adds auxiliary rewards, DCSS [64] uses labeled messages, and Diff Discrete reconstructs signals to send gradients to the sender.Source: §3.8 · p. 17

Supervised

• Supervised methods require labels for communication decisions or content, using classification tasks to guide whether and what agents communicate.ATOC [39] and Gated-ACML use Q-value differences; I2C uses causal effects, R-MACRL labels malicious messages, and DCSS [64] uses observation-to-symbol examples.Source: §3.8 · pp. 17–18

Reinforced

• Reinforced methods use rewards to train communication content, communication decisions, or the effect of messages on receivers’ action policies.RIAL [73] and HAMMER learn message content but not whether to communicate; MADDPG-M [41] uses intrinsic rewards, while ETCNet penalizes unnecessary communication.Source: §3.8 · p. 18

Regularized

• Regularized methods reduce redundant information by shaping message encodings with information-theoretic objectives, models, or compressed observations.NDQ bounds mutual information with receiver actions, IMAC [66] bounds it with sender observations, and MAIC tailors messages to modeled teammates; Bias [63] targets long-term signaling effects, while AE-Comm [65] uses autoencoders.Source: §3.8 · p. 18

Centralized Learning

• Centralized learning gathers agents’ experiences in one unit to train a controller, but recent Comm-MADRL works usually do not assume a central controller.Fully decentralized learning instead collects each agent’s experiences separately and trains agents independently.Source: §3.9 · p. 18

Fully Decentralized Learning

• In fully decentralized learning, agents train independently; this approach often uses actor-critic methods [40] [41] [74] and is prominent in emergent-language tasks [62] [63] [65].Source: §3.9 · p. 18

Centralized Training and Decentralized Execution

• CTDE collectively optimizes local policies using joint experiences and central gradients, then discards gradients for decentralized execution.For homogeneous agents with identical sensory inputs, actuators, and model structures, parameter sharing can reduce model size and potentially improve learning efficiency while allowing distinct behaviors from different observations.Source: §3.9 · p. 18

Independent Policies

• With independent policies, a central unit gathers agents’ experiences for guidance while each policy retains its own learning parameters; training can use REINFORCE or actor-critic methods [43] [66] [64].Source: §3.9 · p. 20

Parameter Sharing

• Parameter sharing trains local policies or value functions with common parameters, using DQN-like methods [73], value decomposition such as QMIX [28] [31] [32], actor-critic [39], or REINFORCE [48] [61] [49].DQN-like methods learn a shared local Q-function collectively; value decomposition supports learning from factorized rewards in cooperative settings.Source: §3.9 · p. 20

Concurrent

• Concurrent CTDE avoids central storage by having agents back up experiences under an assumption of observing others’ actions and observations; each keeps its own policy parameters and receives global-information guidance from a local unit.Concurrent actor-critic approaches give each agent its own central critic to guide its local actor [42].Source: §3.9 · p. 20

3.10 Possible Relations of Dimensions

• Limited bandwidth implemented as fewer communication opportunities can make full communication infeasible, linking communication constraints and communication policy.The paper says these dimensions are not inherently dependent under the classification criteria; the dependency arises from the implementation.Source: §3.10 · p. 20
• Message-content categories depend on whether future knowledge is simulated and used, not on communicatee type or communication constraints.A proxy or corrupted-message constraints may change message value during implementation, while the classification criteria remain independent of those dimensions.Source: §3.10 · p. 20

4.1 Findings and Discussions

• Non-cooperative settings and communication constraints remain underexplored, motivating trustworthy messaging, asynchronous validation, and fair resource allocation that can favor lower-utility agents.The survey also notes that limited budgets require agents to allocate communication resources effectively when goals vary.Source: §4.1 · pp. 21–22
• Comm-MADRL can move beyond binary communication and vector messages through targeted proxies, finer-grained policies, richer message formats, and more context-aware integration.Model-based RL could improve predictions underlying communicated intentions, while graphs and logical expressions could convey knowledge concisely but demand more sophisticated encoding and decoding signals.Source: §4.1 · p. 22
• Self-evaluated communication and fully decentralized learning remain significant challenges, despite their potential where synchronizing agents’ knowledge is computationally demanding.Parameter sharing with centralized training and decentralized execution is common, but accessing others’ memories and parameters raises privacy concerns; decentralized learning also faces limited information and increasing non-stationarity as agent numbers grow.Source: §4.1 · pp. 22–23
• Evaluation comparisons are hampered by differing platforms, metrics, and experimental settings, while communication-efficiency metrics are underused and message interpretability and architecture selection remain challenges.Emergence Degree is mainly used in emergent-language studies, but could also help explain communication across other Comm-MADRL systems.Source: §4.1 · p. 24, Figure 5 · p. 25, §4.2.5 · pp. 26–27

4.2.1 Multimodal Communication

• Multimodal communication could extend Comm-MADRL beyond visual inputs, but agents must coordinate heterogeneous modalities, preferences, and low-dimensional representations without losing essential information.Separate modality channels or a learned joint representation are possible approaches; adapting multimodal methods to multi-agent settings remains unresolved.Source: §4.2.1 · p. 25

4.2.2 Structural Communication

• Structural communication could help large, restricted-connectivity systems by routing targeted information through bridge agents and coordinating safe, efficient paths and shared protocols.Existing approaches use bridge agents across groups, but information sharing through them is not targeted; future systems must also prioritize paths and agree on encoding and decoding.Source: §4.2.2 · pp. 25–26

4.2.3 Robust Centralized Unit

• Robust centralized proxies and critics are needed to prevent adversarial messages or polluted feedback from misleading agents across a communication system.Potential approaches include varying encoding protocols across agent groups and detecting malicious messages to reconstruct the originals.Source: §4.2.3 · p. 26

4.2.4 Learning Tasks with Emergent Language

• Emergent-language research could advance by learning complex symbolic messages and their relations, especially in non-cooperative settings, while solving difficult encoding, decoding, and message-combination problems.The survey identifies this sub-area as limited; current works learn atomic symbols or combinations, not the relations needed to express facts or conjectures.Source: §4.2.4 · p. 26

5 Conclusions

• The survey proposes nine dimensions for classifying Comm-MADRL, categorizes existing works under them, and uses literature trends to identify research directions.The dimensions also provide a basis for designing systems and comparing research from distinct perspectives.Source: §5 · p. 27
• Comm-MADRL communication could extend to non-cooperative settings, heterogeneous players, larger agent populations, and diverse data such as voice and text.The survey notes that communication currently draws on raw image inputs and handcrafted features, among other sources.Source: §5 · p. 27
• Future work could develop novel metrics to clarify communication’s contribution to learning and adapt effective solutions from the MARL community.Source: §5 · p. 27