PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust and Efficient Communication for Multi-Agent Learning

Rafael Pina, Varuna De Silva, Corentin Artaud

arXiv:2609.15361v1cs.LGcs.AIcs.MA

TL;DR

In cooperative MARL, agents need informative messages despite partial observability and practical communication constraints. MARC uses attention and an information-inspired regularizer, with DCT for compression; the paper reports consistent performance across tested scenarios and robustness under significant information loss.

  • Problem

    In partially observable cooperative MARL, messages must be representative and stable, while limited bandwidth and signal noise constrain communication.

    Source: §I · p. 1

  • Method

    MARC combines attention-based message generation with an information-inspired regularizer, and explores DCT-based message compression.

    Source: §I · p. 1, A. Multi-Agent Regularized… · p. 4

  • Results

    MARC shows consistently good performance across the experimented scenarios, while methods such as COMMNET and TARMAC are inconsistent across cases.

    Source: B. Main Results: Performance in SMAC… · p. 8

  • Takeaways & Limitations

    The authors report that DCT compression maintains robust distributed performance under significant information loss, supporting communication-based cooperation under resource constraints.

    Source: §I · p. 2, §VI · p. 10

Abstract

from arXiv · show

Effective communication is a cornerstone of distributed intelligence in Multi-Agent Reinforcement Learning (MARL), yet ensuring that generated messages are both informative and robust to physical constraints remains a significant challenge. This paper introduces Multi-Agent Regularized Communication (MARC), a novel framework inspired by information-theoretic principles of conditional mutual information. MARC employs an attention-based architecture coupled with a unique message regularization mechanism designed to minimize uncertainty regarding future system states, thereby inducing the learning of highly representative communication protocols. Crucially, we evaluate MARC under stringent communication bottlenecks and lossy channels, simulating the real-world constraints of autonomous robotic networks and decentralized systems. Our results demonstrate that MARC significantly outperforms state-of-the-art methods in complex cooperative domains. Furthermore, we provide a deep analysis of message characteristics, proving that MARC maintains high operational performance even under significant data compression, offering a scalable path for deploying intelligent agents in resource-constrained environments.

I. INTRODUCTION

Agents in partially observable cooperative tasks need representative, stable messages, but bandwidth limits and signal noise constrain communication. MARC combines attention and information-inspired regularization with DCT compression to address message quality and communication overhead.

I. INTRODUCTION

  • Partial observability partitions critical environmental information across agents, while learned messages must remain representative and stable despite limited bandwidth and signal noise.The introduction frames communication as a way for agents to share encoded perceptions and coordinate toward a collective goal.Source: §I · p. 1
  • MARC combines attention with an information-inspired regularizer that targets uncertainty about future observations and is designed for representative communication within CTDE.The paper describes MARC as modular and compatible with diverse value-function factorization methods.Source: §I · pp. 1–2
  • DCT-based lossy compression is presented as a way to reduce message size and communication overhead without compromising system-level performance.The introduction positions this approach as relevant to resource-constrained hardware.Source: §I · pp. 1–2
  • The paper also contributes empirical analysis of learned message structures under ideal and lossy conditions, alongside information-theoretic grounding for its regularizer.The analysis examines adaptive agent behaviors.Source: §I · p. 2

II. RELATED WORK

Prior work spans cooperative value factorization, communication for sharing experience, and studies of learned message content and compression. The section situates communication within Dec-POMDPs and CTDE, where agents execute decentralized policies that can use teammates’ messages.

II. RELATED WORK

  • Related work covers cooperative value factorization and communication methods that vary what, whom, and when agents message, as well as language analysis and bandwidth-aware compression.The cited communication literature includes attention, customized messages, message aggregation, and sharing intentions.Source: §II · p. 2

A. Decentralised Partially Observable Markov Decision Processes (Dec-POMDPs)

  • The Dec-POMDP formulation models local observations and action-observation histories, with agents choosing actions under policies conditioned on incoming messages and sharing team rewards.The team objective is to maximize an action-value function defined over the discounted return.Source: A. Decentralised Partially Observable… · pp. 2–3

B. Centralised Training with Decentralised Execution (CTDE) and Communication in MARL

  • CTDE trains agents centrally but requires decentralized execution; communication lets them broadcast encoded information peer-to-peer, while inadequate messages can overload learning networks and harm performance.The section contrasts peer-to-peer communication with reliance on a centralized oracle.Source: B. Centralised Training with… · p. 3

C. Value Function Factorisation in MARL

Value function factorisation decomposes a joint action-value function into agent-wise functions, while the DCT provides a way to compress messages for constrained communication. The paper uses VDN and QMIX as factorisation methods and considers lossy compression.

C. Value Function Factorisation in MARL

  • Factorisation methods decompose a joint action-value function into agent-wise functions, with local optimal actions required to maximize the joint function under IGM.The mixing function combines individual action-value functions into Qtot.Source: C. Value Function Factorisation in MARL · p. 3
  • VDN sums individual Q-functions, whereas QMIX extends this to a family of monotonic functions; both satisfy IGM.The paper uses VDN and QMIX to demonstrate its communication approach on standard methods without communication.Source: C. Value Function Factorisation in MARL · p. 3
  • The factorisation methods use a temporal-difference loss based on DQN, with a replay buffer and target network.The loss is defined with respect to Qtot.Source: C. Value Function Factorisation in MARL · p. 3

D. Discrete Cosine Transform (DCT)

  • The DCT encodes a sequence of data points into a smaller representation, and the paper uses its type-II form.The inverse DCT reconstructs the message with some information loss.Source: D. Discrete Cosine Transform (DCT) · pp. 3–4
  • For message compression, the DCT is applied to the last dimension of agent-message vectors before transmission, then inverted at the destination.The compressed representation is intended for communication channels with size constraints.Source: D. Discrete Cosine Transform (DCT) · p. 3
  • The paper considers lossy compression because lossless encoding does not reduce message size for limited communication channels.The paper hypothesizes that agents can still benefit from communication if information loss is not too heavy.Source: D. Discrete Cosine Transform (DCT) · p. 4

IV. METHODS

MARC uses attention to generate messages from local observations and adds a regularizer that trains agents to predict future observations. Its objective combines this auxiliary loss with the value-factorisation loss, and the paper states a convergence result under tabular conditions.

A. Multi-Agent Regularized Communication (MARC) in MARL

  • MARC encodes agents’ local observations into initial messages and uses attention to learn their relative importance.The attention operations use keys, queries, and values embedded at dimension dk.Source: A. Multi-Agent Regularized… · p. 4
  • The regularizer is motivated by the hypothesis that messages useful for predicting an agent’s next observations contain meaningful information about its observations.The theorem states that conditioning on previous messages as well as previous observations reduces uncertainty about observations.Source: A. Multi-Agent Regularized… · p. 4
  • A recurrent LSTM encoder receives previous messages and observations and predicts future observations; experiments predict one timestep ahead.The predictions supply an additional loss alongside the value-factorisation loss.Source: A. Multi-Agent Regularized… · pp. 4–5
  • The DCT and inverse DCT blocks appear in the architecture only for compression analysis, not in the main experiments.In the compression setup, generated messages are compressed before transmission and decompressed at the destination.Source: Figure 2 · p. 5, A. Multi-Agent Regularized… · p. 5
  • The overall objective combines the value-factorisation loss with the observation-prediction loss to train the communication and regularizer modules.The architecture uses parameter sharing and is described as compatible with existing value-function factorisation methods.Source: A. Multi-Agent Regularized… · p. 5
  • The convergence demonstration assumes tabular conditions, while the experiments do not use that assumption.The stated result bounds value-function error and observation-prediction error as iterations tend to infinity.Source: A. Multi-Agent Regularized… · pp. 5–6

B. Message Compression for Lossy Communication in MARL

The paper uses the DCT to compress generated messages for lossy communication, then applies the inverse transform at the destination. The message networks retain their original sizes because compression affects the communication channel.

B. Message Compression for Lossy Communication in MARL

  • Compression applies to the communication channel, so it does not require changing agent-network sizes; messages are reconstructed to their original size at the other end.Source: B. Message Compression for Lossy… · p. 6
  • Messages are compressed with the DCT after generation and decompressed with the inverse DCT when they reach their destination.The reconstructed messages are used alongside each receiving agent’s own observations.Source: B. Message Compression for Lossy… · p. 6
  • The lossy compression reduces message size to lower potential communication overhead.The paper frames this as a way to study communication when messages may be lost or dropped.Source: B. Message Compression for Lossy… · p. 6

V. EXPERIMENTS AND RESULTS1

The experiments evaluate MARC on top of QMIX and VDN, comparing it with methods both with and without communication. The section states that its V-B results do not use compression.

V. EXPERIMENTS AND RESULTS1

  • The experiments apply MARC with QMIX and VDN, two value-function factorisation methods.The paper notes that MARC can be built on top of any such method.Source: §V · p. 6
  • The comparison includes vanilla methods without communication and MASIA, COMMNET, and TARMAC.Source: §V · p. 6
  • The V-B results use no compression; the DCT and inverse DCT are reserved for the message-compression investigation in V-D.Source: §V · p. 6

A. Hyperparameters and Implementation Details

The experiments use shared recurrent Q-networks with GRU width 64 and set MARC message length to 10. Training uses a replay buffer, epsilon-greedy exploration, and RMSProp.

A. Hyperparameters and Implementation Details

  • MARC uses 64-dimensional attention embeddings and produces messages of length 10 at each timestep.With compression, a 40% setting yields messages of size 6.Source: A. Hyperparameters and Implementation… · p. 6
  • Agents use a shared deep recurrent Q-network with GRU width 64; QMIX mixer hidden layers have size 32.Parameter sharing speeds learning and requires agent IDs as network inputs.Source: A. Hyperparameters and Implementation… · p. 6
  • Training samples batches of 32 episodes from a replay buffer of 5000 and anneals epsilon from 1 to 0.05 over 50000 episodes.The discount factor is 0.99, target networks update every 200 episodes, and RMSProp uses learning rate 5 × 10−4.Source: A. Hyperparameters and Implementation… · p. 6

B. Main Results: Performance in SMAC and PredatorPrey

MARC improves performance most clearly in more complex cooperative scenarios, with gains across the tested SMAC tasks and consistent success across PredatorPrey penalty settings. In the 2c vs 64zg behavior analysis, MARC agents stay together, unlike agents without communication or those using COMMNET’s suboptimal policy.

B. Main Results: Performance in SMAC and PredatorPrey

  • In the simpler 3s vs 5z scenario, communication helps QMIX solve the task sooner but is less useful for VDN; all methods converge quickly.The paper suggests communication may be less important for efficient strategies in this scenario.Source: B. Main Results: Performance in SMAC… · pp. 6–7
  • In complex SMAC scenarios, MARC improves learning speed and performance in 2c vs 64zg and 1o2r vs 4r, with particularly strong VDN+MARC gains in 2c vs 64zg.QMIX+MARC is strongest in MMM2; communication benefits are less prominent there, and TARMAC, MASIA, and COMMNET are inconsistent across tasks.Source: B. Main Results: Performance in SMAC… · p. 7, Figure 3 · p. 7
  • In 2c vs 64zg, MARC agents stay close together, whereas agents without communication move apart and QMIX+COMMNET agents lose one agent under a suboptimal policy.The COMMNET agents spend long periods in the enemy team and lose health; this behavior is not observed with MARC.Source: B. Main Results: Performance in SMAC… · p. 7, Figure 5 · p. 8
  • MARC performs consistently across PredatorPrey penalties; at p = −0.75 and p = −1.0, only MARC and COMMNET achieve positive rewards.At p = −0.50, communication does not appear critical; TARMAC and MASIA achieve positive rewards only at that penalty.Source: B. Main Results: Performance in SMAC… · pp. 7–8, Figure 4 · p. 7

C. Performance in Other Complex Scenarios

MARC substantially improves performance over baselines in Lumberjacks and TrafficJunction, where communication has a strong impact. These tests extend evaluation beyond settings where communication is especially necessary.

C. Performance in Other Complex Scenarios

  • The additional environments were chosen to test MARC beyond scenarios where communication may be less important.The authors emphasize solving a wide range of environments, not only tasks that require communication.Source: C. Performance in Other Complex… · p. 8
  • Lumberjacks requires four agents to chop 12 trees on an 8 × 8 map, with tree difficulty determined by how many agents must cooperate.Each timestep incurs a −1 penalty and each cut tree yields +10; the high step penalty and cooperation demand make the task challenging.Source: C. Performance in Other Complex… · p. 8
  • TrafficJunction has 10 agents crossing a four-entry junction toward assigned destinations, with penalties for delay and collisions.The penalties are −0.01 for each step and −10 for a collision.Source: C. Performance in Other Complex… · p. 8
  • In Lumberjacks and TrafficJunction, communication has a strong impact and MARC substantially improves performance over baselines.The paper also notes MARC’s robustness across these environments, while other methods can perform poorly in some settings.Source: C. Performance in Other Complex… · p. 8

1) Compressing Messages:

The experiments apply DCT lossy compression to MARC messages while reconstructing them at the destination, leaving communication endpoint sizes unchanged. Both VDN+MARC and QMIX+MARC can still learn under compression, although performance decreases as compression increases.

1) Compressing Messages:

  • DCT compresses messages for transmission and reconstructs them at the destination, so endpoint sizes need not change across compression levels.The experiments test message sizes of 20%, 40%, 60%, and 80% of their original size.Source: 1) Compressing Messages · p. 9, Figure 7 · p. 9, Figure 8 · p. 9
  • VDN+MARC still learns with DCT-compressed messages, although performance decreases; in Fig. 7(b), it remains better than VDN without communication.The comparison is reported for the tested scenarios, including the 20%, 40%, 60%, and 80% message-size settings.Source: 1) Compressing Messages · p. 9, Figure 7 · p. 9
  • QMIX+MARC still learns under high DCT compression, while its performance decreases as compression increases.The authors identify constrained-bandwidth settings as a reason to preserve the ability to cooperate using compressed messages.Source: 1) Compressing Messages · p. 9, Figure 8 · p. 9
  • At 40% compression, QMIX+MARC messages occupy narrower value ranges, especially near the episode’s end.The authors interpret this as agents seeking more specific message values when messages will be compressed.Source: 1) Compressing Messages · p. 9

2) Messages Learned:

MARC’s message analyses report distinct observation-linked messages and regularizer-associated compactness, while the conclusion reports improved complex-task performance and robustness under compression. The authors identify further training and deployment settings as future work.

2) Messages Learned:

  • In Switch, each agent’s message has a distinct meaning tied to its observation, and during negotiation one agent waits for the other to cross.The illustrated negotiation spans t = [3, . . . , 5].Source: 2) Messages Learned · p. 9

VI. CONCLUSION AND FUTURE WORK

  • MARC significantly enhances performance in complex tasks by fostering highly representative communication protocols.Source: §VI · p. 10

E. Ablations: Message Regularizer

VI. CONCLUSION AND FUTURE WORK

  • With DCT message compression, MARC maintains robust distributed intelligence under significant information loss.A plotted successful 2c vs 64zg episode uses 40% DCT compression.Source: §VI · p. 10, Figure 10 · p. 10
  • Future work includes varying network capacity and agent heterogeneity, fully decentralized training, adaptive compression across network scales, and robotic testbeds.Source: §VI · p. 10

Related papers

Loading 2609.15361v1…