Source-linked AI summary
Decoupling Communication from Policy: Robust MARL under Bandwidth Constraints
Alexi Canesse, Benoît Goupil, Jesse Read, Sonia Vanier
TL;DR
Bandwidth constraints make MARL communication difficult because shared latent representations couple message size to policy capacity. The paper introduces β and SLIM, which standardise bandwidth evaluation and decouple communication from policy execution. Across partially observable benchmarks, SLIM remains competitive at high bandwidth and robust as communication is constrained.
Problem
Bandwidth-constrained MARL needs smaller messages, but shared latent representations make reducing message dimensions also restrict policy capacity.
Method
The paper defines β to unify message dimension, transmission rate, and graph sparsity, and introduces SLIM with a dedicated communication pathway separate from the policy representation.
Results
SLIM is competitive at high bandwidth and more robust as bandwidth decreases across four partially observable MARL benchmarks.
Takeaways & Limitations
Separating communication compression from policy representation supports communication-efficient MARL while preserving policy capacity under constrained bandwidth.
Takeaways & Limitations
β abstracts away packet headers, quantisation, latency, routing overhead, packet loss, and medium contention, so the study does not fully model real wireless deployment.
Abstract
from arXiv · showhide
Communication enables coordination in multi-agent reinforcement learning (MARL), but many real-world applications, e.g., search-and-rescue with drone swarms, operate under severe bandwidth constraints. Many communication architectures still expose a coupled bottleneck in which a shared latent representation is used for both policy execution and inter-agent communication. Consequently, reducing message size directly limits the policy's latent space, often leading to significant performance degradation. We address this with two contributions. First, we introduce $β$, a normalised per-agent bandwidth budget that unifies sparsity, rounds, and message dimension into a single comparable constraint. Second, we provide SLIM, a minimal architecture that decouples the communication pathway from the policy's latent representation, allowing us to isolate the effect of bandwidth from the effect of policy capacity while benefiting from in-step communication. We evaluate our method on several partially-observable MARL benchmarks, where communication is essential. Our approach achieves state-of-the-art performance and exhibits scalability and robustness under limited communication, with only marginal degradation as bandwidth is reduced.
1 Introduction
The paper targets communication-efficient MARL under bandwidth constraints, where shared policy and communication representations make message compression costly. It introduces β and SLIM to separate bandwidth effects from policy capacity while retaining in-step communication.
- Motivation: Bandwidth constraints are essential deployment concerns for MARL systems coordinating through communication.Applications include search-and-rescue drone swarms, autonomous vehicle fleets, and underwater exploration robots.
- Problem: Shared latent representations couple message size to policy capacity, so shrinking messages can degrade both transmitted information and action selection.This coupling creates a double penalty when communication dimensions are reduced.
- Method: SLIM decouples communication from policy execution through a dedicated communication module, allowing smaller messages without restricting the policy’s higher-dimensional latent representation.Encoded observations bypass communication to reach the policy, while messages are separately compressed.
- Evaluation Protocol: β unifies message dimension, transmission rate, and graph sparsity into one normalised bandwidth constraint for comparable evaluation.The protocol supports systematic benchmarking under identical physical bandwidth limitations.
- Results: SLIM is reported as competitive at high bandwidth and more robust as bandwidth decreases across four partially observable MARL benchmarks.Its cache is useful in environments that are not jointly fully observable.
2 Related Work
Prior MARL communication work spans continuous, sparse, scheduled, event-triggered, and information-theoretic approaches. SLIM is distinguished by separating communication from policy representation while focusing on message dimensionality and in-step exchange.
- Communication in MARL: MARL communication methods address partial observability through discrete, interpretable, continuous, and differentiable communication mechanisms.The related work contrasts these approaches with SLIM’s continuous and differentiable communication.
- Continuous Communication: CommNet and IC3Net communicate through hidden states, with IC3Net adding gating to learn when agents communicate.In fully cooperative settings, the gating mechanism may remain effectively open, yielding dense communication.
- Dedicated Communication Pathways: TarMAC separates communication from the policy latent space but communicates between timesteps, introducing delay and making multi-round mitigation bandwidth-intensive.The communication cost increases linearly with the number of rounds.
- Communication Sparsity: Sparse, dynamic, scheduled, and event-triggered methods reduce communication through graph structure, agent selection, or transmission frequency.These methods primarily address channel contention rather than individual message dimensionality.
- Information Theory: Information-theoretic methods seek compact informative messages at the encoding level, whereas SLIM addresses representation and policy architecture.The paper identifies potential complementarity between these approaches.
3 Subdivided Lightweight Inter-agent Messaging (SLIM)
SLIM separates message generation from policy execution while using current and historical messages to support decentralized decisions under partial observability. Its training uses centralized value information, while execution remains local to each agent’s observation and received messages.
- Architecture: SLIM uses an observation encoder, communication module, and policy network to implement separate message-generation and action-selection stages.Encoded observations are projected into compact transmitted messages, while the policy retains its local representation.
- Message History Cache: Each agent maintains a cache of exchanged messages so historical information can supplement observations in non-jointly observable environments.The cache can be disabled when the current observation is sufficient for optimal decision-making.
- Message History Cache: Cache memory scales linearly with the communication dimension d because it stores transmitted messages rather than full observation embeddings.This links the history-cache footprint directly to the communication representation size.
- Temporal Attention Aggregation: A transformer attention block aggregates current and cached messages into a context vector using temporal and sender embeddings.The resulting summary weights who sent what and when rather than privileging recent messages through a recurrent hidden state.
- Action Selection: The policy selects actions by conditioning on each agent’s local representation and the context vector derived from received communication.During execution, agents access only local observations and messages from other agents.
- Centralised Training: SLIM is trained with MAPPO using centralized value information from concatenated agent inputs, while the overall loss combines policy and value losses across agents.The centralized value function outputs a value estimate for each agent from the concatenated encoded observations.
4 Experiments
The experiments benchmark SLIM and established communication strategies under a unified bandwidth constraint across partially observable MARL tasks. SLIM matches strong baselines at high bandwidth and remains more robust as bandwidth decreases, while its cache improves learning where temporal context matters.
- Experimental setup: β is the per-agent transmission capacity normalized by population, combining message dimension, communication frequency, and graph sparsity.The constraint is expressed as σ × k × d ≤ β.
- Experimental setup: The experiments evaluate SLIM against established communication baselines across four partially observable MARL benchmarks.The protocol uses a shared normalised bandwidth constraint and four random seeds per model and bandwidth value.
- High-bandwidth performance: At β = 26, SLIM consistently achieves state-of-the-art performance across all benchmarks, outperforming or matching established baselines.This high-bandwidth regime tests each architecture’s performance ceiling.
- Bandwidth robustness: As β decreases from 20 to 26 on a logarithmic scale, SLIM remains robust while CommFormer, IC3Net, CommNet, and usually TarMAC degrade more sharply.SLIM maintains high rewards under strict constraints, with only a minor performance drop in Traffic Junction under severe limitations.
- Bandwidth robustness: In Navigation, SLIM achieves the best high-bandwidth performance and is more resilient to bandwidth reduction.Figure 5 reports standard-error regions for the bandwidth sweep.
- Cache ablation: The temporal cache improves training results and stability in Predator-Prey and SHAPES, where tracking trajectories over time is advantageous.The cache-enabled model uses fewer steps in Predator-Prey and achieves higher rewards in SHAPES, except at very low Predator-Prey bandwidths.
5 Limitations
The study evaluates algorithmic communication efficiency rather than complete wireless deployment. Its bandwidth proxy omits several network effects, and cache memory can grow with episode length.
- Scope boundary: The normalised bandwidth budget β abstracts away packet headers, quantisation, latency, routing overhead, packet loss, and medium contention.Extending β and evaluating SLIM under more realistic network conditions remain future work.
- Scope boundary: The message history cache has negligible overhead in the experiments, but its memory cost grows linearly with episode length.Windowing or compressed memory representations are suggested for very long-horizon tasks.
6 Conclusion
The paper proposes SLIM to separate inter-agent message encoding from policy execution and introduces β to benchmark communication under shared bandwidth limits. Across partially observable benchmarks, SLIM matches high-bandwidth state-of-the-art performance, remains robust under tighter constraints, and benefits from a history cache in non-jointly observable environments.
- Conclusion: SLIM decouples inter-agent messaging from policy execution through separate architectural pathways.The design targets modular communication in multi-agent reinforcement learning.
- Conclusion: β unifies message size, transmission frequency, and graph sparsity into one bandwidth constraint for systematic benchmarking.The metric enables comparisons under identical physical limitations.
- Conclusion: Across partially observable benchmarks, SLIM matches state-of-the-art methods at high bandwidth and remains effective as bandwidth limits tighten.Baselines degrade rapidly under severe constraints, whereas SLIM maintains effective coordination with lower bandwidth limits.
- Conclusion: The message history cache improves learning in non-jointly observable environments.This conclusion is supported by the reported ablation studies.
A Experiment Details
Table 2 lists SLIM’s hyperparameter configuration across benchmarks. The configuration was optimized with a fixed message dimensionality to support robustness across bandwidth settings without constraint-specific overfitting.
- Hyperparameter configuration: Table 2 reports SLIM hyperparameters across all benchmarks, optimized with fixed message dimensionality d = 64 (2^6).The fixed dimensionality is intended to preserve robustness across varying bandwidths without overfitting to specific constraints.
A.1 Detailed Communication Parameters
Table 3 defines normalized agent bandwidth configurations by combining graph density, communication rounds, and message dimension under β.
- For each β, Table 3 reports graph density σ, communication rounds k, and the largest feasible message dimension d.Configurations satisfy σ × k × d ≤β.
- TarMAC is evaluated in the dense one-pass setting used for the bandwidth sweeps.
- Dense CommFormer cannot satisfy β = 20 with integer d ≥1, and en-dashes mark inaccessible configurations.
B Additional Results
Additional navigation results show SLIM scaling to larger agent populations, with a substantial reward advantage over TarMAC at 20 agents.
- SLIM illustrations scale navigation to 4, 20, 24, 32, and 42 agents while agents reach individual goals and avoid collisions without seeing one another.
- 0.95 ± 0.003 reward for SLIM exceeds TarMAC’s 0.02 ± 0.01 reward in the 20-agent setting.This comparison uses TarMAC as the strongest baseline when using 4 agents.
- The 20-agent comparison is limited to TarMAC because of computational constraints.
Reproducibility
The paper supports reproducibility through released code and specified environment licensing, while reporting a cache ablation table for two environments.
- The code and experiment scripts are publicly released on GitHub with environment files specifying the library versions.
- Table 4 reports cache-ablation results using mean episode length for Predator-Prey and rewards for SHAPES across communication dimensions.Lower episode length is better, whereas higher rewards are better.
- The implementation is released under the MIT licence, while baseline code uses the licences specified for each baseline.
- Predator-Prey and Traffic Junction use MIT licences, VMAS uses GPL3, and the reimplemented SHAPES environment uses MIT.
Environments Details
The environments test communication under partial observability through cooperation, collision avoidance, navigation, and agents pursuing separate visual targets.
- Predator-Prey: Predator-Prey requires locally observing predators to cooperate in locating a stationary prey, with a message history cache enabled because past observations contain state information.
- Traffic Junction: Traffic Junction requires vision-less agents to cross intersections along stochastic routes, making communication essential for collision avoidance.Agents may move forward along their route or wait.
- Navigation: Navigation places randomly spawned agents in a bounded continuous world where they pursue individual goals without seeing one another or colliding.Agents observe position, speed, and relative goal distance; actions control acceleration.
- SHAPES: SHAPES assigns agents possibly different random-colour targets in images while restricting them to local observations.Past observations can inform another agent’s goal because agents have different objectives.