PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

Teoman Kaman

arXiv:2608.14559v1cs.AIcs.LG

TL;DR

Multi-agent reinforcement learning lacks a principled way to decide when communication is valuable under partial observability. This paper gates communication using KL divergence between learned belief distributions, finding that calibrated gating improves Predator-Prey performance in larger environments while the belief head also benefits MPE coordination.

  • Problem

    Agents need to decide when communication is valuable under partial observability, but existing gates lack an explicit notion of belief disagreement.

  • Method

    The method replaces IC3Net’s learned gate with a deterministic KL-divergence threshold over agents’ learned belief distributions.

  • Results

    Across tasks, KL-belief helps in larger Predator-Prey environments at ε = 0.5 but not 10×10, while its belief head improves MPE performance even without selective gating.

  • Takeaways & Limitations

    Communication gating is useful when its KL threshold is calibrated to environment complexity, while belief representations can benefit coordination independently of gating.

  • Takeaways & Limitations

    The KL-belief gate offers no advantage over IC3Net in PP 10×10, and its threshold must be calibrated across environments.

Abstract

from arXiv · show

Effective communication in multi-agent reinforcement learning requires agents to decide not only \textit{what} to communicate, but when? Existing approaches either communicate at every timestep or learn a binary gate through REINFORCE policy gradients \cite{singh2019}, a high-variance signal that produces unstable and uninterpretable gating behavior. I propose a principled alternative: agents communicate only when the KL divergence between their learned belief distributions exceeds a fixed threshold. Each agent maintains a belief distribution over a latent world state computed as a softmax over its LSTM hidden state, and communicates only when belief disagreement is large enough to justify information exchange. I evaluate this approach on the Predator-Prey benchmark from IC3Net \cite{singh2019} across two environment sizes with 5 seeds each, and on MPE simple\_spread \cite{lowe2017}, comparing against IC3Net, CommNet, and an independent controller. On PP 10$\times$10, IC3Net outperforms KL-belief at all thresholds. On the harder PP 20$\times$20, a threshold ablation over $\varepsilon \in \{0.1, 0.3, 0.5, 1.0\}$ reveals an inverted U-shape: $\varepsilon=0.5$ achieves 73.84 average steps and 42\% success rate versus IC3Net's 75.31 steps and 31\%, a gap of 1.47 steps and 11 percentage points with tighter seed variance. On MPE, the belief head improves mean reward by 12 points and reduces variance by 26$\times$ even when gating is inactive, suggesting two orthogonal contributions: principled gating when beliefs can converge, and improved latent representations that benefit coordination regardless.

1 Introduction

The paper proposes replacing IC3Net’s high-variance REINFORCE gate with an interpretable communication trigger based on disagreement between agents’ belief distributions. Experiments on Predator-Prey and MPE compare this KL-belief approach with established communication baselines across environment sizes and thresholds.

  • Motivation: Communication is costly in partially observed multi-agent tasks, where agents must coordinate using incomplete local views of the environment.Sharing information can help agents construct a better picture of the world, but broadcasting at every timestep is expensive in large systems.
  • Limitations of prior work: IC3Net uses a learned binary REINFORCE gate, but its high-variance reward signal produces fluctuating, difficult-to-interpret communication behavior.The gate lacks an explicit account of why communication should occur at a particular moment.
  • Proposed approach: The proposed method triggers communication when agents’ belief distributions over the latent world state differ sufficiently, making disagreement the basis for information exchange.Each belief is computed by applying a linear layer and softmax to an agent’s LSTM hidden state; the trigger uses KL divergence between previous-timestep beliefs and threshold ϵ.
  • Evaluation: IC3Net outperforms KL-belief at all thresholds on PP 10×10, while the proposed trigger is evaluated across Predator-Prey sizes and MPE simple spread against IC3Net, CommNet, and an independent controller.The experiments use 5 seeds for each Predator-Prey environment size.

2 Background and Related Work

Multi-agent coordination under partial observability depends on agents’ differing belief states, which motivate communication. Prior methods learn continuous messaging, binary gates, or structured protocols, whereas this work uses KL divergence between belief distributions to determine when agents communicate.

  • Dec-POMDP formulation: In a Dec-POMDP, agents act from local observations while sharing a hidden environment state and receiving rewards from state-action transitions.The hidden state is shared, but each agent’s observation does not fully reveal it.
  • Belief states: Belief states are sufficient statistics for optimal decisions under uncertainty, but agents observing different parts of the environment develop different beliefs that motivate communication.The paper frames belief disagreement as the central multi-agent extension of single-agent POMDP reasoning.
  • CommNet: CommNet continuously broadcasts LSTM hidden states and averages teammates’ representations, enabling differentiable end-to-end training but preventing selective communication in competitive or mixed settings.Its global average reward also makes individual credit assignment difficult.
  • IC3Net: IC3Net adds individual rewards and a learned binary communication gate trained with REINFORCE, supporting cooperative, competitive, and mixed settings.The gate can learn profitable behavior, but its reward-only mechanism lacks an explicit uncertainty or belief-disagreement signal and produces unstable, erratic communication during training.
  • Related communication methods: Prior structured methods improve message targeting or interpretability, but attention-based communication does not provide principled uncertainty, discrete-symbol methods do not address when to communicate, and CTDE methods omit explicit gating.Hill et al. instead address what to communicate through predicted future plans and world-model messages.
  • Novelty: This work replaces IC3Net’s REINFORCE-trained gate with a KL-divergence trigger between learned belief distributions, bridging POMDP belief representations and multi-agent communication.The proposed signal is belief disagreement: agents communicate when their learned beliefs differ sufficiently.

3 Methodology

The method preserves IC3Net’s shared LSTM architecture and individual-reward training while replacing its learned REINFORCE gate with a deterministic KL-divergence trigger. A belief head produces policy-shaped distributions over latent world states, enabling communication decisions based on belief disagreement.

  • Base architecture: The method keeps IC3Net’s core architecture unchanged and replaces only its communication gate.Each agent uses an LSTM to process local observations, while action and value heads operate from the hidden state.
  • Belief representation: The belief head maps each agent’s LSTM hidden state to a probability distribution over K latent world-state categories.The output uses a softmax, with K = 8 categories in all experiments.
  • KL-based gating: Agents communicate exactly when pairwise KL divergence between their belief distributions exceeds a fixed threshold ϵ.This deterministic trigger replaces the original IC3Net gate trained with REINFORCE.
  • KL-based gating: The gate uses previous-timestep beliefs to avoid circularity, ensuring each communication decision depends only on information available before communication.At episode start, uniform previous beliefs imply zero KL divergence, so all agents communicate at the first timestep.
  • Training: The full model is trained with REINFORCE and individual rewards, while belief-head parameters receive the same policy-gradient signal and the communication gate receives no gradient.The belief head has no separate loss, so its representations are shaped by policy usefulness.

4 Experiments

Experiments evaluate KL-belief on two Predator-Prey difficulty levels and MPE simple_spread, comparing communication regimes and systematically varying the KL threshold. The design uses standardized training settings, multiple seeds, and metrics covering performance and gating stability.

  • Benchmarks: KL-belief is evaluated on IC3Net’s Predator-Prey benchmark using 5 agents on 10×10 grids and 10 agents on 20×20 grids.Episodes allow up to 40 and 80 steps, respectively, in the mixed cooperation setting.
  • Benchmarks: The method is also tested on MPE simple_spread, where 3 agents cover 3 landmarks while minimizing collisions.This provides a cooperative benchmark with a different reward structure and observation space.
  • Baselines: Comparisons include IC3Net’s learned gate, CommNet’s always-communicate policy, and an Independent Controller with no communication.T2MAC is not directly compared because its published codebase evaluates it only on SMAC.
  • Evaluation: Results report average steps, success rate, and communication-rate standard deviation across training, with lower steps and variability preferred.PP experiments use 5 random seeds, and the communication-rate deviation measures gate stability.
  • Ablations: The KL threshold is ablated over ε ∈ {0.1, 0.3, 0.5, 1.0} on PP 20×20 with 5 seeds per setting.Low thresholds can approximate always-on communication, whereas high thresholds may suppress coordination.

5 Results

Results show that KL-belief gating is environment- and threshold-dependent: it loses to IC3Net on PP 10×10 but outperforms it on PP 20×20 at ε=0.5. On MPE, the belief head improves performance even when communication remains unsuppressed, while K=8 provides the best representation tradeoff.

  • Predator-Prey 10×10: 27.22 average steps and 88.6% success make IC3Net the best PP 10×10 method, while no KL-belief threshold matches either metric.KL-belief’s closest setting is ε=0.3 at 27.26 steps and 84.7% success.
  • Predator-Prey 20×20: 75.49 steps and 28% success at ε=0.1 make KL-belief essentially comparable to IC3Net’s 75.31 steps and 31% on PP 20×20.CommNet and IC achieve 0% success across all seeds.
  • Threshold ablation: 73.84 average steps and 42% success at ε=0.5 beat IC3Net by 1.47 steps and 11 percentage points, producing an inverted U-shaped threshold response.Performance improves from ε=0.1 to 0.5, then degrades at ε=1.0 as communication becomes too frequent or too sparse.
  • Communication behavior: ≈0.38 communication rate settles into a stable band for KL-Belief ε=0.5, whereas IC3Net’s REINFORCE gate fluctuates throughout training.At ε=0.1, belief disagreement exceeds the threshold almost constantly, causing erratic gating similar to always communicating.
  • MPE simple_spread: Approximately 12 reward points and ±0.32 versus ±8.49 variance favor KL-belief over IC3Net on MPE at ε≥1.0, despite communication remaining at 1.0.This supports separate benefits from selective gating when beliefs converge and improved latent representations when gating is inactive.
  • Belief size and baseline limitations: K=8 achieves the best PP 20×20 performance and tightest seed variance, while K=4 lacks capacity and K=16 produces 0.340 communication-rate standard deviation.The original IC3Net result of 52.4 steps was not reproduced; this work’s baseline was approximately 75 steps across five seeds.

6 Conclusion

The paper proposes replacing IC3Net’s REINFORCE-trained gate with a KL-divergence trigger over learned belief distributions. Experiments find that calibrated gating helps most in harder Predator-Prey environments, while the reported conclusion also identifies benefits when selective gating is absent.

  • Contribution: The proposed gate replaces IC3Net’s REINFORCE-trained black-box mechanism with a KL-divergence trigger over learned belief distributions.Each agent derives its belief distribution from its LSTM hidden state.
  • Method: Agents communicate only when the KL divergence between their beliefs exceeds a fixed threshold, using a softmax belief head trained end-to-end with the existing policy gradient.The belief head consists of a small linear layer followed by a softmax and requires no additional training signal.
  • Experimental findings: 1.47 steps and 11pp improvement are reported for PP 20×20 at ε = 0.5, where KL-belief shows a clear inverted U-shape.PP 10×10 shows no advantage, while the performance advantage over IC3Net grows with environment complexity when the threshold is properly calibrated.
  • Experimental findings: The experiments also report benefits when the gate does not selectively communicate.The supplied conclusion passage introduces this second finding but does not provide its quantitative details.

A Additional Results

On PP 20×20, KL-Belief ε=0.5 achieves a lower average-steps plateau than IC3Net over training. Higher ε generally suppresses communication, while IC3Net remains lowest throughout.

  • PP 20×20: 73.84 steps: KL-Belief ε=0.5 converges to a lower average-steps plateau than IC3Net at 75.31 steps on PP 20×20.Results use 5 seeds and report mean ± std.
  • PP 20×20: Higher ε generally suppresses communication rate across all ε values on PP 20×20.Communication rate is measured over training with 5 seeds and mean ± std.
  • PP 20×20: IC3Net remains lowest in communication rate throughout training on PP 20×20.The comparison covers all ε values shown in the training curves.
Loading 2608.14559v1…