Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control

• MARL agents act from partial observations while strategy changes can destabilize training and overfitting to other agents’ behavior can deteriorate performance.Source: §1 · p. 1
• Full communication adds latency and bandwidth overhead, and excessive communication may introduce useless or harmful information that impairs learning convergence.Source: §1 · p. 2
• VBC adds a variance penalty to exchanged messages to extract informative content without an extra module for dynamically choosing communication patterns.Source: §1 · p. 2
• Agents request help when confidence in their preliminary local decision is low and reply only when their message is informative.Source: §1 · p. 2
• 20% higher winning rates and 2-10× lower communication overhead on average than other benchmark algorithms were reported for VBC on StarCraft Multi-Agent Challenge.Source: §1 · p. 2

• IQL lets agents learn and act independently, but does not account for interaction-driven environment dynamics and can lead to collided actions during execution.Source: §2 · p. 2
• VDN, QMIX, actor-critic methods, and COMA use centralized training with decentralized execution, and assume no communication during execution [18] [14].Source: §2 · p. 2
• CommNet and BiCNet use all-to-all communication, which can create substantial communication overhead and latency [13].Source: §2 · pp. 2–3
• Other methods select communication patterns, but may restrict partners or the number of agents that communicate; VBC avoids hard pattern constraints and an extra scheduling module [8].Source: §2 · p. 3

• DQN represents action values with a deep neural network, learns by minimizing a temporal-difference loss, and commonly selects actions using an ϵ-greedy policy.Source: §3 · p. 3
• In cooperative MARL, each agent receives a local observation and chooses an action, while the agents seek to maximize their joint reward.Source: §3 · p. 3
• DRQN uses a recurrent neural network to integrate prior observations and knowledge into action-value estimates.Source: §3 · p. 3
• VDN represents the joint action value as a sum of individual action values, while QMIX uses a neural network conditioned on global state information.Source: §3 · p. 3

4 Variance Based Control

• VBC limits the variance of transferred messages; local decision ambiguity is measured by the gap between the two largest action values, and replies are selected for high-variance feedback.Source: §4 · pp. 3–4

4.1 Agent Network Design

• Each agent combines local action values with encoded messages; shared local-generator and message-encoder parameters are intended to prevent lazy-agent behavior and reduce complexity.Source: §4.1 · p. 4

4.2 Loss Function Definition

• Training aggregates agents’ action values through a mixing network and adds a loss penalty on the variance of message-encoder outputs.Source: §4.2 · p. 4

4.3 Communication Protocol Design

• An agent broadcasts a low-bandwidth request when its local decision is uncertain, and other agents reply only when their messages have high variance and may change that decision.Source: §4.3 · p. 5

5 Convergence Analysis

• The convergence analysis uses a tabular setting and omits the action-value function’s dependence on previous knowledge h_t.Source: §5 · p. 6
• The stated bound assumes finite sets of possible actions and states and is obtained by iterating equation 2.Source: §5 · p. 6
• The analysis states that iterated updates approach the optimal joint action-value function, with asymptotic error bounded by λNG.The bound applies for all observations and actions as k→∞.Source: §5 · p. 6

6 Experiment

• SMAC evaluation covers six symmetric and asymmetric battles; each algorithm is trained 15 times, with winning rates tested every 200 episodes.Source: §6 · p. 6, §6.1 · p. 7

6.1 Results

• Communication-enabled algorithms beat VDN and QMIX across all six tasks; VBC variants beat SchedNet and match or sometimes outperform FC.Source: Figure 3 · p. 7, §6.1 · p. 7

6.2 Communication Overhead

• 10× lower communication overhead for MMM and 2s3z, and 2–6× less traffic for the remaining tasks, versus SchedNet.Source: §6.2 · p. 8, Table 1 · p. 8

6.3 Learned Strategy

• In 6h_vs_8z, VBC forms a Hydralisk row and communicates mainly during formation or when deaths disrupt it; SchedNet and FC form less stable rows.Source: §6.3 · pp. 8–9
• In 3s_vs_4z, VBC and FC use healthier Stalkers as shields while distant units fire, communicating when the group breaks and needs realignment.Source: §6.3 · p. 9
• In 6z_vs_24zerg, VBC splits the Zealots to exploit Zergling sight limits, while FC and SchedNet do not fine-tune positions to kill the remaining enemies.Source: §6.3 · p. 9
• In symmetrical battles, VBC, FC, and SchedNet prioritize enemy Zealots before Stalkers; in MMM, nearly all methods target the Medivac first.Source: §6.3 · p. 9

6.4 Evaluation on Cooperative Navigation and Predator-prey

• VBC achieves the best performance in Cooperative Navigation and Predator-prey, with communication overheads of 8.80% and 10.07%, respectively.Source: Figure 5 · p. 8, §6.4 · p. 9

7 Conclusion

• VBC constrains the variance of exchanged messages during training to improve communication efficiency and cooperation among agents.The conclusion characterizes it as a simple and effective approach for MARL.Source: §7 · p. 9
• StarCraft Multi-Agent Challenge tests indicate that VBC significantly outperforms other state-of-the-art methods in winning rate and communication overhead.Source: §7 · p. 9
• The conclusion describes VBC as a simple and effective approach to efficient communication among MARL agents.Source: §7 · p. 9

8.1 Proof for theorem 1

• The proof assumes finite k and a finite number of possible actions and states.Source: §8.1 · p. 10
• As k approaches infinity, the norm of the Q-value error is bounded by λNG for all observations and actions.Source: §8.1 · p. 10
• The proof states that one error component converges to zero with probability one.Source: §8.1 · p. 10

9.1 StarCraft micromanagement challenges

• Six StarCraft II battles comprise three symmetrical matchups and three harder unsymmetrical matchups with different compositions and fewer user-controlled units.The user controls allied units against the built-in AI, aiming to defeat enemy units while minimizing damage to its own units.Source: §9.1 · pp. 10–11
• The benchmark provides local observations of nearby allied and enemy units, and QMIX's global state includes unit status, previous actions, and map-relative coordinates.Source: §9.1 · p. 11

9.2 Hyperparameter

• Each agent passes its input through a single-layer MLP and GRU before a fully connected local action-value layer; a two-layer message encoder feeds a combiner that sums outputs elementwise.The MLP intermediate result is size 64; message-encoder layers have 196 and 64 units, respectively.Source: §9.2 · p. 11
• Training uses γ = 0.99, a 5,000-episode replay buffer, batch sizes 32 for training and 8 for testing, and RMSprop at η = 5 × 10−4.Exploration ϵ decreases from 1.0 to 0.05 over 200,000 timesteps and stays at 0.05; α = 0.99.Source: §9.2 · pp. 11–12