Source-linked AI summary
Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
Hangyu Mao, Zhibo Gong, Zhengchao Zhang, Zhen Xiao, Yan Ni
TL;DR
Limited bandwidth is a practical constraint on multi-agent communication that existing DRL methods have largely overlooked. The paper introduces a gating mechanism to adaptively prune unprofitable messages. It reports pruning more than 80% of messages with little performance damage, while outperforming several DRL-based and rule-based methods across packet-routing and benchmark tasks.
Problem
Existing DRL communication methods largely ignore limited bandwidth, even though real systems constrain message transmission resources and require fewer messages without sacrificing performance.
Source: §1 · p. 1Method
The paper extends actor-critic communication with a gating mechanism that encourages messages judged beneficial to the team and prunes unprofitable ones toward a desired message quantity.
Source: §1 · p. 1, §4.2 · pp. 3–4Results
The method prunes more than 80% of messages with little performance damage and outperforms several state-of-the-art DRL-based and rule-based methods on packet-routing and benchmark tasks.
Source: §1 · p. 1Takeaways & Limitations
The gating mechanism can adapt message quantity around a desired threshold while retaining desirable performance under limited-bandwidth restrictions.
Source: §1 · p. 1, §4.2 · p. 4An alternative GatingNet for deciding whether to prune messages is left for future evaluation.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Communication is an important factor for the big multi-agent world to stay organized and productive. Recently, the AI community has applied the Deep Reinforcement Learning (DRL) to learn the communication strategy and the control policy for multiple agents. However, when implementing the communication for real-world multi-agent applications, there is a more practical limited-bandwidth restriction, which has been largely ignored by the existing DRL-based methods. Specifically, agents trained by most previous methods keep sending messages incessantly in every control cycle; due to emitting too many messages, these methods are unsuitable to be applied to the real-world systems that have a limited bandwidth to transmit the messages. To handle this problem, we propose a gating mechanism to adaptively prune unprofitable messages. Results show that the gating mechanism can prune more than 80% messages with little damage to the performance. Moreover, our method outperforms several state-of-the-art DRL-based and rule-based methods by a large margin in both the real-world packet routing tasks and four benchmark tasks.
1 Introduction
The paper addresses limited bandwidth in multi-agent communication, a practical constraint that prior DRL communication methods have largely ignored. It introduces a gating mechanism and evaluates it on packet-routing and benchmark tasks.
1 Introduction
- The paper focuses on limited-bandwidth multi-agent communication and applies its method to packet routing, a representative task and a critical Internet function.Source: §1 · p. 1
- Limited bandwidth is defined as a constraint on message-transmission resources, requiring agents to generate as few messages as possible while maintaining performance.Source: §1 · p. 1
- Bandwidth constraints are common in real systems, including limited link capacity in packet routing and limited battery capacity in the Internet of Things, but have been largely ignored by existing DRL methods.Source: §1 · p. 1
- The proposed gating mechanism learns to encourage communication when a message benefits the agent team and discourage it otherwise, adaptively pruning unprofitable messages toward a desired quantity.It extends Actor-Critic with Message Learning (ACML), whose agents otherwise send messages every control cycle.Source: §1 · p. 1
- GACML prunes more than 80% of messages with little damage to performance and outperforms several state-of-the-art DRL-based and rule-based methods on packet-routing and benchmark tasks.The paper frames packet routing as both representative of bandwidth-constrained systems and critical to the Internet.Source: §1 · p. 1
2 Background
The background formulates the task as cooperative decision-making under partial observability and introduces the reinforcement-learning foundations used by the method. In execution, actor-critic methods require only the actor, which motivates pruning messages among actors.
2 Background
- The task is framed as a DEC-POMDP: agents act from local observations in a partially observable cooperative setting with shared reward and joint actions.The formulation is attributed to Bernstein et al., 2002.Source: §2 · pp. 1–2
- Each agent learns a policy from its observation history to maximize expected discounted return.The paper uses observation history rather than only the current observation as the policy input.Source: §2 · p. 2
- Reinforcement learning is presented as solving single-agent DEC-POMDP problems, with policies chosen to maximize Q-values.The cited RL reference is Sutton and Barto, 1998.Source: §2 · p. 2
- DDPG uses neural networks for a deterministic actor and critic, with target networks and experience replay to stabilize training and improve data efficiency.The cited methods are DPG and DDPG [Lillicrap et al., 2015].Source: §2 · p. 2
- Because the critic is used only during training and execution needs only the actor, GACML prunes messages among actors.Source: §2 · p. 2
3 Related Work
Related work ranges from predefined or model-based communication to DRL methods that commonly communicate every control cycle. The paper positions GACML among adaptive communication methods, emphasizing its claimed ability to support heterogeneous agents and control message quantity toward a threshold.
3 Related Work
- Traditional approaches may predefine messages or optimize them for a predefined control policy; Value of Communication instead compares expected values of communicating and staying silent but requires environment knowledge.Source: §3 · p. 2
- DRL communication models such as CommNet, DIAL, BiCNet, and AMP use messages in every control cycle, limiting their fit to bandwidth-constrained systems.The passage also cites [Mao et al., 2017; Mordatch and Abbeel, 2018; Mao et al., 2018].Source: §3 · p. 2
- Independent-policy methods such as MADDPG and COMA [Foerster et al., 2017] avoid communication but suffer from partial observability because agents do not exchange information.Source: §3 · p. 2
- Adaptive methods decide whether to send messages, but ATOC and IC3Net suit homogeneous agents and cannot control the number pruned; GACML is described as supporting both homogeneous and heterogeneous agents and a pruning threshold.Source: §3 · p. 2
- Other adaptive approaches have task or design constraints: MADDPG-M is limited to 2D tasks with measurable agent-target distances, while SchedNet relies on a hard-coded communication count M.Source: §3 · p. 2
- GACML is presented as the first formal method, to the authors’ knowledge, that controls message quantity to a desired threshold, while prior RL communication models remain inflexible under bandwidth restrictions.Source: §3 · pp. 2–3
4 The Proposed Method
ACML combines message generation, coordination, and a shared critic; GACML adds a learned gate to prune messages whose communication benefit falls below a threshold. Its training signal and zero-padding design support adaptive pruning while preserving useful communication.
4.1 ACML
- ACML combines separate actor and message-generator branches with shared coordinator and critic networks, making the model fully observable and training stationary.The agents exchange local messages; the coordinator derives agent-specific global messages used with local observations to select actions.Source: §4.1 · p. 3
- ACML’s DNN components make message generation and control policy jointly optimizable by backpropagation.The shared CriticNet is used during training, while the other components are needed during execution.Source: §4.1 · p. 3
4.2 GACML
- Each GACML agent adds a GatingNet that generates a probability from its observation, thresholds it at 0.5, and gates the local message before coordinator exchange.When a message is pruned, zero padding also signals that the corresponding global message was pruned.Source: §4.2 · pp. 3–4
- Zero padding is preferred to caching a previous global message because, after long gaps between valid messages, the old message can poorly approximate the current one.Zero padding signals that the message was actually pruned.Source: §4.2 · p. 4
- The auxiliary task labels an observation positive when the communication-based action’s Q-value exceeds the independent action’s Q-value by more than T, training p to encourage communication.The threshold T controls how many messages are pruned; p above 0.5 encourages communication.Source: §4.2 · p. 4
- GACML’s gate is trained to prune messages whose communication benefit falls below threshold T, controlling message quantity while aiming to maintain performance.The auxiliary label compares Q-values for communication-based and independent actions; T sets the required Q-value gain.Source: §4.2 · p. 4
- A fixed T is selected by sorting recent Q-value differences and choosing an index for the desired pruning rate, keeping the actual rate close to that target.A dynamic T instead uses an exponential moving average; the resulting labels can adapt to changing environments.Source: §4.2 · p. 4
5 Experiments
The experiments evaluate communication and routing policies on three packet-routing topologies, including settings with limited bandwidth. ACML performs strongly on the more complex topology, while GACML prunes messages with little damage to performance.
5.1 The Experimental Settings
- The experiments compare IND-AC, MADDPG, AMP, and ACML; the four benchmark tasks are reported in the appendix, while this section presents packet-routing evaluations.The reported model settings include a 64-neuron first hidden layer, 32-neuron other hidden layers, and a replay buffer size of 106.Source: §5.1 · p. 5
5.2 The Packet Routing System
- The routing tasks use small and moderate classical Internet traffic-engineering topologies and a larger, more complex topology based on an industrial collaborator's needs.Routers cooperate because multiple routers can share a link, which can otherwise be overloaded or underloaded; the network capacity imposes a strict bandwidth limit.Source: §5.2 · p. 5
A Paths: AC AEFC BEFD BD
- Figure 3 presents small and moderate routing topologies, with paths including AF, A12F, B12G, and B34G.Routers choose flow-splitting ratios across available paths to minimize maximum link utilization.Source: Figure 3 · p. 5, A Paths: AC AEFC BEFD BD · p. 5
5.3 The Experimental Results
- On the moderate topology, ACML achieves much smaller MLU than IND-AC, AMP, and MADDPG; DRL-based methods also achieve much smaller MLU than rule-based methods in both topologies.ACML achieves the largest convergence ratio in both topologies, and its convergence ratio decreases less on the moderate topology than those of other methods.Source: Figure 4 · p. 5, §5.3 · pp. 5–6
- With a dynamic threshold on synthetic flow, GACML performs much better than MADDPG and stays close to ACML while pruning 74.3% of messages.The testing rewards are shown in Figure 5 with β = 0.8 for GACML.Source: §5.3 · p. 6, Figure 5 · p. 6
- In the small-topology synthetic-flow analysis, GACML adapts faster than MADDPG to decreasing flow and obtains rewards similar to ACML after dozens of steps.The reported bursty-flow cases usually produce gate value g = 1.Source: §5.3 · p. 6, Figure 6 · p. 6
- In the large-topology evaluation with real flow, GACML prunes more than 82% of messages with less than 7% performance damage; removing all messages causes a large reward decrease.The authors report that the remaining 1.5% of messages are important for maintaining performance.Source: Table 1 · p. 6, §5.3 · p. 6
6 Conclusion
The paper jointly learns communication strategies and control policies for cooperative agents under limited bandwidth. Its gating mechanism prunes unprofitable messages while maintaining performance, targeting packet-routing systems with limited bandwidth.
6 Conclusion
- The paper jointly learns communication strategies and control policies for multiple cooperative agents.It focuses on limited-bandwidth restrictions and proposes gating to prune unprofitable messages with little performance damage.Source: §6 · p. 6
- The resulting method is presented as applicable to real-world packet-routing systems with limited bandwidth.The paper describes its approach as a principled method for controlling message quantity to a desired threshold.Source: §6 · p. 6
- The authors characterize their approach as the first formal method to address this restriction in a novel and principled way, to their knowledge.Source: §6 · p. 6