PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Schedule Communication in Multi-agent Reinforcement Learning

Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, Yung Yi

arXiv:1902.01554v1cs.AIcs.LGcs.MA

TL;DR

SchedNet addresses cooperative multi-agent reinforcement learning when communication is bandwidth-limited and agents contend for a shared medium. It jointly learns communication scheduling, message encoding, and action selection, with empirical results and ablation evidence indicating effective learned encoding and scheduling behavior.

  • Problem

    Cooperative partially observable multi-agent tasks require agents to coordinate communication under limited bandwidth and shared-medium contention, where only a restricted number may transmit.

  • Method

    SchedNet uses centralized training and distributed execution to learn each agent’s scheduling weight, encoded message, and action, allowing higher-value observations to access the shared medium.

  • Results

    Empirical results and an ablation study indicate that SchedNet learns effective message encoding and scheduling behavior, outperforming communication-free baselines in predator-prey.

  • Takeaways & Limitations

    Intelligent scheduling can coordinate which agents broadcast and help utilize limited bandwidth and shared-medium access in cooperative multi-agent tasks.

  • Takeaways & Limitations

    The studied setting assumes cooperative partially observable environments with agents having different observation power and restricts communication through a simple bandwidth-and-scheduling model.

Abstract

from arXiv · show

Many real-world reinforcement learning tasks require multiple agents to make sequential decisions under the agents' interaction, where well-coordinated actions among the agents are crucial to achieve the target goal better at these tasks. One way to accelerate the coordination effect is to enable multiple agents to communicate with each other in a distributed manner and behave as a group. In this paper, we study a practical scenario when (i) the communication bandwidth is limited and (ii) the agents share the communication medium so that only a restricted number of agents are able to simultaneously use the medium, as in the state-of-the-art wireless networking standards. This calls for a certain form of communication scheduling. In that regard, we propose a multi-agent deep reinforcement learning framework, called SchedNet, in which agents learn how to schedule themselves, how to encode the messages, and how to select actions based on received messages. SchedNet is capable of deciding which agents should be entitled to broadcasting their (encoded) messages, by learning the importance of each agent's partially observed information. We evaluate SchedNet against multiple baselines under two different applications, namely, cooperative communication and navigation, and predator-prey. Our experiments show a non-negligible performance gap between SchedNet and other mechanisms such as the ones without communication and with vanilla scheduling methods, e.g., round robin, ranging from 32% to 43%.

1 INTRODUCTION

The paper studies cooperative MARL with partial observability, limited communication bandwidth, and contention for a shared medium. It introduces SchedNet to learn which agents communicate, how messages are encoded, and how agents act using received messages.

  • Motivation: Cooperative MARL agents must coordinate sequential decisions while observing only partial, agent-specific information.The agents share a common discounted-reward objective and operate in a partially observable environment.
  • Communication constraints: Limited bandwidth requires messages that are concise yet meaningful, while a shared medium requires arbitration among transmitting agents.Wireless communication introduces collision-avoidance constraints and restricts simultaneous medium access.
  • SchedNet: SchedNet uses centralized training and distributed execution to learn scheduling weights, encoded messages, and action selection.Its weight-based scheduler grants access to agents whose partially observed information is judged more valuable.
  • SchedNet: The architecture targets wireless broadcasting, where a MAC protocol must arbitrate agents contending for access to a shared communication medium.The paper frames selecting communicating nodes as an additional constraint beyond bandwidth limitation.
  • Research gap: Prior communication-learning work addressed protocol learning or limited bandwidth, but not intelligent scheduling under both limited bandwidth and shared medium access.The paper identifies this combined scheduling scenario as insufficiently studied.

2 BACKGROUND

The paper frames RL with MDPs and extends the setting to decentralized partially observable control among multiple agents. Its actor-critic background uses policy optimization with a learned critic and centralized feedback for distributed actors.

  • Reinforcement Learning: An MDP specifies states, actions, rewards, transitions, and a discount factor, with RL maximizing expected discounted return.The policy induces state-value and action-value functions for evaluating future returns.
  • Actor-critic Method: The policy-gradient method adjusts policy parameters in the objective's gradient direction to maximize expected return.The gradient is expressed using sampled states and actions together with an action-value function.
  • Actor-critic Method: Actor-critic learning uses an actor to update the policy and a critic to approximate action values for gradient updates.A baseline such as the state value can reduce the variance of those updates.
  • MARL: Centralized Critic and Distributed Actor: DEC-POMDPs model multiple agents that choose joint actions from partial observations when global state access is unavailable.This formalism generalizes MDPs to distributed multi-agent control.
  • MARL: Centralized Critic and Distributed Actor: Centralized training lets the critic use the global state while each distributed actor selects actions from its own partial observation.The centralized critic provides more accurate feedback to actors with limited observation horizons.

3 METHOD

SchedNet coordinates communication under bandwidth and shared-medium constraints by jointly learning message encoding, scheduling weights, and action selection. Centralized training uses a critic, while distributed execution schedules only K_sched agents whose encoded messages are broadcast to others.

  • Weight-based scheduling: Top(k) selects the k highest-weight agents, whereas Softmax(k) samples k agents from the softmax distribution over weights.Both rules are implemented through weight-based scheduling mechanisms compatible with distributed execution.
  • Communication environment and problem: SchedNet addresses limited bandwidth and shared-medium contention by scheduling only K_sched agents to broadcast messages of limited size L_band.The goal is cooperative action selection based on selectively exchanged information.
  • Architecture: Each agent’s network includes a message encoder, action selector, and weight generator, while the scheduler maps agent weights to a schedule profile.The schedule profile indicates which agents may broadcast their encoded messages.
  • Distributed execution: Only scheduled agents’ messages are concatenated and broadcast, after which every agent selects an action using the received messages.For example, a schedule vector c determines which encoded messages enter the communication aggregate.
  • Centralized training: Centralized training uses a critic that estimates V_θc(s) for action selectors and encoders and Q^π_θc(s,w) for weight generators.The critic can access the global state during training, while actor components are updated with policy gradients and temporal-difference backups.
  • Centralized training: Weight generators are trained with DDPG over scheduling profiles satisfying ∑c_i ≤ k, using critic-estimated Q_θc(s,w) values.This separates weight-generator optimization from the encoder and action-selector updates because WSA is nondifferentiable.

4 EXPERIMENT

Experiments evaluate SchedNet in predator-prey and cooperative communication and navigation tasks against communication and scheduling baselines. Learned scheduling improves coordination, favors agents with more informative observations, and produces more informative messages.

  • Experimental setup: SchedNet is evaluated in Predator and Prey (PP) and Cooperative Communication and Navigation (CCN), alongside DIAL, round robin, and full-communication baselines.Models are trained until convergence and evaluated over 1,000 iterations; plots report 95% confidence intervals from 6–10 runs.
  • Predator and Prey: SchedNet and DIAL outperform communication-free IDQN and COMA in PP by enabling agents to share prey-location information.SchedNet also handles medium contention, whereas DIAL with round-robin scheduling can reuse outdated messages from non-scheduled agents.
  • Predator and Prey: 43% improvement over RR is achieved by SchedNet-Top(1) in PP, where deterministic scheduling outperforms comparable IDQN, RR, and SchedNet-Softmax(1) performance.The comparison attributes the variation to the scheduler while holding other training conditions fixed.
  • Predator and Prey: Agents with greater observation power receive more scheduling opportunities in PP, with scheduling chances distributed as (14, 3, 4, 4) and average weights (0.74, 0.27, 0.26, 0.26).Agent 1 has the widest observation horizon and is scheduled most often.
  • Predator and Prey: SchedNet-Top(1) messages show greater variance when an agent observes the prey, consistent with encoding more informative location-related information.Messages are projected onto a 2D plane for the PP task.
  • Cooperative Communication and Navigation: 32% improvement over RR is reported for CCN, with SchedNet outperforming IDQN and RR and SchedNet-Top(1) slightly exceeding SchedNet-Softmax(1).In CCN, Agent 1 is scheduled more frequently than Agent 2 (7 vs. 18) because Agent 2 starts farther from its destination.

5 CONCLUSION

The paper concludes that SchedNet learns distributed communication scheduling together with message encoding and action selection for fully cooperative multi-agent tasks. Empirical results and ablations indicate that learned encoding and scheduling each improve coordinated performance.

  • 5 CONCLUSION: SchedNet uses centralized critic feedback to train each agent’s message encoder, action selector, and scheduling-weight generator.The weight generators determine which k agents with more valuable observations access the shared medium.
  • 5 CONCLUSION: Empirical results and ablation studies indicate that learned message encoding and scheduling each significantly improve multi-agent performance.The conclusion connects intelligent distributed scheduling with more efficient, coordinated, and rewarding behavior.

A SCHEDNET TRAINING ALGORITHM

The training algorithm initializes actor, scheduler, critic, and target-network parameters, then iterates through episodes and time steps while computing each agent’s scheduling priority.

  • SchedNet initializes actor, scheduler, critic, target scheduler, and target critic parameters before training.
  • Each training episode begins by observing the initial state and proceeds for T time steps.
  • At each time step, the algorithm computes agent i’s priority as w_i = f_i_wg(o_i).

B.1 ENVIRONMENTS: PP AND CCN

The evaluation uses predator-prey and cooperative communication-and-navigation environments with partial observations, heterogeneous agent situations, and shared task rewards or completion criteria.

  • Predator and prey: In predator-prey, cooperating predators capture a randomly moving prey in a discretized grid world using limited local observations.The task terminates when all agents observe the prey.
  • Cooperative communication and navigation: In CCN, two agents move on separate one-dimensional grids toward distinct destinations and receive a joint reward when both arrive.The agents have zero local observation horizon but can observe the other agent, with differing initial agent-destination distances.

B.2 EXPERIMENT DETAILS

The experiments use common neural-network design choices across tasks, with task-specific complexity reflected in network sizing and completion-time evaluation.

  • The networks use rectified linear units, with three hidden layers for critics, encoders, and weight generators, and one for actors.The critic’s scheduler and action-selector components share their first two layers.
  • SchedNet evaluates CCN using the number of time steps required to complete the task.

C ADDITIONAL EXPERIMENT RESULTS

Additional experiments examine bandwidth and scheduling capacity, joint encoding and scheduling, learned message content, and recurrent networks under partial observability.

  • Impact of bandwidth and number of schedulable agents: As message bandwidth L increases, agents can encode more information for other agents’ action selection.
  • Impact of joint scheduling and encoding: 2.030 times as long as FC, SchedNet-Top(1) completed PP, compared with 3.408 times as long for the pre-trained auto-encoder variant.The comparison evaluates normalized performance in the PP environment and supports jointly training the scheduler and encoder.
  • What messages agents broadcast: SchedNet agents encode prey-location information when they observe the prey, and recipients use it to select actions.
  • What messages agents broadcast: The PP message projections show general trends across classes defined by prey and predator relative locations.
  • Partial observability issue in SchedNet: Adding a GRU layer to each agent’s encoder, action selector, and weight generator integrates recurrent processing into SchedNet.Each GRU cell has 64 hidden nodes.
  • Partial observability issue in SchedNet: IDQN with RNN requires slightly fewer average completion steps than feed-forward IDQN, while SchedNet-RNN and SchedNet perform similarly.The passage attributes the smaller recurrent-network effect in SchedNet to communication addressing partial observability in this simple environment.

C.3 COOPERATIVE COMMUNICATION AND NAVIGATION

In cooperative communication and navigation, communication improves coordination, while contention makes scheduling quality consequential. SchedNet outperforms communication-free baselines and handles contention better than DIAL.

  • SchedNet and DIAL outperform IDQN and COMA in the cooperative communication and navigation evaluation.IDQN and COMA allow no communication, and their similar performance reflects the absence of cooperation without exchanged observations.
  • Under contention, DIAL(1) requires more steps than SchedNet-Top(1) to complete the task.The passage reports that this tendency also appears in the predator-prey environment.

D SCHEDULER FOR DISTRIBUTED EXECUTION

The scheduler addresses shared-medium access by selecting only k of n agents for communication during distributed execution. SchedNet uses observation-derived weights with simple scheduling rules related to practical wireless MAC protocols.

  • D SCHEDULER FOR DISTRIBUTED EXECUTION: Only k < n agents may broadcast their encoded messages because the shared medium imposes a communication-access constraint.The value k depends on the wireless environment; for a single shared channel with mutual interference, k = 1.
  • D SCHEDULER FOR DISTRIBUTED EXECUTION: The scheduler must operate distributively while balancing implementation and training simplicity against faithful MAC-protocol behavior.
  • D SCHEDULER FOR DISTRIBUTED EXECUTION: WSA maps each agent’s observation-derived weight vector w = [w_i]_n to a scheduling profile c with exactly k selected agents.The architecture uses a weight generator and a scheduling algorithm to produce the profile.
  • D SCHEDULER FOR DISTRIBUTED EXECUTION: Top(k) selects the k highest-weight agents, whereas Softmax(k) samples k agents according to probabilities proportional to their softmax weights.
  • D SCHEDULER FOR DISTRIBUTED EXECUTION: Top(k) abstracts the MaxWeight principle, while Softmax(k) provides a simplified model of CSMA-based access used in 802.11 Wi-Fi.
  • D SCHEDULER FOR DISTRIBUTED EXECUTION: Distributed Top(k) uses CSMA-style backoff and holding times, and trained agents were successfully scheduled 98% of the time in Contiki simulations.The reported 2% failures resulted from probabilistic collisions handled by Contiki’s default collision-avoidance mechanism.
Loading 1902.01554v1…