PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Individually Inferred Communication for Multi-Agent Cooperation

Ziluo Ding, Tiejun Huang, Zongqing Lu

arXiv:2006.06455v2cs.LGcs.MAstat.ML

TL;DR

Broadcast communication in multi-agent cooperation can be impractical and redundant, while agents need useful information to coordinate. I2C learns whom to contact using causal influence inferred from a joint action-value function, and experiments report improved performance across cooperative scenarios alongside reduced communication overhead.

  • Problem

    Broadcast communication in multi-agent cooperation can consume bandwidth, add delay, and provide redundant information that may impair learning.

    Source: §1 · p. 1

  • Method

    I2C learns a local-observation-based prior for whom to contact, using causal effects inferred from the joint action-value function, and regularizes policies to use messages.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    I2C outperforms existing methods across cooperative scenarios and can reduce communication overhead while improving performance.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    I2C’s one-to-one communication within agents’ fields of view is presented as more practical for real-world applications than one/all-to-all communication.

    Source: §1 · p. 2

    The work is not yet at the application stage and does not have broader impact.

    Source: Broader Impact · p. 9

Abstract

from arXiv · show

Communication lays the foundation for human cooperation. It is also crucial for multi-agent cooperation. However, existing work focuses on broadcast communication, which is not only impractical but also leads to information redundancy that could even impair the learning process. To tackle these difficulties, we propose Individually Inferred Communication (I2C), a simple yet effective model to enable agents to learn a prior for agent-agent communication. The prior knowledge is learned via causal inference and realized by a feed-forward neural network that maps the agent's local observation to a belief about who to communicate with. The influence of one agent on another is inferred via the joint action-value function in multi-agent reinforcement learning and quantified to label the necessity of agent-agent communication. Furthermore, the agent policy is regularized to better exploit communicated messages. Empirically, we show that I2C can not only reduce communication overhead but also improve the performance in a variety of multi-agent cooperative scenarios, comparing to existing methods. The code is available at https://github.com/PKU-AI-Edge/I2C.

1 Introduction

Cooperative MARL faces unstable training and execution-time miscoordination, while broadcast communication can be costly and redundant. I2C learns whom to contact from local observations and evaluates targeted communication across cooperative scenarios.

1 Introduction

  • Partial observability and stochasticity can disrupt learned cooperative strategies at execution, while policy updates make training non-stationary.These challenges can produce catastrophic miscoordination despite centralized training and decentralized execution.Source: §1 · p. 1
  • Broadcast communication consumes bandwidth and adds delay, and redundant or unhelpful messages can impair learning.Source: §1 · p. 1
  • I2C learns a prior from local observations to identify relevant and influential agents for communication, using causal effects inferred through the joint action-value function.A feed-forward network maps local observations to beliefs about whom to contact, and policy regularization helps agents use received messages.Source: §1 · p. 2
  • I2C uses one-to-one communication within agents’ fields of view and is evaluated in cooperative navigation, predator prey, and traffic junction.The paper reports improved performance in the first two scenarios and reduced communication overhead alongside improved performance in traffic junction.Source: §1 · p. 2

2 Related work

Prior communication methods include all-agent channels, gates, schedulers, and attention-based approaches, but the paper argues they retain broadcast communication. I2C instead selects relevant agents from local observations and limits contact to the field of view.

2 Related work

  • CommNet and BiCNet connect each agent to all others, while CommNet averages hidden layers and can lose information as agent count grows.DIAL permits gradients to flow between agents during training but is limited to discrete messages.Source: §2 · p. 2
  • IC3Net gates whether an agent communicates, SchedNet schedules broadcasters using all agents’ observations, and ATOC gates communication with predefined neighbors.VBC selects informative messages based on action-value variance, with agents deciding whether to broadcast to all.Source: §2 · p. 2
  • DGN and TarMAC use attention, but the paper characterizes TarMAC as all-to-all broadcast; I2C instead selects relevant, influential agents within its field of view.Figure 2 depicts I2C’s request-reply mechanism and field-of-view restriction.Source: §2 · p. 3, Figure 2 · p. 3

3 Methods

I2C combines a local-observation prior network, communication, a message encoder, a policy, and a centralized critic. It estimates communication necessity through causal effects and regularizes policies to use communicated observations effectively.

3 Methods

  • Each agent uses its partial observation to identify visible agents, request messages from selected agents, encode received observations, and choose actions; a centralized critic estimates the joint action-value function.Source: §3 · p. 3

3.1 Learning Prior Network via Causal Inference

  • The prior network learns whom to contact by labeling communication necessity through the causal effect of another agent’s action on the acting agent’s policy.The effect compares action distributions conditioned on the other agent’s action with distributions that omit it; KL divergence measures their discrepancy, estimated using the joint action-value function.Source: §3.1 · pp. 3–4
  • For communication reduction, I2C measures the effect of received messages on an agent’s action distribution to determine whether existing communication is necessary.This allows the prior network to be used with full-communication methods such as TarMAC.Source: §3.1 · p. 4

3.2 Correlation Regularization

  • Correlation regularization trains the policy using communicated observations to approximate a policy conditioned on both those observations and other agents’ actions.Because agents cannot send their actions directly without risking circular dependencies, the regularizer uses the centralized policy’s action distribution as a target.Source: §3.2 · p. 4

3.3 Training

  • The prior network is trained as a binary classifier, either end-to-end or in two phases; experiments use the two-phase approach because it learns faster.The first phase trains the prior from data generated by a pretrained CTDE algorithm; the second fixes the prior while training the rest of the architecture from scratch.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate I2C in cooperative navigation, predator–prey, and traffic junction, building on MADDPG for the first two tasks and TarMAC for traffic junction. Across these settings, I2C combines strong task performance with reduced communication, while learned communication and cooperative behaviors vary by task.

4.1 Cooperative Navigation

  • In cooperative navigation, I2C achieved the best test performance in both reward and occupied landmarks and converged to the highest reward among baselines.Source: §4.1 · p. 6, Table 1 · p. 6
  • Navigation ablations show I2C beats full and matched-volume random communication; FC beats MADDPG, RC falls below it, and correlation regularization improves on I2C-R.Source: §4.1 · p. 6
  • I2C agents select distinct landmarks and reassign or relinquish targets cooperatively, unlike MADDPG target-sharing and TarMAC/IC3Net hesitation.Source: §4.1 · pp. 6–7, Figure 4 · p. 6
  • Navigation communication overhead rises above 80% early, then declines to about 10% as agents approach targets and become less influential on one another’s policies.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2 Predator Prey

  • In predator–prey, I2C converges to the highest mean reward and performs best in testing with about 48% communication overhead; agents also learn to drive prey to corners and combine pursuits into larger encirclements.The gap between I2C and FC narrows because communication is more crucial and there is less communication redundancy in this task.Source: §4.2 · pp. 7–8, Figure 7 · p. 7, Table 2 · p. 8

4.3 Traffic Junction

  • In hard traffic junction, I2C+TarMAC alone exceeds 90% success, versus TarMAC 89.24%, IC3Net 40.47%, and no communication 48.10%; medium-mode I2C+TarMAC and TarMAC both exceed 97%.The authors note that methods perform similarly in easier settings and that their results differ from the original papers because traffic is heavier and car vision is restricted.Source: §4.3 · pp. 8–9, Table 3 · p. 9
  • Traffic-junction communication overhead falls nearly 34% in medium mode and nearly 37% in hard mode, concentrating before junctions and easing after crossings.Source: Figure 8 · p. 8, §4.3 · p. 9

5 Conclusions

I2C learns a prior for agent-agent communication from causal effects between agents, while regularizing policies to use messages more effectively. The paper reports performance gains across cooperative multi-agent scenarios and compatibility with CTDE frameworks using a joint action-value function.

5 Conclusions

  • I2C learns a communication prior from causal effects between agents, which the paper says captures the necessity of communication.The agent policy is also regularized to better use communicated messages.Source: §5 · p. 9
  • I2C can serve as a communication-reduction component and can be instantiated in CTDE frameworks that use a joint action-value function.Source: §5 · p. 9
  • Experiments demonstrate that I2C outperforms existing methods across a variety of cooperative multi-agent scenarios.Source: §5 · p. 9

Broader Impact

The authors describe I2C as a promising approach to targeted communication based on causal influence, but state that it is not yet at the application stage and has no broader impact. Its one-to-one communication is presented as making it more practical for real-world applications.

Broader Impact

  • The authors state that I2C is not yet at the application stage and does not have broader impact.Source: Broader Impact · p. 9
  • I2C learns one-to-one rather than one/all-to-all communication, which the authors say makes it more practical for real-world applications.Source: Broader Impact · p. 9
  • The authors characterize their experimental results as encouraging evidence for targeted communication based on causal influence.Source: Broader Impact · p. 9

A Environmental Settings

The appendix describes cooperative navigation and predator-prey settings alongside traffic-junction scenarios, specifying agent interactions, task layouts, actions, and reward elements.

A Environmental Settings

  • In cooperative navigation, seven agents occupy seven landmarks; in predator-prey, seven predators pursue three faster, moving preys, and each agent can communicate with only its three closest same-type agents.The agents use discrete actions; predators and agents in cooperative navigation have task-specific acceleration settings.Source: A Environmental Settings · p. 12
  • Traffic Junction uses two-way-road grids: medium mode has one junction on a 14 × 14 grid, while hard mode has four junctions on an 18 × 18 grid.The maximum episode lengths are 40 and 80 steps, respectively; Nmax and parrive are 10 and 0.05 in medium mode and 20 and 0.03 in hard mode.Source: A Environmental Settings · p. 12
  • The traffic-junction environment is illustrated as a grid-based road setting.Source: Figure 9 · p. 12
  • Traffic-junction cars follow specific routes and choose gas or brake; collisions incur a −10 reward and time incurs a −0.01 reward.The individual car reward also depends on collisions and time spent in the junction.Source: A Environmental Settings · p. 12

B Implementation Details

Implementation uses task-specific network designs, and the causal-effect threshold is selected from percentile candidates by grid search.

B Implementation Details

  • For cooperative navigation and predator-prey, I2C uses MADDPG with three-layer centralized critics and policy networks, a two-layer prior network, and two LSTM message-encoder layers.Hidden layers have size 128, and Leaky ReLU units are used.Source: B Implementation Details · p. 12
  • For traffic junction, I2C is built on TarMAC; its centralized critic and prior network have two fully connected layers, while a GRU state encoder processes trajectories, historical messages, and observations.Hidden layers have size 128, and Tanh units are used.Source: B Implementation Details · p. 12
  • The causal-effect threshold δ is selected by grid-searching candidate values drawn from percentiles of the training-set causal-effect distribution.The examples given are the 90th, 80th, and 70th percentiles.Source: B Implementation Details · p. 12
  • Tables 4 and 5 summarize the experimental hyperparameters for the navigation and predator-prey tasks, and for traffic junction, respectively.Source: B Implementation Details · p. 12, Table 4 · p. 12, Table 5 · p. 13

C Additional Experiment

In cooperative navigation, lowering the prior-network threshold improves learning through 70%, while further decreases worsen convergence. Communication overhead rises as the threshold falls.

C Additional Experiment

Loading 2006.06455v2…