Source-linked AI summary
Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks
Amanpreet Singh, Tushar Jain, Sainbayar Sukhbaatar
TL;DR
Multi-agent continuous communication can train efficiently but has been restricted mainly to fully cooperative settings. IC3Net combines gated communication with individualized rewards, and experiments report strong performance across cooperative, mixed, and competitive tasks, including improved convergence and scalability as scale increases.
Problem
Continuous communication models support efficient back-propagation training but assume full cooperation and global rewards, limiting their use in mixed and competitive settings.
Method
IC3Net combines continuous communication with a reinforcement-learned binary gating mechanism and trains each agent using its individualized reward.
Results
IC3Net performs well in cooperative, mixed, and competitive settings, learns to communicate only when necessary, and shows scalability with improved convergence as scale increases.
Takeaways & Limitations
Agents can learn scenario-dependent communication, including stopping communication in competitive cases, while individualized rewards support the model’s use across interaction settings.
Abstract
from arXiv · showhide
Learning when to communicate and doing that effectively is essential in multi-agent tasks. Recent works show that continuous communication allows efficient training with back-propagation in multi-agent scenarios, but have been restricted to fully-cooperative tasks. In this paper, we present Individualized Controlled Continuous Communication Model (IC3Net) which has better training efficiency than simple continuous communication model, and can be applied to semi-cooperative and competitive settings along with the cooperative settings. IC3Net controls continuous communication with a gating mechanism and uses individualized rewards foreach agent to gain better performance and scalability while fixing credit assignment issues. Using variety of tasks including StarCraft BroodWars explore and combat scenarios, we show that our network yields improved performance and convergence rates than the baselines as the scale increases. Our results convey that IC3Net agents learn when to communicate based on the scenario and profitability.
1 INTRODUCTION
Multi-agent reinforcement learning must scale beyond fully cooperative settings, where agents can choose when to communicate and continuous communication models face limitations. IC3Net addresses these issues with gated communication and individualized rewards.
- Motivation: Multi-agent reinforcement learning is increasingly important for systems ranging from robot teams to networks of self-driving cars.The paper emphasizes scaling reinforcement learning to multi-agent environments for higher productivity.
- Motivation: Mixed and competitive settings remain less studied than cooperative multi-agent scenarios.Existing algorithms are often designed for a particular interaction structure, especially cooperation.
- Motivation: Agents may need to communicate selectively because communication can improve cooperation but also reveal information to opponents.The paper uses sports teams and prey avoiding predators as examples of strategic communication choices.
- Research gap: Continuous differentiable communication supports efficient back-propagation training, but prior models assume full cooperation and average global rewards.Those assumptions restrict their use in mixed or competitive scenarios.
- Contribution: IC3Net uses individualized rewards and a gating mechanism to support cooperative, mixed, and competitive scenarios.The gate lets agents block communication, while the paper evaluates scaling across multiple environments including StarCraft.
2 RELATED WORK
Prior multi-agent methods commonly struggle with nonstationarity, restricted interaction settings, or limited communication forms. IC3Net extends continuous communication toward settings with different agent rewards and interaction structures.
- Independent control: Independent controllers often perform poorly because multiple agents make the environment nonstationary and undermine naïve experience replay.This approach has been attempted with Q-learning but is reported to work poorly in practice.
- Interaction settings: Most multi-agent algorithms target cooperative interactions and are generally not applicable to competitive or mixed settings.Some methods reach cooperation indirectly through shared policy parameters.
- Continuous communication: CommNet uses continuous communication that is differentiable and trainable by back-propagation, but assumes full cooperation and exposes hidden states to all agents.IC3Net is presented as an all-scenario extension that addresses this restriction.
- Alternative coordination: MADDPG coordinates through centralized critics that access all agents’ actions rather than through direct communication between actors.Its coordination signal therefore differs from IC3Net’s communication mechanism.
- Communication mechanisms: Other work studies attention-based predictive interaction or discrete communication protocols, whereas IC3Net uses multiple continuous communication cycles and supports dynamic agent numbers.The cited comparisons also include StarCraft communication methods restricted to cooperative scenarios.
3 MODEL
IC3Net equips individually controlled LSTM agents with gated communication while retaining continuous message processing. Shared parameters preserve permutation invariance, and individualized optimization supports agents with different rewards.
- Communication gating: Each agent’s hidden state feeds a communication-action module that selects whether to communicate at the next time step.The gating function is trained alongside the action policy with REINFORCE.
- Communication aggregation: An agent receives a communication vector formed from other active agents’ gated hidden states, transformed by a linear matrix before entering its LSTM with the observation.The communication vector averages hidden states from agents whose communication actions permit inclusion.
- Model: IC3Net lets each individually controlled LSTM agent communicate its internal state through a discrete communication gate.The gate determines whether an agent communicates with others.
- Parameterization: The same LSTM parameters are shared across agents, making the model invariant to permutations of agents.Each agent is still controlled separately despite parameter sharing.
- Individual rewards: Training separate controllers with shared parameters allows each agent to maximize its individual reward instead of a single global reward.The paper links this design to applicability in cooperative and competitive scenarios and to addressing credit assignment.
4 EXPERIMENTS1
The experiments test whether IC3Net learns scenario-dependent communication gates and whether individualized rewards improve scaling and training speed relative to baselines. Evaluations vary difficulty and coordination across three environments.
- Experimental questions: The study evaluates IC3Net in cooperative, mixed, and competitive scenarios to understand how its communication mechanism works.The experiments are designed to examine behavior across different interaction structures.
- Experimental questions: The experiments ask whether agents learn to communicate only when needed according to the scenario.This directly tests whether the gating mechanism can learn when to communicate.
- Experimental questions: The study compares individualized and global rewards for scaling and convergence speed in multi-agent communication settings.The comparison tests whether individual rewards perform better than global rewards.
- Experimental design: Experiments vary difficulty and coordination across three chosen environments to assess scalability and performance.The analysis first examines gating behavior and then evaluates the network across the environments.
4.1 ENVIRONMENTS
The paper evaluates IC3Net across predator-prey, traffic-junction, and StarCraft environments spanning cooperative, mixed, and competitive multi-agent settings. These tasks vary observability, scale, routes, map size, and coordination demands.
- Environments: The experiments cover predator-prey, traffic-junction, and StarCraft BroodWars explore and combat tasks.Predator-prey uses limited vision; traffic junction requires communication to avoid collisions; StarCraft tests partially observable exploration and combat.
- Predator-Prey: Predator-prey tasks use agents with limited vision searching for a stationary prey on square grids.The study varies both the number of agents and grid size, including a 10×10 grid with five agents.
- Interaction Settings: The predator-prey environment includes cooperative, mixed, and competitive reward structures to test communication under different interaction types.Rewards increase with more successful predators in cooperative settings, decrease when others reach prey in competitive settings, and are unaffected by communication in mixed settings.
- Traffic Junction: Traffic-junction agents have zero vision and choose gas or brake while avoiding collisions across routes and junctions.Difficulty varies with possible routes, entry points, and junctions; the medium setting uses two connected junctions and 20 total cars.
- StarCraft: BroodWars: StarCraft experiments test scalability under partially observable exploration and combat with varying unit counts, enemy counts, and map sizes.Explore requires finding an enemy unit, while combat requires coordinated discovery and elimination of all enemies.
4.2 ANALYSIS OF THE GATING MECHANISM
The gating analysis examines whether IC3Net learns communication policies appropriate to cooperative, mixed, and competitive conditions. Across predator-prey and StarCraft explore settings, communication changes with profitability and reward structure.
- Experimental Setup: The gating analysis evaluates cooperative, mixed, and competitive settings in Predator-Prey and StarCraft explore tasks.The analysis uses a stationary enemy unit trained alongside predators and also tests a 50×50 StarCraft map.
- Cooperative Settings: In cooperative settings, predators communicate openly with gating values close to 1, while prey eventually learns to remain silent.Communication is generally beneficial for predators, but prey communication reveals its location and reduces its reward.
- Competitive Settings: In competitive settings, agents learn to communicate only when necessary and profitable, then stop near the prey or enemy.For StarCraft teams, reward division near an enemy makes unnecessary communication less favorable.
- Mixed Settings: In mixed settings, agents communicate selectively because communication provides neither benefit nor loss, while prey learns not to communicate.Predators communicate before reaching prey, when communication remains profitable for them.
- Conclusion: The experiments empirically suggest that agents can learn when communication is profitable, allowing one network to operate across interaction settings.This conclusion connects the learned gating behavior to the model’s use across cooperative, mixed, and competitive tasks.
4.3 SCALABILITY AND GENERALIZATION EXPERIMENTS
The experiments evaluate scalability across Predator Prey, Traffic Junction, and StarCraft settings, comparing IC3Net with communication and individualized-reward baselines. IC3Net generally improves performance and convergence as task scale increases, while its advantage depends on the scenario.
- Predator Prey: IC3Net reaches prey faster than baselines as the number of agents and maze size increase, with an almost 24-step gap in the 20×20 version.Its scalability curve also shows faster convergence to a better optimum than CommNet.
- Traffic Junction: IC3Net provides consistently better Traffic Junction success rates than the baselines, especially as scale increases.With zero vision, its performance exceeds CommNet and IRIC by more than 30%.
- StarCraft: In StarCraft Explore tasks, IC3Net outperforms the baselines, and the performance gap remains on the larger 75×75 map despite lower performance for all models.The explore evaluation uses 10 medics finding an enemy medic, while the combat evaluation uses 10 Marines versus 3 Zealots.
- StarCraft: In the StarCraft combat task, IC3Net performs comparably well to CommNet rather than clearly surpassing it.The task evaluates 10 Marines versus 3 Zealots on a 50×50 grid with a 60-step episode limit.
- StarCraft: When controlling three Zealots against ten Marines, both IRIC and IC3Net reach 100% success, while IC3Net converges faster than IRIC.This reverse scenario tests whether the original 10-Marine-versus-3-Zealot setting is difficult because of its side imbalance.
5 CONCLUSIONS AND FUTURE WORK
The paper introduces IC3Net for learning when to communicate across cooperative, mixed, and competitive multi-agent tasks. Experiments report strong performance across these settings and show that agents can communicate selectively, including stopping communication in competitive cases.
- Conclusions: IC3Net combines continuous communication trained by backpropagation with reinforcement-learned discrete gating and individualized rewards.This design targets efficient training, credit assignment, multiple cooperation settings, and larger agent scales.
- Conclusions: Across experiments, IC3Net performs well in cooperative, mixed, and competitive settings and learns to communicate only when necessary.The authors also report scalability experiments and observe that agents learn to stop communicating in competitive cases.
- Future Work: Future work proposes dynamic multi-channel communication, allowing agents to choose where to send information and whether to listen to a channel.The acknowledgements recognize support around StarCraft and TorchCraft.
6 APPENDIX
The StarCraft experiments show task-dependent differences: IC3Net improves exploration but not combat relative to CommNet, while StarCraft results exhibit substantial variance from environmental stochasticity and gating complexity.
- StarCraft results: IC3Net performs better than CommNet in StarCraft Explore but does not outperform it in Combat.Combat can be solved effectively by keeping units together and concentrating attacks, whereas exploration benefits from agents covering separate areas.
- Variance and learning difficulty: IC3Net’s StarCraft results show significant variance because stochastic interactions create many possible states, while gating expands action-state combinations.The variance is especially visible in higher-win-rate models that must learn more state spaces.
- StarCraft results: Individualized rewards support exploration by revealing each agent’s contribution to separately covered areas, unlike global rewards.CommNet agents were observed gathering before exploring, which can slow coverage.
6.3 DETAILS OF PREDATOR PREY
The predator-prey environment evaluates individualized reward designs across cooperative, competitive, and mixed settings at multiple grid sizes. IC3Net remains competitive in cooperative tasks and gains from gating in mixed settings despite added training overhead.
- Reward design: Predator agents receive a timestep penalty until reaching the prey, with cooperative and competitive rewards scaled by the number of agents on the prey.The cooperative reward is rcoop = rprey * n, while the competitive reward is rcomp = rprey / n.
- Behavioral patterns: CommNet agents can gather before searching, producing slack in predator-prey exploration.The accompanying video describes this pattern on the hard task, where agents converge before finding the prey.
- Cooperative setting: IC3Net performs better or equal to CommNet and other baselines across all three cooperative predator-prey difficulty levels.The performance gap closes or increases on larger grids, indicating improved scalability from individualized rewards.
- Cooperative setting: IC3Net performs comparably to CommNet in cooperative settings despite requiring an additional gating action.This supports using the same model across cooperation settings.
- Mixed setting: 50.24±3.4 average maximum steps were obtained by the mixed-setting baseline using individualized rewards with gating fixed to global communication.IC3Net achieved a higher value, while gating added overhead but enabled the same model to operate in competitive settings.
6.4 DETAILS OF TRAFFIC JUNCTION
Traffic Junction varies road structure, grid size, routes, and traffic density across difficulty levels, with curriculum learning used to train the harder configuration. Results indicate that collision frequency and reward assignment affect success rates.
- Training procedure: Curriculum learning keeps the arrival probability fixed for 250 epochs, increases it linearly through epoch 1250, and continues training for 750 additional epochs.The learning rate remains fixed at 0.003.
- Difficulty levels: The easy Traffic Junction uses two one-way roads on a 7 × 7 grid, with two arrival points and two routes per arrival point.Its Ntotal value is 5.
- Difficulty levels: The medium version has two connected two-way-road junctions on a 14 × 14 grid, with four arrival points and three routes per point.Its Ntotal value is 20.
- Difficulty levels: The hard version has four connected two-way-road junctions on an 18 × 18 grid, with eight arrival points and seven routes per point.The configuration increases collision opportunities and has Ntotal = 20.
- Results: IRIC and IC perform worse at medium difficulty than at hard difficulty because the medium version has a higher final arrival rate and more frequent collisions.The hard version starts with a lower arrival rate to accommodate curriculum learning for its larger grid.
6.5 STARCRAFT DETAILS
The StarCraft tasks test exploration, combat, and communication under partial observability. Exploration rewards locating enemies, while combat combines movement, attack actions, health changes, and terminal rewards, with communication accelerating shared discovery.
- Explore: In Explore, agents stop acting once they enter the enemy’s explore-vision range, and medic units prevent them from attacking enemies.The reward structure otherwise matches the predator-prey task.
- Combat: Combat observations include the agent’s position, health, shield, weapon cooldown, previous action, and visible enemies’ relative features.Agents choose among nine basic actions and one attack action for each enemy; attacks work only within sight range.
- Evaluation scope: The StarCraft combat environment is not directly compared with prior work because its setting is described as harder, more restrictive, new, and different.The paper instead evaluates its methods within this environment.
- Combat: Combat supplies a −0.01 timestep reward when an agent is not involved in combat to discourage slack in finding the enemy team.Per-timestep rewards also track changes in the agent’s health and the health of attacked enemies.
- Communication: When one agent finds the enemy in cooperative Explore, communication gives the other agents its location and helps them reach it faster.The state sequence illustrates information sharing after the initial discovery.