Source-linked AI summary
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, Jun Wang
TL;DR
Coordinating multiple agents at a human level remains challenging because individual-action methods lack team-level collaboration mechanisms. This paper introduces BiCNet, which learns communication and coordination through a vectorised actor-critic framework, outperforming baseline models in most tested scenarios and learning diverse human-like strategies.
Problem
Single-agent methods lack a principled mechanism for fostering team-level collaboration, limiting progress toward learned human-level multiagent collaboration.
Method
BiCNet combines bidirectional recurrent communication with a vectorised multiagent actor-critic framework and parameter sharing for scalable StarCraft combat learning.
Results
In 4/5 scenarios, BiCNet outperforms other baseline models and learns diverse coordination strategies, including hit-and-run, cover attack, and focus fire.
Takeaways & Limitations
BiCNet demonstrates effective collaboration across diverse StarCraft combats and reproduces several human-level coordination strategies through end-to-end learning.
Takeaways & Limitations
Combat outcomes are sensitive to which side attacks first, creating an evaluation scope boundary in the tested scenarios.
Abstract
from arXiv · showhide
Many artificial intelligence (AI) applications often require multiple intelligent agents to work in a collaborative effort. Efficient learning for intra-agent communication and coordination is an indispensable step towards general AI. In this paper, we take StarCraft combat game as a case study, where the task is to coordinate multiple agents as a team to defeat their enemies. To maintain a scalable yet effective communication protocol, we introduce a Multiagent Bidirectionally-Coordinated Network (BiCNet ['bIknet]) with a vectorised extension of actor-critic formulation. We show that BiCNet can handle different types of combats with arbitrary numbers of AI agents for both sides. Our analysis demonstrates that without any supervisions such as human demonstrations or labelled data, BiCNet could learn various types of advanced coordination strategies that have been commonly used by experienced game players. In our experiments, we evaluate our approach against multiple baselines under different scenarios; it shows state-of-the-art performance, and possesses potential values for large-scale real-world applications.
Abstract · Introduction
The paper frames human-level collaboration among multiple AI agents as a central challenge for AGI and studies it through StarCraft micromanagement. It proposes BiCNet with multiagent actor-critic learning and parameter sharing, reporting learned human-like coordination and state-of-the-art performance.
- Introduction: Recent AI systems have exceeded human-level perception in some supervised tasks and defeated humans in several games through reward-based learning.The cited examples include visual and speech recognition, Atari, Go, and card games.
- Introduction: Human intelligence also depends on social and collective wisdom, motivating multiagent collaboration as an essential direction toward AGI.The passage illustrates collective problem-solving through crowdsourcing and organized social animals.
- Introduction: A grand challenge is enabling multiple AI agents to learn human-level collaboration or competition from experience under shared incentives and economic constraints.The paper connects this challenge to the growing capabilities of deep reinforcement learning for multiagent problems.
- Introduction: The study uses StarCraft micromanagement, where agents with different unit functions must collaborate to destroy opponents under varying terrain conditions.The game is presented as a difficult computer task and a real-time strategy case study for intelligent collaborative behavior.
- Introduction: The authors formulate StarCraft combat as a zero-sum Stochastic Game and use BiCNet for communication within a multiagent actor-critic framework.BiCNet is the proposed bidirectionally-coordinated network connecting the agents during learning.
- Introduction: Parameter sharing is introduced to address scalability, while BiCNet is designed to support coordination among multiple agents.The method combines communication through BiCNet with actor-critic learning and shared parameters.
- Introduction: BiCNet automatically learns optimal coordination strategies resembling those used by experienced human players, outperforms state-of-the-art methods, and shows potential for real-world multiagent applications.The reported potential extends beyond the StarCraft combat setting to a wide range of multiagent tasks.
Related Work
Prior work studied multiagent collaboration, learned communication, and StarCraft combat, but existing approaches faced limitations in observability, heterogeneity, scalability, or coordination. BiCNet addresses these concerns through bidirectional recurrent communication and latent-space information sharing with efficient gradient propagation.
- Multiagent Collaboration: Multiagent reinforcement learning has long targeted optimal collaboration policies, with learned inter-agent communication as a central component.Early studies began with toy examples before reinforcement learning was applied broadly to multiagent systems.
- Communication Architectures: DIAL uses recurrent networks to produce each agent’s Q-value and a transferable message at every time step in partially observable settings.DIAL permits message passing and treats agents as independent learners.
- Communication Architectures: CommNet uses one shared network and averaged inter-agent messages for joint action learning, but its symmetry limits heterogeneous-agent handling and scalability.The architecture is designed for fully observable settings and embeds communication within the original network.
- StarCraft Multiagent Systems: StarCraft research included full-game interfaces and heuristic or greedy approaches for micromanagement, while significant progress remained difficult for state-of-the-art reinforcement learning.The cited studies focused on deterministic policies or greedy MDP formulations in specific StarCraft games.
- BiCNet: BiCNet connects individual agent policy and Q networks through bidirectional recurrent communication and propagates gradients efficiently across agents.Communication occurs in latent space, allowing high-level information transfer while retaining a simple approach.
- Subsequent Multiagent Methods: Later methods addressed multiagent nonstationarity and credit assignment through replay-buffer improvements, counterfactual rewards, and centralized critics with decentralized actors.These developments included testing fully decentralized DIAL and introducing the COMA model.
Multiagent Bidirectionally-Coordinated Nets
BiCNet introduces bidirectional recurrent actor and critic networks that enable communication among cooperative agents before action selection, addressing the lack of principled team-level collaboration in single-agent methods. Its ordered recurrent communication supports full inter-agent dependency while preserving social conventions and roles, within a stochastic-game formulation for StarCraft combat.
- Problem formulation: The combat task is formulated as a zero-sum stochastic game between N cooperative agents and M enemies sharing a state space, with homogeneous action spaces within each camp.The framework also permits heterogeneous agents through different parameters and action spaces, while the presented enemy policy is fixed, reducing the problem to an MDP.
- Motivation: BiCNet enables communication among agents immediately before individual actions, providing a principled mechanism for team-level collaboration beyond single-agent policy-gradient methods.The method was proposed because deterministic policy-gradient approaches learn individual actions but lack a mechanism for fostering team-level collaboration.
- Architecture: BiCNet uses multiagent actor and critic networks based on bidirectional RNNs, combining shared observations and local views to produce each agent’s action.The recurrent structure serves as both a communication channel and local memory.
- Learning: Gradients from all agents’ rewards and actions are aggregated and propagated through the network to update shared policy parameters.Learning can be viewed as unfolding the network across N controlled agents and applying backpropagation through time.
- Communication structure: Unlike CommNet, BiCNet permits full dependency among agents while using a fixed recurrent joining order to preserve social conventions and resolve ties between optimal joint actions.The fixed order introduces asymmetry rather than fully symmetric communication.
Experiments
Experiments evaluate BiCNet across StarCraft combat settings against rule-based and deep reinforcement learning baselines, then analyze convergence, learned coordination, and communication. BiCNet generally performs strongly, especially in larger combats and coordination-dependent tasks, while additional studies clarify learned representations and message behavior.
- Combat experiments: BiCNet is evaluated across combat difficulties defined by varying unit types and numbers, against built-in AI, Attack the Weakest, Attack the Closest, IND, FC, and GMEZO baselines.The experiments measure winning rates across different StarCraft combat settings.
- Training analysis: BiCNet converges much quicker than the two main StarCraft baselines, while batch size 32 achieves the highest winning rate and mean Q-value after 600k training steps.The batch-size study uses 800 training episodes and 100 independent test games in the 2 Marines versus 1 Super Zergling combat.
- Combat experiments: In 4/5 scenarios, BiCNet outperforms the other baseline models, with its performance gap over the second best increasing when the number of agents exceeds 10.BiCNet is trained for 100k steps and evaluated by average winning rate over 100 test games.
- Combat analysis: In 5 M vs. 5 M focus-fire combat, IND and FC perform relatively poorly, whereas GMEZO, CommNet, and BiCNet leverage explicit or implicit multiagent collaboration.The result is attributed to the absence of explicit collaboration mechanisms in IND and FC training.
- Learned coordination: High-Q states cluster together in t-SNE embeddings, including coordinated cover attacks against distant enemies that cannot attack the agents.The embeddings use critic-network last-hidden-layer values collected over 10k steps.
Emerged Human-level Coordination
With adequate training from scratch, BiCNet discovers human-like collaboration strategies, including coordinated movement, Hit and Run, cover attacks, focused fire, and heterogeneous-agent collaboration. These learned policies reduce collisions and support coordinated behavior in both small and large combats.
- Coordinated moves without collision: After typically over 40k steps in near 50 episodes, collisions reduce dramatically and coordinated moves emerge as training stabilizes.Initially, nearby agents often unintentionally block one another’s paths.
- Hit and Run tactics: BiCNet rapidly learns Hit and Run, moving agents away under attack and fighting back once they are safe, in single- and multi-agent settings.The paper describes Hit and Run as a common human-player tactic and a basis for more advanced strategies.
- Coordinated cover attack: BiCNet masters coordinated cover attacks by having one agent draw enemy attention while others exploit the resulting time or distance gap to attack.The strategy requires sequentially coordinated Hit and Run movements by multiple agents.
- Focus fire without overkill: BiCNet focuses agents’ fire on particular enemies while spreading agents to the sides, allocating attacks efficiently and avoiding overkill.The approach avoids both scattering fire across enemies and wasting attacks by concentrating excessively on one target.
- Collaborations between heterogeneous agents: Heterogeneous collaboration is implemented by limiting parameter sharing to units of the same type, enabling coordination among units with different functionalities and action spaces.StarCraft includes many unit types with distinct strengths and weaknesses.
Conclusions
The paper introduces BiCNet, a deep multiagent reinforcement-learning framework using vectorised actor–critic learning and bidirectional recurrent communication. End-to-end learning enables effective coordination strategies across diverse combats, while future work will test human competition and investigate communication in more complex settings.
- Contributions: BiCNet introduces deep multiagent reinforcement learning with a vectorised actor–critic framework whose dimensions correspond to individual agents.The framework learns actions jointly across agents.
- Coordination mechanism: Bidirectional recurrent communication in internal layers enables agents to coordinate through end-to-end learning.The communication mechanism is integrated into the network rather than relying on external supervision.
- Empirical conclusion: BiCNet successfully learns several effective coordination strategies and demonstrates collaboration across diverse combats.These capabilities were demonstrated through the paper’s experiments.
- Future work: Future experiments will evaluate machine competition with human players at different levels.The authors identify human-versus-machine evaluation as a next step.
- Future work: Future work will investigate policy communication in more complicated settings and whether a specific language emerges among StarCraft agents.The proposed investigation concerns communication over the networks among agents.
Supplementary Material
The supplementary material derives a multiagent deterministic policy-gradient theorem under regularity, boundedness, continuity, and deterministic-policy assumptions. It also specifies the BiCNet training algorithm, combining replay sampling, Bi-RNN targets, gradient updates, Adam optimization, and target-network updates.
- Theoretical derivation: Regularity and boundedness conditions enable Leibniz’s integral rule and Fubini’s theorem in deriving the multiagent deterministic policy-gradient theorem.Because the policy and environment transition matrix are deterministic, expectation is taken only over the initial state.
- Theoretical derivation: Continuity of Qaθ_i(s, aθ(s)) justifies exchanging differentiation and integration, while Q-value definitions and iterated expansions produce the final theorem.The derivation combines terms, repeatedly expands the second term, and uses Fubini’s theorem to exchange integration order.
- BiCNet algorithm: BiCNet initializes actor, critic, target networks, and replay memory before running episode- and timestep-level interaction loops.The algorithm also initializes a random exploration process and receives the initial observation state.