Source-linked AI summary
Learning Transferable Cooperative Behavior in Multi-Agent Teams
Akshat Agarwal, Sumit Kumar, Katia Sycara
TL;DR
Multi-agent reinforcement learning must handle partial observability, limited communication, and environments whose entity structure is often discarded. The paper introduces a shared agent-entity graph with invariant representations and message passing, achieving successful decentralized control across tasks and transfer to different team sizes.
Problem
Multi-agent reinforcement learning faces partial observability and limited communication, while prior graph-based approaches often treat environmental structure as a black box.
Method
The method represents agents and environmental entities as graph vertices, exchanges messages along communication edges, and uses invariant fixed-size representations for variable agent and entity counts.
Results
The proposed method solves coverage, formation, and line-control tasks under partial observability, while baselines largely fail and all baselines achieve no success for six-agent tasks.
Takeaways & Limitations
Learned policies show strong zero-shot generalization to different team sizes and support curriculum learning for complex cooperative tasks in decentralized execution.
Takeaways & Limitations
The method assumes static entities and access to all entity positions at the beginning of each episode.
Abstract
from arXiv · showhide
While multi-agent interactions can be naturally modeled as a graph, the environment has traditionally been considered as a black box. We propose to create a shared agent-entity graph, where agents and environmental entities form vertices, and edges exist between the vertices which can communicate with each other. Agents learn to cooperate by exchanging messages along the edges of this graph. Our proposed multi-agent reinforcement learning framework is invariant to the number of agents or entities present in the system as well as permutation invariance, both of which are desirable properties for any multi-agent system representation. We present state-of-the-art results on coverage, formation and line control tasks for multi-agent teams in a fully decentralized framework and further show that the learned policies quickly transfer to scenarios with different team sizes along with strong zero-shot generalization performance. This is an important step towards developing multi-agent teams which can be realistically deployed in the real world without assuming complete prior knowledge or instantaneous communication at unbounded distances.
1 Introduction
The paper addresses cooperative decision-making under partial observability and limited communication by representing agents and environmental entities in a shared communication graph. Its MARL model is designed to support variable team and entity counts while enabling decentralized cooperation.
- Partial observability, limited communication, non-stationarity, large joint spaces, and credit assignment complicate cooperative multi-agent reinforcement learning.
- Existing graph-based multi-agent systems commonly treat the environment as a black box and stack agent or entity information into a single vector or image.
- The proposed shared agent-entity graph represents agents and environmental entities as vertices, with communication-enabled pairs connected by edges.
- The MARL model is invariant to the number of agents and entities and to entity ordering, supporting policy transfer across team and environment sizes.
- Figure 1 distinguishes inter-agent messages with red edges from agent-entity messages with blue edges.
2 Related Work
Related work addresses multi-agent learning, centralized training with decentralized execution, and learned communication, but differs in scalability, interaction modeling, communication restrictions, or execution decentralization.
- Independent Q-learning treats other agents as part of the environment, creating non-stationarity that prevents scaling beyond two agents.
- Centralized-critic actor-critic methods train with global information while allowing decentralized action selection during execution.
- The related methods motivate explicit coordination mechanisms for tasks requiring coordinated behavior.
- CommNet averages neighbor states, whereas VAIN uses attention to selectively weight messages from other agents.
- Prior communication studies learned protocols using differentiable or discrete messages, including compositional language with limited vocabularies.
- TarMAC permits unrestricted communication and centralized execution, while DGN communicates with three closest neighbors using Q-learning.
3 Method
The method encodes agents and entities in a shared graph, aggregates variable-sized observations into fixed-size representations, and passes attention-weighted messages over communication edges. Multi-hop communication and parameter sharing support decentralized execution, transfer, and curriculum learning.
- 3.1 Agent-Entity Graph: The graph contains agent and entity nodes, with edges determined by communication ability; entities are static within episodes and agents know their positions initially.
- 3.1 Agent-Entity Graph: Communication is either restricted by an inter-agent distance threshold or unrestricted through a fully connected agent graph.
- 3.1 Agent-Entity Graph: Graph structure provides an inductive bias by encoding that nearby agents generally have greater influence on one another.
- 3.2 Learning to communicate: Each agent encodes its local position and velocity, then combines this state encoding with a GNN-derived fixed-size embedding of the environment.
- 3.2 Learning to communicate: Entity message passing uses entities’ positions relative to the agent and produces a fixed-size environment representation independent of entity count.
- 3.2 Learning to communicate: The fixed-size representation preserves permutation invariance and avoids padding-based representations that require an upper bound on entities.
- 3.2 Learning to communicate: Agents exchange attention-weighted messages formed from learned query, key, and value vectors, then update their embeddings through a nonlinear transformation.
- 3.2 Learning to communicate: Multi-hop message passing propagates information across sparsely connected chains before value and policy heads select actions.
4 Experiments
The experiments evaluate the proposed model on coverage, formation, and line control under decentralized communication settings. Results show successful cooperation with partial observability, transfer across larger teams through curriculum learning, and zero-shot evaluation across team sizes.
- Task setup: The study evaluates coverage, formation, and line control in 2D multi-agent particle environments with decentralized action selection.Agents operate under unrestricted or distance-limited communication, and performance is measured by success rate, episode time, and average landmark distance for coverage.
- Baseline comparison: Only MADDPG solves coverage and formation for M = 3 among the compared baselines, while the proposed method solves all tasks with partial observability.The compared baselines use global state during training; the proposed method operates successfully despite partial observability.
- Baseline comparison: For M = 6, all baselines achieve no success, whereas the proposed method solves formation and line control in both communication settings and outperforms baselines on coverage.Coverage performance is not near-perfect, but remains better than every baseline.
- Curriculum training: Curriculum learning transfers a policy from M = 3 to larger teams, while entity message passing avoids the added complexity and sample requirements caused by padded observation vectors.Without entity message passing, restricted communication fails after adding two members, and unrestricted communication fails when moving from five to seven agents.
- Curriculum training: With entity message passing and curriculum learning, teams of up to 10 agents learn cooperative strategies across increasingly difficult tasks.The approach also transfers efficiently on formation and line control, where the authors attribute transferable skills to the environment structure encoded by the model.
- Zero-shot generalization: A policy trained for M = 5 achieves zero-shot success across different team sizes in both unrestricted and restricted communication settings without fine-tuning.The policy is evaluated directly on all three tasks, using the shared agent-entity graph to exploit related prior experience.
5 Conclusion and Future Work
The paper uses a shared agent-entity graph for decentralized cooperation and reports strong transfer and zero-shot generalization across team sizes. Curriculum learning helps solve complex tasks that are difficult to learn directly from scratch.
- The shared agent-entity graph represents agents and environmental entities as vertices, with cooperation learned through message exchange along graph edges.
- The model is invariant to the number of agents or entities, enabling curriculum learning and transfer across team sizes.
- The learned policies achieve state-of-the-art results on coverage and formation control in a fully decentralized execution framework.
- The learned policies show strong zero-shot generalization to scenarios with different team sizes.
- Curriculum learning solves complex tasks that are difficult to solve by directly training policies from scratch.