Source-linked AI summary
Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
Heechang Ryu, Hayong Shin, Jinkyoo Park
TL;DR
MARL has largely emphasized decentralized cooperation, leaving mixed-agent modeling and transfer across tasks with different agent compositions insufficiently addressed. HAMA combines hierarchical graph attention with multi-agent actor-critic learning to represent inter-agent and inter-group relationships, outperform existing MARL methods, transfer policies to new compositions, and interpret learned strategies.
Problem
Most MARL studies focus on decentralized cooperative policies and rarely address transfer to new tasks with different agent compositions and goals.
Method
HAMA combines hierarchical graph attention for inter-agent and inter-group representation learning with multi-agent actor-critic policy learning.
Results
HAMA outperforms existing MARL algorithms across cooperative and competitive environments and transfers policies from small games to new games with more agents.
Takeaways & Limitations
Inter-agent and inter-group attention support policy transfer across agent compositions and interpretation of learned strategies.
Takeaways & Limitations
HAMA assumes agents can be easily clustered into groups using prior knowledge.
Abstract
from arXiv · showhide
Most previous studies on multi-agent reinforcement learning focus on deriving decentralized and cooperative policies to maximize a common reward and rarely consider the transferability of trained policies to new tasks. This prevents such policies from being applied to more complex multi-agent tasks. To resolve these limitations, we propose a model that conducts both representation learning for multiple agents using hierarchical graph attention network and policy learning using multi-agent actor-critic. The hierarchical graph attention network is specially designed to model the hierarchical relationships among multiple agents that either cooperate or compete with each other to derive more advanced strategic policies. Two attention networks, the inter-agent and inter-group attention layers, are used to effectively model individual and group level interactions, respectively. The two attention networks have been proven to facilitate the transfer of learned policies to new tasks with different agent compositions and allow one to interpret the learned strategies. Empirically, we demonstrate that the proposed model outperforms existing methods in several mixed cooperative and competitive tasks.
Introduction
MARL commonly targets decentralized cooperation, but realistic mixed cooperative-competitive settings expose limitations in modeling flexibility, scalability, and transferability. HAMA addresses these issues with hierarchical graph attention and multi-agent actor-critic learning, outperforming existing methods and transferring policies across agent compositions.
- Introduction: Most MARL methods focus on decentralized policies for shared rewards, limiting their ability to model heterogeneous agents in mixed cooperative-competitive tasks.The paper identifies modeling flexibility, scalability, and transferability as unresolved issues.
- Introduction: HAMA combines hierarchical graph attention for representation learning with multi-agent actor-critic policy learning in an end-to-end model.Inter-agent and inter-group attention extract state-dependent relationships to adjust high-level cooperative or competitive strategies.
- Introduction: HAMA outperforms existing MARL algorithms across four different game scenarios.The reported evaluations include mixed cooperative and competitive environments.
- Introduction: Policies trained by HAMA in a small game can be applied directly to control more agents in a new game.The transfer result targets different agent compositions and supports policy reuse across tasks.
Related Work
Prior MARL work derives consensus through centralized critics or learned communication, while HAMA uses hierarchical graph attention to combine graph-based representation with decentralized policy learning. This design is presented as scalable, transferable, and applicable to mixed cooperative-competitive environments.
- Related Work: Learning-for-consensus methods derive decentralized actors from centralized critics built from global or individual rewards.MADDPG extends DDPG to mixed cooperative-competitive environments, while COMA and DGN use attention or graph networks for critic modeling.
- Related Work: Learning-to-communicate methods train agents to exchange and process messages during execution to coordinate individual actions.Communication procedures are learned during centralized training and used by agents during decentralized execution.
- Related Work: HAMA uses HGAT to model both centralized critics and decentralized actors, combining graph representation with attention-based state processing.The approach is framed as a learning-for-consensus method rather than a communication-based method.
- Related Work: HAMA represents partial observations through graph embeddings instead of messages processed from other agents.This distinction supports its stated use in mixed cooperative and competitive environments.
Background
The paper formulates multi-agent interaction as a partially observable Markov game and reviews deterministic actor-critic learning alongside graph attention for structured state representation. These components motivate decentralized policies based on centralized value estimation and relational embeddings.
- Background: A partially observable Markov game represents global state, local observations, agent actions, rewards, transitions, and discounted returns for multiple agents.Each agent optimizes its discounted return using its available local observation.
- Background: Deterministic policy gradients optimize a policy through the action gradient of a critic’s Q-function, with DDPG implementing actor and critic networks using deep neural networks.Experience replay stores state, action, reward, and next-state samples for optimization.
- Background: MADDPG extends DDPG to multi-agent settings by learning decentralized policy networks with Q-networks that use all agents’ observations and actions.The critic loss trains each agent’s Q-network, while the policy network is optimized through the critic gradient.
- Background: Graph attention networks compute node embeddings by aggregating neighboring node representations with learned attention weights.The attention weight αij quantifies node j’s importance to node i during embedding computation.
Methods
HAMA combines hierarchical graph attention for contextual multi-agent representations with multi-agent actor-critic policy learning. Its inter-agent and inter-group attention layers model relationships at individual and group levels while supporting scalable policy inputs and transfer across agent or group sizes.
- HAMA combines hierarchical graph representation learning with a multi-agent actor-critic network to derive decentralized policies.The graph state representation produces node embeddings that feed the actor-critic components.
- Agent Clustering.: Agents are clustered into groups using prior knowledge or data, with the framework also allowing non-acting environmental components such as terrain or obstacles to form groups.The method assumes agents can be easily clustered into K groups.
- Hierarchical State Representation Using Multi-Graph Attention.: Inter-agent attention aggregates relationships between an agent and agents in each group, while inter-group attention weights the relative importance of different groups.Higher inter-group attention toward an agent’s own group indicates cooperation; attention toward other groups indicates competition.
- HAMA computes agent embeddings from local observations without requiring other agents to exchange hidden vectors.This differs from learning-to-communicate approaches that require agents to send hidden representations.
- Multi-Agent Actor-Critic: The resulting node embeddings provide contextualized representations whose dimension remains unchanged as the number of agents grows, supporting scalability and transfer to different agent or group sizes.Agents in the same group share actor and critic networks for generalization.
- Multi-Agent Actor-Critic: During centralized training, joint observations and actions support critic learning, whereas execution uses each agent’s local observation to select actions through shared group-specific actors.The critic and actor use HGAT-derived embeddings, with group membership determining the shared networks.
Experiments
The experiments evaluate HAMA in cooperative and mixed cooperative-competitive environments, including transfer to larger agent populations. In cooperative navigation, HAMA converges quickly and achieves lower testing penalties than the compared methods.
- HAMA is evaluated in cooperative navigation and mixed cooperative-competitive environments, including 3 vs. 1 and 3 vs. 3 predator-prey scenarios.
- Cooperative Navigation: HAMA converges quickly to the lowest training penalty in three-agent cooperative navigation.Figure 3 reports penalties averaged every 10,000 steps through 3 million training steps.
- Cooperative Navigation: HAMA achieves a lower testing mean penalty than ATOC, MADDPG, and MAAC in cooperative navigation.Table 1 reports the mean and standard deviation of cooperative-navigation penalties.
- Transfer Learning: Policies trained with three agents transfer to games with 50 and 100 agents, where HAMA obtains lower average penalties and higher landmark occupation percentages.The transfer experiments use agents reduced to one twenty-fifth of their original size, with local observations of three nearest agents and three landmarks.
3 vs. 1 Predator-Prey
The 3 vs. 1 predator-prey game requires predators to cooperate while competing against a faster prey. HAMA achieves higher or similar predator scores against prey policies trained by other methods.
- HAMA predators achieve higher or similar scores against prey trained by MADDPG and MAAC.Each algorithm is self-play trained, then evaluated against policies trained by different models.
3 vs. 3 Predator-Prey
In the 3 vs. 3 predator-prey game, HAMA addresses ambiguous cooperation and competition strategies with hierarchical graph attention and outperforms alternative architectures. Its learned policies also transfer across agent compositions, while attention weights expose strategic decisions.
- The game requires predators to choose between cooperating to chase prey and pursuing prey individually, making the optimal strategy nontrivial.Predators may also choose which prey to chase.
- HAMA achieves the highest scores for both predators and prey against policies trained by all compared algorithms.The paper attributes this performance to better representation of hierarchical relationships in the dynamic game.
- The full hierarchical graph with inter-agent and inter-group attention outperforms the tested architectural variants.Hierarchical-graph variants score higher than the single-graph variant, and the variant using both attentions performs best when attention mechanisms are compared.
- When HAMA policies transfer from 3 vs. 3 to m vs. n games, predator success is close to 1 when m exceeds n.Predator success is defined as the fraction of episodes in which all prey are captured.
- Inter-agent and inter-group attention weights reveal distinct cooperation and competition patterns during predator pursuit.Predators 1 and 3 increase cooperative attention to corner a prey, while predator 2 attends competitively to pursue another prey.
- HAMA also outperforms other models in the more-the stronger game, where clustered prey require an equally large or larger predator group to be captured.The rule creates a setting in which grouped predators and prey have corresponding offensive and defensive strength.
Conclusions
The paper concludes that HAMA combines hierarchical graph-attention state representation with multi-agent actor-critic learning. It reports stronger performance across cooperative and competitive environments, transferable policies across agent compositions, and interpretable learned strategies.
- HAMA combines hierarchical graph-attention state representation with a multi-agent actor-critic model.
- The learned model outperforms other MARL models across a variety of cooperative and competitive multi-agent environments.
- HAMA facilitates policy transfer to new tasks with different agent compositions and supports interpretation of learned strategies.
Supplementary Material
The supplementary material describes HAMA’s training procedure, implementation settings, additional games, strategy visualizations, and learning curves. These materials include algorithmic updates, transfer-related results, and interpretations of attention during predator-prey play.
- HAMA trains actors and critics with replay-buffer samples, target networks, and multi-agent actor-critic updates.The algorithm initializes actor, critic, and target networks, collects transitions, updates critics and actors, and then updates target parameters.
- The experiments use two-layer MLP components with 256 units, Adam learning rates of 1e−4 for actors and 1e−3 for critics, discount factor 0.95, and soft-target coefficient τ = 1e−3.
- In the more-the stronger game, HAMA outperforms other models under clustered-group capture rules.A predator group must be at least as large as a clustered prey group to capture it.
- During an interpreted episode, predators shift attention to uncaptured prey after capturing the chased prey.Already captured prey can interrupt predators and delay capture of remaining prey.
- During training, MADDPG predators score higher than HAMA predators, although the primary results report HAMA outperforming MADDPG.The supplementary discussion attributes this pattern to HAMA learning more intelligent prey that counter intelligent predators.