Source-linked AI summary
Graph Convolutional Reinforcement Learning
Jiechuan Jiang, Chen Dun, Tiejun Huang, Zongqing Lu
TL;DR
Cooperative multi-agent learning must represent interactions despite rapidly changing agents and neighborhoods. DGN models agents as a graph, uses relation-aware graph convolution and temporal regularization, and substantially outperforms existing methods across cooperative scenarios.
Problem
Rapidly changing agents and neighborhoods make it difficult to learn abstract representations of mutual interplay, although cooperation is important in multi-agent environments.
Method
DGN applies graph convolution with relation kernels over gradually increased receptive fields and temporally regularizes relation representations to learn consistent cooperative policies.
Results
DGN substantially outperforms existing methods in jungle, battle, and routing scenarios, with higher and more stable mean reward in the reported battle experiments.
Takeaways & Limitations
Graph convolution, relation kernels, and temporal regularization each improve cooperation, according to the paper’s ablation studies.
Takeaways & Limitations
For temporal relation regularization, the next-state relation representation is produced by the current network rather than the lagging target network.
Abstract
from arXiv · showhide
Learning to cooperate is crucially important in multi-agent environments. The key is to understand the mutual interplay between agents. However, multi-agent environments are highly dynamic, where agents keep moving and their neighbors change quickly. This makes it hard to learn abstract representations of mutual interplay between agents. To tackle these difficulties, we propose graph convolutional reinforcement learning, where graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment, and relation kernels capture the interplay between agents by their relation representations. Latent features produced by convolutional layers from gradually increased receptive fields are exploited to learn cooperation, and cooperation is further improved by temporal relation regularization for consistency. Empirically, we show that our method substantially outperforms existing methods in a variety of cooperative scenarios.
1 INTRODUCTION
The paper frames cooperation in multi-agent environments as important but difficult because dynamic interactions and changing neighborhoods complicate representation learning. It proposes DGN, which uses graph convolution, relation kernels, receptive fields, and temporal regularization to learn cooperation.
- Dynamic multi-agent interactions make it difficult to learn abstract representations of mutual interplay as agents and neighbors change quickly.
- DGN models the multi-agent environment as a graph whose nodes are agents, whose features encode local observations, and whose edges connect neighboring agents.
- Multi-head attention serves as a convolution kernel that extracts relation representations and aggregates neighboring features.
- DGN uses latent features from gradually increased receptive fields and temporally regularized relation representations to learn consistent cooperative policies.
- DGN shares weights across agents, is trained end-to-end on a deep Q network, and substantially outperforms existing methods across jungle, battle, and routing scenarios.
2 RELATED WORK
Related work covers centralized and parameter-sharing MARL methods alongside graph convolutional approaches for extracting locally connected features from arbitrary graphs. Existing cooperation methods face scalability, information-sharing, or object-labeling constraints.
- MADDPG and COMA use centralized critics over all agents’ observations and actions, making them difficult to scale.
- PS-TRPO shares policy parameters to improve cooperation, but cooperation remains limited without sharing information among agents.
- Graph convolutional networks take node feature matrices as input and output node-level feature matrices for arbitrary graphs.
- Some prior cooperation methods require environment objects to be explicitly labeled, which is infeasible in many real-world applications.
3 METHOD
DGN models dynamic multi-agent environments as graphs and uses shared-weight graph convolutional Q networks to learn cooperation from local neighborhoods. Multi-head relation kernels, expanding receptive fields, and temporal regularization capture agent interplay and promote stable cooperative policies.
- Graph construction: DGN represents agents as graph nodes, encodes local observations as node features, and connects each agent to dynamically determined neighbors.Neighborhoods can be determined by distance or other environment-specific metrics and may vary over time.
- DGN architecture: The architecture uses observation encoders, graph convolutional layers, and a Q network, with shared weights across agents and accumulated gradients for training.Encoders use MLPs for low-dimensional observations or CNNs for visual inputs.
- Graph convolution: Stacking convolutional layers enlarges each agent’s receptive field, gathers information from increasingly distant agents, and broadens the scope of cooperation.One layer provides one-hop information, while two layers can incorporate information from two-hop neighborhoods.
- Dynamic graphs: DGN handles changing graph structure by organizing agent features into a matrix and constructing per-agent adjacency matrices that select local receptive fields.The adjacency matrix encodes the central agent and its neighbors, while the Q function uses the selected observations to output that agent’s Q value.
- Relation kernel: Multi-head dot-product attention serves as an order-independent relation kernel that weights neighboring features, concatenates head outputs, and extracts higher-order interaction representations.Additional convolutional layers can capture higher-order relations, while multiple heads provide diverse relation representations.
- Temporal relation regularization: Temporal relation regularization minimizes KL divergence between relation representations across successive states to encourage stable, consistent cooperation in dynamic environments.The next-state relation representation acts as the target, and upper-layer regularization is intended to support long-term consistent action policies.
4 EXPERIMENTS
Experiments evaluate DGN against parameter-sharing baselines in battle, jungle, and routing, showing stronger cooperation, performance, and strategy development across these scenarios.
- 4.1 BATTLE: Battle pits weaker agents against stronger enemies, requiring coordinated tactics and sustained cooperation to defeat them.Agents are evaluated on reward, kills, deaths, and kill-death ratio.
- 4.1 BATTLE: DGN converges to higher and more stable battle rewards, while DQN and CommNet settle on passive policies after initially high rewards.DQN and CommNet initially gather defensively, but uneven rewards from that behavior lead to lower final performance.
- 4.1 BATTLE: DGN learns tactical maneuvers such as encircling and attacking an enemy flank, whereas DQN and CommNet fail to develop comparable cooperation.Relation kernels extract higher-order relations that can be exploited for cooperative behavior.
- 4.1 BATTLE: Removing temporal relation regularization slightly reduces performance and produces less consistent, less synchronous behavior during formation-based cooperation.Without regularization, agents are more easily distracted and abandon intended trajectories, reducing successful encirclements.
- 4.2 JUNGLE: In jungle’s resource-sharing dilemma, DGN achieves higher reward with fewer inter-agent attacks by selecting nearby food and allocating it rationally.Inter-agent attacks are reported as 2× lower than MFQ.
- 4.3 ROUTING: In routing, DGN converges faster to higher mean reward and learns shortest paths while diverting packets when congestion is imminent.Communication baselines outperform DQN but do not develop DGN’s sophisticated strategies.
- 4.3 ROUTING: DGN generalizes from N = 20 to substantially heavier traffic without retraining, outperforming shortest-path routing with bandwidth limits up to N = 140.The reported scaling behavior comes from integrating neighboring agents’ features according to their relations.
5 CONCLUSIONS
The paper proposes graph convolutional reinforcement learning, using dynamic graph structure, relation kernels, expanding receptive fields, and temporal regularization to learn consistent cooperative strategies. DGN significantly outperforms existing methods across cooperative multi-agent scenarios.
- DGN adapts graph convolution to the changing underlying graph of a multi-agent environment.
- Relation kernels extract representations of mutual interplay between agents.
- Stacked convolutional layers expand receptive fields to gather information for cooperative strategy learning.
- Temporal regularization makes relation representations, and therefore cooperation, more consistent over time.
- DGN significantly outperforms existing methods across varied cooperative multi-agent scenarios.
A HYPERPARAMETERS
This appendix summarizes the hyperparameters used by DGN and the experimental baselines.
- Table 4 summarizes the hyperparameters used by DGN and the baselines.
B EXPERIMENTAL SETTINGS
The experimental settings specify reward structures for jungle and battle, plus bandwidth and packet-size conditions for routing.
- Jungle: Jungle rewards movement with 0, food attacks with +1, other-agent attacks with +2, being attacked with −4, and blank-grid attacks with −0.01.
- Battle: Battle rewards attacking an enemy with +5, penalizes being killed by −2, and penalizes blank-grid attacks by −0.01.
- Routing: Routing uses unit link bandwidth and assigns each data packet a random size between 0 and 1.
C ADDITIONAL EXPERIMENTS
Additional experiments examine receptive fields, graph stability, communication, routing traffic, and temporal regularization. Results support larger but bounded receptive fields, dynamic-graph handling, selective communication, and scenario-dependent benefits from temporal consistency.
- Receptive fields: Two convolutional layers produce a more stable learning curve than one layer in battle.
- Receptive fields: Performance improves as each jungle agent’s neighbor count rises from 1 to 3, but drops at 4.
- Graph dynamics: Fixed graphs converge faster than unfixed graphs, supporting graph stability across two successive timesteps.
- Communication: DGN outperforms ATOC and TarMAC in battle; multi-head attention captures relations better than the LSTM kernel, while redundant information can hurt performance.
- Routing: DGN outperforms Floyd with Bandwidth Limit through N = 140 traffic participants, after which Floyd with BL performs better under heavier congestion.
- Temporal regularization: DGN and DGN-R outperform DGN-M; DGN slightly leads DGN-R in routing but performs similarly in jungle.