Source-linked AI summary
The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems
Seongho Son, Stephen Hailes, Mirco Musolesi
TL;DR
The paper addresses how network topology and opponent information affect cooperation, extending beyond imitation-based models of multi-agent adaptation. It uses deep reinforcement learning for agents playing two-player IPD on graphs with action-history and identity information, finding that sparse, longer-path networks support cooperation while opponent identity can hinder it.
Problem
Existing cooperation studies often model adaptation through strategy imitation based on cumulative payoffs, leaving the effects of learned IPD behaviour, graph topology, and opponent information to investigate.
Method
Agents learn two-player IPD with deep reinforcement learning as graph nodes interacting with neighbours, while experiments vary topology, action-history length, partner selection, and binary opponent identity.
Results
Longer average paths and sparser graphs yield more mutual cooperation, whereas opponent identity hinders cooperation; longer action histories alleviate this effect.
Takeaways & Limitations
The number of neighbours, average path length, partner selection, and opponent-state information are central factors shaping cooperation in networked multi-agent IPD.
Takeaways & Limitations
The payoff analysis assumes a benefit-to-cost ratio b : c = 5 : 1 and rescales shifted payoff values for neural-network training.
Abstract
from arXiv · showhide
Several works have investigated the influence of graph topology on cooperation among artificial agents, while the majority of the literature has focused on modelling agents' adaptation through strategy imitation, which relies solely on the cumulative payoffs of others. This paper investigates scenarios in which each agent learns to play the two-player Iterated Prisoner's Dilemma (IPD) using deep reinforcement learning. Each agent is represented as a node in a graph, where its neighbours constitute the pool of opponents with whom it can interact. During each IPD episode, agents are provided with different types of information about their opponent, consisting of action history and opponent identity. Experimental results across different graph topologies show that the number of neighbours per node and the average path length are the main factors affecting the emergence of cooperation. We also show that, while partner selection fosters mutual cooperation by limiting the diversity of the opponent pool, providing agents with the identity of their opponent hinders the proliferation of cooperative strategies.
1 Introduction
The paper examines how graph structure and opponent information shape cooperation when agents learn social dilemmas through reinforcement learning. It studies IPD interactions on networks and finds that topology, partner selection, action history, and opponent identity affect cooperative outcomes.
- Social dilemmas pit individually optimal selfish choices against socially optimal cooperation, motivating study of cooperation among multiple agents.
- Reinforcement learning models agents that adapt behaviour to maximise discounted returns while allowing researchers to vary rewards and environmental information.
- Multi-agent learning violates environmental stationarity and often involves partial observability, creating obstacles to stable and scalable training.
- Prior graph-based cooperation studies largely model adaptation through imitation, with agents copying neighbours’ strategies according to accumulated-payoff differences.
- Opponent information can promote cooperation through reputational action histories but can also expose cooperative agents to exploitation by defectors.
- The paper uses deep reinforcement learning for networked IPD and reports that longer paths and sparser graphs produce more mutual cooperation.
- Opponent identity can hinder cooperation by helping defectors identify fully cooperative agents and avoid retaliatory opponents, while longer action histories alleviate this effect.
2 Iterated Prisoner’s Dilemma on Graphs
Agents are nodes in an undirected graph and interact with neighbouring agents in two-player IPD. The experiments vary graph families, action-history information, and partner-selection inputs, including optional binary opponent identity.
- Game Structure: Each agent is a node in an undirected graph, interacts with a neighbour, and chooses cooperation or defection using Prisoner’s Dilemma payoffs.
- Game Structure: A valid Prisoner’s Dilemma satisfies T > R > P > S and 2R > T + S, distinguishing temptation, reward, punishment, and sucker outcomes.
- Game Structure: The payoff parameterisation uses benefit-to-cost ratio b : c = 5 : 1, shifts values so R + P = T + S = 0, and rescales rewards for training stability.
- Game Structure: Agents receive concatenated one-hot opponent action histories of length l as state input to a dilemma-playing Deep Q-network.
- Synthetic Graphs: The experiments use Erdős-Rényi, Watts-Strogatz, and Barabási-Albert graphs with sampled wiring, rewiring, or preferential-attachment parameters.
- Partner Selection: Partner selection supplies neighbouring agents’ concatenated state vectors to a module that outputs a selection probability for each candidate.
- Partner Selection: Opponent identity is optionally added as binary encoding, increasing the interaction-state dimension to 2×l + ⌈log2 N⌉.
3 Reinforcement Learning Implementation
Agents learn IPD actions with independent Deep Q-learning networks, while partner selection uses separate networks trained with Soft Actor-Critic. The interaction and training procedures differ depending on whether opponents are sampled from graph neighbours or selected using partner-selection modules.
- 3.1 Playing the IPD: Each agent uses a separate neural network for IPD action selection and independently trains on stored transitions.Transitions contain states, actions, rewards, and next states; buffers are discarded after each update to maintain relevance to changing dynamics.
- 3.1 Playing the IPD: The IPD policy selects the highest-Q action with probability 1 − ϵ and otherwise samples uniformly for exploration.The policy is updated by minimising a Q-learning objective using a target network.
- 3.1 Playing the IPD: The IPD networks use target-network stabilisation, with target parameters copied every Z = 16 interaction rounds, exploration rate ϵ = 0.05, and discount rate γ = 0.99.The networks have two hidden layers with 32 and 16 nodes and use Rectified Linear Unit activations.
- 3.1 Playing the IPD: Without partner selection, each agent uniformly chooses an opponent among its graph neighbours, and both selected-opponent and selected-by-neighbour experiences train its IPD policy.Figure 1 summarises this interaction and training loop from agent i’s perspective.
- 3.2 Partner Selection: With partner selection, agents apply separate selection networks to neighbour information before IPD play, producing Q-values or action probabilities for candidate partners.Selection occurs before any population agent plays an IPD episode, using neighbours’ information at t = 1.
- 3.2 Partner Selection: Partner-selection networks are trained with Soft Actor-Critic using an entropy-regularised objective and slowly updated target soft-Q parameters.The entropy coefficient is α = 0.01, and target parameters use τ = 0.05 for gradual updates.
4 Results
Across graph types, sparser graphs and longer average path lengths are associated with more mutual cooperation. In partner-selection experiments, opponent identity mainly suppresses cooperation when action histories are short, while longer histories reduce that difference.
- IPD with Random Matching: Higher graph-generation parameters generally correspond to more mutual defection across ER, WS, and BA graphs.On ER graphs, pER > 0.2 is often associated with complete mutual defection; increasing pWS or m also raises mutual defection.
- IPD with Random Matching: Lower average path length is associated with higher mutual defection across all three graph types.In BA graphs, pmd stays below 0.6 when dpath > 3 but often exceeds 0.8 when dpath < 2.5; in WS graphs, pmd stays below 0.6 when dpath > 3.5 and exceeds 0.9 when dpath < 3.
- IPD with Partner Selection: In partner selection without opponent identity, changing action-history length does not produce a statistically significant difference in final mutual cooperation.This comparison uses the average proportion of mutual cooperation over the last 100 rounds across 20 runs.
- IPD with Partner Selection: With opponent identity, final mutual cooperation drops below 0.3 at l = 1 but increases above 0.5 at l = 5, 10.At longer histories, confidence intervals overlap with experiments without identity information, making the identity effect insignificant as history length increases.
- IPD with Partner Selection: Without opponent identity, mutual cooperation rises rapidly at l = 1, while l = 5 and l = 10 show a multi-stage learning pattern with a later peak.For l = 5 and l = 10, mutual cooperation initially rises, declines amid exploitation and mutual defection, then increases again; the peak occurs around 100,000 rounds.
- IPD with Partner Selection: With opponent identity, mutual cooperation at l = 1 rises initially to 0.4, falls to 0.2, and then continues declining.For l = 5 and l = 10, mutual cooperation instead shows a stable increase after the first quarter of the experiments.
5 Discussion
The discussion links cooperation to graph structure, opponent information, and partner-selection dynamics in IPD. Shorter paths and denser connectivity favor mutual defection, while opponent identity can undermine cooperation despite partner selection.
- Effect of Graph Topology on the Emergence of Cooperation: When agents face tit-for-tat-like retaliators with γ = 0.99, cooperation yields Gc = 20, compared with Gd = −19.5 for always defecting.The discussion uses these return values to explain why agents repeatedly facing retaliators can shift toward cooperation.
- Effect of Graph Topology on the Emergence of Cooperation: In denser graphs and graphs with shorter average path lengths, mutual defection is more likely to spread.The analysis relates this pattern to agents encountering more varied opponents and to graph-structural effects on learning.
- Effect of Opponent Information on the Emergence of Cooperation: Partner selection can favor cooperation by allowing agents with cooperative strategies to choose other cooperative partners.Without opponent identity, selection relies on neighbours’ action histories, restricting the effective opponent pool.
- Effect of Opponent Information on the Emergence of Cooperation: Providing opponent identity makes mutual cooperation less prevalent because defectors can target selected partners while cooperators must adapt to more opponents.The resulting asymmetry gives defectors fewer learning requirements and exposes cooperative agents to exploitation.
- Effect of Opponent Information on the Emergence of Cooperation: Longer action histories reduce interaction inequality and enable more mutual cooperation as partner-selection dynamics proceed.The figure descriptions associate short histories with concentrated selections and longer histories with broader partner participation.
6 Conclusion
The paper analyzes cooperation in multi-agent IPD under different network structures and opponent-information formats. It reports that graph density, path length, action history, partner selection, and opponent identity shape the emergence of cooperation within the studied IPD settings.
- Conclusion: Denser graphs and shorter path lengths make mutual defection more likely to spread.The conclusion attributes this finding to experiments across three graph types.
- Conclusion: Longer action histories slow mutual cooperation by enabling defective agents to target cooperative agents while avoiding retaliators.This conclusion concerns the partner-selection experiments with varying action-history lengths.
- Conclusion: Providing opponent identity lowers mutual cooperation when action-history information is insufficient.The paper states this result as an experimental finding within its studied IPD scenarios.
- Conclusion: The study investigates limited cases of the Iterated Prisoner’s Dilemma.This marks the scope boundary of the reported analysis.