Source-linked AI summary
Multi-agent Reinforcement Learning in Sequential Social Dilemmas
Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, Thore Graepel
TL;DR
Matrix games treat cooperation and defection as atomic actions, whereas real-world social dilemmas require temporally extended policies under partial information. The paper defines sequential social dilemmas, analyzes Gathering and Wolfpack with independent deep-Q-learning agents, and finds that sequential structure differentiates games that share Prisoner’s Dilemma payoff matrices. These results show that environmental factors can have opposite effects on cooperation across the two domains.
Problem
Matrix-game social dilemmas omit critical sequential features of real-world interaction, including temporally extended policy implementation.
Method
The paper defines SSDs as Markov games whose policy-induced empirical payoff matrices satisfy the social-dilemma inequalities, then analyzes Gathering and Wolfpack with multiagent reinforcement learning.
Results
Gathering and Wolfpack both have Prisoner’s Dilemma empirical payoff matrices, yet sequential analysis yields different and sometimes opposite predictions about cooperation.
Takeaways & Limitations
Sequential structure makes the choice between Gathering-like and Wolfpack-like models critical because their cooperation dynamics differ despite matrix-level equivalence.
Takeaways & Limitations
The model addresses only the part of cooperative behavior that arises by habit rather than conscious deliberation.
Abstract
from arXiv · showhide
Matrix games like Prisoner's Dilemma have guided research on social dilemmas for decades. However, they necessarily treat the choice to cooperate or defect as an atomic action. In real-world social dilemmas these choices are temporally extended. Cooperativeness is a property that applies to policies, not elementary actions. We introduce sequential social dilemmas that share the mixed incentive structure of matrix game social dilemmas but also require agents to learn policies that implement their strategic intentions. We analyze the dynamics of policies learned by multiple self-interested independent learning agents, each using its own deep Q-network, on two Markov games we introduce here: 1. a fruit Gathering game and 2. a Wolfpack hunting game. We characterize how learned behavior in each domain changes as a function of environmental factors including resource abundance. Our experiments show how conflict can emerge from competition over shared resources and shed light on how the sequential nature of real world social dilemmas affects cooperation.
1. INTRODUCTION
Matrix-game social dilemmas capture mixed incentives but omit temporal extension, policy-level cooperation, partial information, and quasi-simultaneous decisions. The SSD model preserves the mixed-motivation structure while revealing that sequential Gathering and Wolfpack games can yield different cooperation dynamics despite equivalent Prisoner’s Dilemma payoff matrices.
- Motivation: Matrix-game social dilemmas model cooperation and defection as atomic actions, limiting their representation of temporally extended real-world decisions.Real-world decisions may also involve partial information and quasi-simultaneous interaction.
- Motivation: Cooperation and defection in sequential dilemmas are properties of policies implementing strategic decisions, and cooperativeness may be graded.This shifts analysis from elementary actions to learned policy behavior.
- Contribution: The SSD model preserves the mixed incentive structure of matrix-game social dilemmas while applying analogous inequalities to temporally extended Markov games.The model is designed to capture sequential structure without discarding the social-dilemma payoff conditions.
- Results: Both Gathering and Wolfpack map to Prisoner’s Dilemma payoff matrices, yet their sequential formulations are distinct and can produce opposite predictions about cooperation.Strictly using the matrix-game paradigm would make the two situations appear formally similar.
- Results: Environmental factors can promote cooperation in Gathering while discouraging it in Wolfpack, and vice versa.The paper therefore treats model choice as critical for representing real-world social dilemmas.
- Method: Deep Q-networks are used to find equilibria in SSDs because their sequential structure makes standard matrix-game evolution and learning approaches computationally inadequate.The paper describes multiagent reinforcement learning as the methodological route for this problem.
2. DEFINITIONS AND NOTATION
The paper models sequential social dilemmas as partially observable Markov games in which agents learn policies whose induced payoffs satisfy matrix-game social-dilemma conditions. This framework treats cooperation and defection as policy-level properties rather than atomic actions.
- Sequential social dilemmas are general-sum Markov games with partial observations, requiring agents to learn policies while coexisting with one another.
- A partially observable Markov game specifies states, observations, actions, transitions, player rewards, and policies mapping observations to action distributions.
- Matrix games are one-state, perfectly observable Markov games whose atomic actions are cooperate and defect.
- The empirical payoff outcomes R, P, S, and T are induced by cooperative and defecting policies through long-term expected payoffs.
- A Markov game is a sequential social dilemma when some induced empirical payoff matrix satisfies the social-dilemma inequalities.
- Cooperative and defecting policy sets may not cover all legal policies because cooperativeness is treated as a graded property thresholded by a continuous social-behavior metric.
3. LEARNING ALGORITHMS
The paper takes a descriptive approach to multiagent learning, studying the social effects and learning dynamics produced by independent agents using deep reinforcement learning. Each agent learns from its own partial observations while treating the other learner as part of a changing environment.
- Unlike prescriptive work that asks what agents should do, the paper studies what social effects emerge from a particular learning rule.
- Deep Q-networks are used to study equilibria and learning dynamics in sequential social dilemmas, whose complexity makes standard matrix-game approaches inapplicable.
- Each agent represents Q_i: O_i × A_i → R with a deep Q-network, using its own observation rather than the true state directly.
- Agents select greedy actions with probability 1 − ϵ and uniformly random actions with probability ϵ to encourage exploration.
- Each agent updates its network from stored experienced transitions using a continually refreshed batch, allowing adaptation to learning-induced changes in experience.
- The independence assumption makes each agent treat the other learner as part of a non-stationary environment, without recursive reasoning about the other's learning.
4. SIMULATION METHODS
Both games are implemented as two-dimensional grid-world environments in which agents act from local visual observations and choose among movement, rotation, beam, and waiting actions.
- The simulations use a 2D grid-world whose state and joint action determine the next state.
- Each agent receives a local RGB observation with dimensions R3×16×21, determined by its position and orientation.
- The observation window extends 15 grid squares forward and 10 squares laterally.
- Agents have eight centered actions: forward, backward, left, right, rotate left, rotate right, use beam, and stand still.
5. RESULTS
Across Gathering and Wolfpack, learned social behavior changed systematically with environmental and agent parameters. Scarcity and costly conflict increased aggression in Gathering, while group benefits and capture radius increased cooperation in Wolfpack; network capacity had opposite effects across the games.
- 5.1 Experiment 1: Gathering: Low abundance or high conflict-cost produced highly aggressive Gathering policies, whereas high abundance or low conflict-cost produced less aggressive policies.Abundance was controlled by apple respawn time Napple, and conflict-cost by the duration Ntagged that a tagged agent was removed.
- 5.1 Experiment 1: Gathering: Gathering policies classified as cooperative came from high-abundance, low-conflict-cost training, while defecting policies came from low-abundance, high-conflict-cost training for empirical payoff analysis.The resulting policy sets were used to estimate empirical payoff matrices by repeatedly playing policy pairs.
- 5.2 Experiment 2: Wolfpack: Greater group capture benefit and larger capture radius increased the average number of wolves per Wolfpack capture, indicating more cooperation.Two cooperative policies emerged: wolves could hunt together after finding one another, or one wolf could wait near prey for the other to arrive.
- 5.2 Experiment 2: Wolfpack: Wolfpack empirical payoff matrices contained all three classic matrix-game social dilemmas: Chicken, Stag Hunt, and Prisoner’s Dilemma.The analysis used policies learned in high-radius/group-bonus environments as cooperators and low-radius/group-bonus environments as defectors.
- 5.3 Experiment 3: Agent parameters influencing the emergence of defection: Greater discount parameters increased defection in both games, but very high Gathering abundance eliminated the motivation to defect regardless of discount rate.In Gathering, tagging provides delayed benefits through increased opportunity to collect apples without interference.
- 5.3 Experiment 3: Agent parameters influencing the emergence of defection: Increasing network size increased defection in Gathering but reduced defection in Wolfpack.Gathering defection requires targeting the other agent with a beam, whereas Wolfpack cooperation requires coordinating hunting behavior.
6. DISCUSSION
Sequential social dilemmas can make cooperation and defection differ in learning difficulty and coordination demands, even when their embedded matrix games look similar. The Gathering and Wolfpack models therefore support distinct predictions about default behavior and how environmental or agent factors affect defection.
- 6. DISCUSSION: Wolfpack’s defecting lone-wolf policy is easier to learn, whereas Gathering’s cooperative policy is easier to learn.Wolfpack cooperation requires partner-conditioned actions and greater coordination; Gathering defection requires coordination to aim at the rival.
- 6. DISCUSSION: Gathering and Wolfpack show opposite network-size effects on defection: larger networks increase defection in Gathering but decrease it in Wolfpack.This contrast is consistent with the games assigning different coordination demands to cooperation and defection.
- 6. DISCUSSION: Gathering and Wolfpack contain embedded Prisoner’s Dilemma-type payoffs but make different predictions as sequential social dilemmas.Gathering predicts cooperation as the easier-to-learn default, while Wolfpack predicts defection as the easier-to-learn default.
- 6. DISCUSSION: Sequential models capture temporally extended cooperation and defection, including coordination that may be required by either strategic choice.The relevant coordination burden depends on the situation: cooperation can require making way for a partner, while defection can require blocking a rival.
- 6. DISCUSSION: Discounting and batch size affect defection in the same direction across Gathering and Wolfpack, whereas network size has the opposite effect.Figure 7 varies temporal discount, batch size, and network size while tracking aggressiveness in Gathering and lone-wolf behavior in Wolfpack.
- 6. DISCUSSION: The complexity of learning cooperation and defection may be unequal, and multiple implementations of either policy can require different amounts of coordination.These implementation differences cannot be represented solely by changing outcome values in a matrix-game model.