Source-linked AI summary
Inequity aversion improves cooperation in intertemporal social dilemmas
Edward Hughes, Joel Z. Leibo, Matthew G. Phillips, Karl Tuyls, Edgar A. Duéñez-Guzmán, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin R. McKee, Raphael Koster, Heather Roff, Thore Graepel
TL;DR
Intertemporal social dilemmas remain difficult for models limited to stateless matrix games and for multi-agent reinforcement learning agents in temporally extended settings. The paper generalizes inequity aversion to Markov games and finds that it promotes cooperation through mechanisms that improve policy learning and temporal credit assignment. The results differ across public-goods and commons dilemmas and carry stated limitations concerning exploitability, stochasticity, and population heterogeneity.
Problem
Existing behavioral models are limited to matrix games, while robust human-like cooperation in temporally extended multi-agent social dilemmas remains unresolved.
Method
The paper extends inequity aversion to partially observable Markov games, using direct and indirect inequity-related rewards within reinforcement-learning agents.
Results
Advantageous inequity aversion is especially effective in Cleanup, whereas disadvantageous inequity aversion promotes cooperation in Harvest through punishment and temporally aligned feedback.
Takeaways & Limitations
Inequity aversion can help explain cooperation in intertemporal social dilemmas by bridging short-term incentives and long-term collective outcomes.
Takeaways & Limitations
The method has exploitable guilty agents, relies on outcomes rather than predictions in stochastic environments, and requires appropriately setting population heterogeneity.
Abstract
from arXiv · showhide
Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.
1 Introduction
Intertemporal social dilemmas pit short-term individual incentives against long-term collective interests, yet humans often cooperate despite classical models predicting otherwise. The paper proposes extending inequity aversion to Markov games to address collective action and temporal credit assignment.
- Classical rational-choice models predict cooperation is impossible in these situations, despite fieldwork and laboratory experiments documenting human cooperation.
- Behavioral game-theory models explain cooperation mainly in matrix games, limiting their applicability to temporally extended environments.
- Intertemporal dilemmas require groups to coordinate policies while associating short-term defection with long-term collective harm.
- Inequity-averse agents balance selfish rewards against reward differences and can resolve social dilemmas by resisting defection and punishing defectors.
- The paper generalizes inequity aversion to Markov games, distinguishing disadvantageous and advantageous inequity aversion as mechanisms addressing the two challenges.
- Inequity aversion improves temporal credit assignment by giving short-term negative feedback to defectors and motivating cooperators to punish them.
2 Reinforcement learning in sequential social dilemmas
The paper formulates sequential social dilemmas as partially observable Markov games in which independently learning agents face collective-action and temporal tensions. It defines these dilemmas using policy sets and Schelling-diagram payoff relationships, then instantiates public-goods and commons environments.
- Partially observable Markov games: Agents independently learn behavior policies from partial observations and individual extrinsic rewards while interacting in a stochastic multi-agent environment.
- Reinforcement learning: A3C uses actor and critic estimates, policy gradients, and asynchronously generated gradients from 24 independent environment copies.
- Intertemporal social dilemmas: An intertemporal social dilemma has short-term individually optimal strategies that produce poor long-term group outcomes.
- Intertemporal social dilemmas: Sequential social dilemmas are defined by policy profiles satisfying mutual-cooperation, non-exploitation, and fear or greed payoff properties.
- Dilemma types: Public-goods dilemmas require costly provision of a shared resource, whereas commons dilemmas involve benefits that deplete a shared resource.
- Examples: Cleanup models a public good where agents clean waste to restore apple production, creating a free-riding tension between private harvesting and group provision.
- Examples: Harvest models a commons where rapid individual harvesting can permanently deplete resources, while group welfare favors restraint.
3 The model
The model extends matrix-game inequity aversion to sequential Markov games by incorporating temporally smoothed reward traces into agents’ subjective rewards. This enables inequity-related intrinsic rewards to operate when players’ rewards occur at different times.
- The paper extends an inequity-aversion model that was directly applicable only to stateless games to sequential multi-state problems using deep reinforcement learning.
- Disadvantageous inequity aversion penalizes receiving less than others, while advantageous inequity aversion penalizes outperforming others.
- The Markov-game extension introduces per-player temporal smoothing because different players’ rewards may occur on different timesteps.
- Each player’s reward is defined from its state and action, and the model constructs a subjective reward from these temporally smoothed reward traces.
- The smoothed reward traces are updated each timestep using the discount factor γ and hyperparameter λ, analogously to eligibility traces.
4 Results
The results show that inequity aversion can sustain cooperation in intertemporal social dilemmas, with advantageous and disadvantageous forms working through different mechanisms and task-specific strengths.
- Advantageous inequity aversion resolves certain intertemporal dilemmas without punishment by supplying a temporally correct intrinsic reward.
- Advantageous-inequity-averse agents outperform A3C at maintaining cooperation in both public goods and commons games, especially in Cleanup.
- Groups of 5 advantageous-inequity-averse agents can sustain a Cleanup strategy where 2 agents consistently remove large amounts of waste, yielding a large collective return.
- Disadvantageous-inequity-averse agents outperform A3C in commons games by punishing defectors, and a single such agent can fine defectors to generate sustainability.
5 Discussion
The discussion explains why each inequity-aversion type fits different tasks and identifies efficiency, population composition, exploitability, and stochasticity as important boundaries.
- In Cleanup, advantageous inequity aversion directly encourages public-good contribution, whereas punishment does not directly increase waste-cleaning attractiveness.
- In Harvest, advantageous inequity aversion provides a noisy sustainability signal, while punishment shapes learning by discouraging overconsumption at the correct time and place.
- Disadvantageous inequity aversion can sustain Harvest cooperation inefficiently because large amounts of collective resources are lost to fines.
- The approach remains limited because guilty agents are exploitable, outcome-based policies struggle under high stochasticity, and population heterogeneity requires careful tuning.
A.1 Motivating research on emergent cooperation
The paper motivates emergent cooperation research through individual-level inductive biases informed by human behavioral science, while recognizing that inequity aversion is task-conditional and sometimes exploitable.
- The research program seeks to identify individual-level inductive biases that promote group-level cooperation in humans and build human-like agents for complex multi-agent tasks.
- Maximizing joint utility is an unsuitable baseline because it assumes away emergent individual altruism, suffers from spurious rewards, and may require unavailable collective-reward information.
- The paper recasts Fehr and Schmidt’s inequity-aversion model as an intrinsic reinforcement-learning reward to evaluate its strengths and weaknesses in social dilemmas.
- Inequity aversion succeeds only conditionally across tasks, and the resulting policies can be exploitable, motivating extensions such as evolved intrinsic rewards.
A.2 Illustrative Schelling diagrams for 2-player matrix games and SSDs
Figure 1 presents Schelling diagrams and payoff matrices for canonical two-player matrix games, using dominant-strategy social pressure to identify pure-strategy Nash equilibria.
- Figure 1 pairs Schelling diagrams with payoff matrices for Chicken, Stag Hunt, and Prisoner’s Dilemma.
- The diagrams identify equilibrium pressure by whether the dominant strategy is defection, which decreases cooperation, or cooperation, which increases it.
- For Chicken, the pure-strategy Nash equilibria are (c, d) and (d, c).
A.3 Parameters for Cleanup and Harvest games
Cleanup and Harvest are defined by distinct resource dynamics and action constraints that make cooperation necessary for sustained rewards.
- Environment parameters: All agents can fine others, while Cleanup additionally provides a cleaning beam and both games reward apple collection with 1.Fining costs the user −1 reward and imposes −50 reward on the fined individual.
- Cleanup: Cleanup produces waste probabilistically until river saturation, with apple spawning increasing with waste saturation.The river saturates when waste covers 40%, and apples spawn with probability 0.125x for saturation x.
- Harvest: Harvest apple spawning depends on nearby apple density, creating linked regions that sustainable policies must harvest selectively.Spawn probabilities are 0, 0.005, 0.02, and 0.05 for 0, 1, 2, and at least 3 nearby apples, respectively.
A.4 Social outcome metrics
The paper uses multiple social outcome metrics because mixed-incentive multi-agent systems lack a single scalar performance measure.
- Metric rationale: Multi-agent systems with mixed incentives require several social outcome metrics because no scalar metric adequately tracks system state.The metrics summarize group behavior and facilitate analysis.
- Core metrics: The Utilitarian metric U measures collective return as the average across players of their total rewards.It captures the sum total of rewards obtained by all agents.
- Core metrics: The Equality metric E is defined using the Gini coefficient, while Sustainability S measures the average time at which rewards are collected.These metrics capture reward distribution and temporal concentration, respectively.
- Cleanup metric: For Cleanup, public-good contribution P is the number of waste cells cleaned.This is also described as a measure of total contribution to the public good.
A.5 Dictate apples, Give apples and Take apples games
Three two-player gridworlds reproduce behavioral-economics paradigms, while theoretical payoff analyses explain how inequity aversion can make cooperation learnable in intertemporal dilemmas.
- Behavioral paradigms: Three gridworld games model Dictate apples, Give apples, and Take apples paradigms using button-mediated interaction between isolated players.The games vary how players transfer, give, or take apples across separate rooms.
- Behavioral results: A selfish A3C baseline never pressed the buttons, whereas advantageous- and disadvantageous-inequity-averse agents pressed them in the predicted games.Advantageous inequity aversion increased pressing in Give and Dictate, while disadvantageous inequity aversion increased pressing in Take.
- Theoretical setup: In intertemporal dilemmas, defection dominates cooperation in the short term, creating a temporal credit-assignment challenge.The analysis denotes defector payoff by D, cooperator payoff by C, and average population payoff by R̄, with d > c.
- Advantageous inequity aversion: Advantageous inequity aversion lowers defection’s effective payoff so cooperation can dominate in some short-term states and become self-sustaining.The transformed defector payoff is D̃ = d − α(d − c), and this mechanism explains success in Cleanup when early behavior generates multiple cooperators.
- Disadvantageous inequity aversion: Disadvantageous inequity aversion can shift the Nash equilibrium toward cooperation by motivating cooperators to fine defectors, but requires policing to emerge first.This mechanism is possible in Harvest, where early behavior is defection, but not in Cleanup, where early behavior is cooperation.