Prosocial learning agents solve generalized Stag Hunts better than selfish ones

1 Introduction

• Independent learners can converge to Nash equilibria with low payoffs even when higher-payoff equilibria exist, and initial conditions can affect long-run payoffs.In the Stag Hunt, sufficiently low payoffs for hunting alone make learners more likely to choose the safe action, which makes partners less likely to hunt later and reinforces the inefficient equilibrium.Source: §1 · p. 2
• The work builds on reward-shaping research but targets settings where only one agent is controlled, actions can be temporally extended, and policies may require function approximation, including learning from raw pixels.Reward shaping changes agents’ updating rules using a potential function; the paper recasts joint-reward optimization as prosocial preferences that value a partner’s rewards.Source: §1 · p. 2
• In 2 × 2 Stag Hunts, making a single agent prosocial can increase convergence to a good equilibrium; the result extends to N-strategy, two-player games whose 2 × 2 sub-games are Stag Hunts.The proposed mechanism is that the Stag Hunt equilibria share a payoff ranking, so one agent learning toward an equilibrium leads others to follow.Source: §1 · pp. 2–3
• The paper focuses on game-theoretic issues and treats MARL convergence problems caused by non-stationarity as important but orthogonal.Standard RL techniques assume a stationary environment, an assumption that can lead to convergence issues in MARL.Source: §1 · p. 2
• The experiments probe whether the analytical insights extend beyond dyads, including games on networks and multiplayer weak-link games.The introduction identifies these as studied extensions of the Stag Hunt, without reporting their experimental outcomes here.Source: §1 · p. 3
• In grid-based Markov games and an Atari game, adding prosociality helped policy gradient converge to good outcomes, improving the prosocial agent’s long-run payoff.The reported outcomes were convergence to either the safe or risky equilibrium.Source: §1 · p. 3

2 Risk Dominance, the Stag Hunt, and Equilibrium Selection

• A generalized Stag Hunt has payoff-dominant mutual hunting and risk-dominant mutual foraging equilibria, with foraging safe and hunting risky.The defining payoff condition is h > c ≥ m > g.Source: §2 · p. 3
• Initial beliefs determine which equilibrium’s basin agents enter, and the critical belief threshold depends on all payoff entries, including off-equilibrium payoffs.Agents hunt above the threshold and forage below it; when beliefs straddle it, the learning rule influences which attractor region they reach.Source: §2 · p. 4
• Prosocial agents weight their own and their partner’s rewards using α: α = 0 is selfish, α = .5 weighs rewards equally, and α = 1 is selfless.Their utility is Ui(ai, aj) = (1 − α)Ri(ai, aj) + αRj(ai, aj).Source: §2 · p. 4
• In generalized Stag Hunts, increasing both players’ prosociality enlarges the basin of attraction of (Hunt, Hunt), and sufficient prosociality by either player makes it the unique interior attractor.The threshold condition is that either agent has α ≥ ᾱ.Source: §2 · p. 4
• In qualifying symmetric N × N games, one player above a prosociality threshold makes the payoff-dominant equilibrium the unique convergence point.The result assumes that the only pure equilibria are symmetric and every pairwise payoff-matrix subset is a generalized Stag Hunt.Source: §2 · p. 4
• Outside games whose strategy payoffs satisfy Stag Hunt structure, increasing prosociality may backfire; α near 1 can produce policies that are self-harming and socially inefficient.The paper focuses its experiments on α ∈ {0, .5}; α ≤ .5 ensures an agent accepts personal payoff losses only when total payoff increases.Source: §2 · p. 5

3.1 Matrix Stag Hunt

• In the matrix Stag Hunt, both risk and social preferences affect convergence, and one prosocial agent is almost as effective as making both prosocial.The experiments varied the hunting-alone payoff and prosociality after training randomly initialized policy-gradient agents.Source: §3.1 · p. 5

3.2 Prosociality With More than 2 Players

• On a star network, making only the center agent prosocial greatly increases payoff-dominant convergence; individual influence is limited in fully connected networks, and even all-prosocial weak-link groups coordinate poorly.In the weak-link experiment, agents’ effort incurs a linear cost and project success depends on the lowest effort.Source: Figure 2 · p. 6, §3.2 · pp. 6–7

3.3 Markov Games with Stag Hunt Properties

• The Markov experiments use grid games and an Atari game with Stag Hunt-like coordination, including Escalation Pong learned from raw pixels.Grid-game policies use a convolutional network on board pixels; the experiments compare selfish, prosocial, and single-agent-prosocial pairings.Source: §3.3 · pp. 7, 9
• Across Markov games, prosocial preferences help policy-gradient dyads coordinate on payoff-dominant strategies, while risk and whether one or both agents are prosocial also matter.The weakest effect occurs in Markov Stag Hunt and Harvest at extremely high risk.Source: Figure 4 · p. 8, §3.3 · p. 9
• The effect of prosociality varies by game: Escalation payoffs change smoothly with risk and prosociality, unlike the always-or-never policies of Harvest and Markov Stag Hunt.Escalation has many joint policies that vary in coordination duration, payoff, and risk.Source: §3.3 · p. 9
• Single-agent prosociality benefits Harvest more than Markov Stag Hunt: random policies can discover mature corn, while stag-chase exploration can teach avoidance after a failed hunt.In Harvest, when one agent waits, the other can readily learn to wait; in Markov Stag Hunt, both agents being prosocial encourages hunting to protect a partner.Source: §3.3 · p. 9
• Equilibrium-selection basins depend on payoffs of non-equilibrium policies as well as equilibria, and the difficulty of discovery may also depend on the RL function approximation.In a stag-runs-away variant, agents do not learn to hunt when exploration rarely brings both to the stag simultaneously.Source: §3.3 · p. 9

• Prosocial preferences in one agent can enlarge the payoff-dominant equilibrium’s basin of attraction in matrix games and extend to complex Markov games with function approximation.Source: §4 · p. 9
• Outside Stag Hunts, prosociality may create equilibria that are suboptimal for the agent, including cases where a partner can exploit it.Source: §4 · p. 9
• When an agent’s actions weakly affect others’ payoffs, prosocial rewards increase observed-reward variance and may slow learning; selfish play is better if the partner is insufficiently adaptive.Source: §4 · p. 10
• The authors identify explicit models of partner learning and combining them with heuristic methods as future directions; productive human-machine coordination also requires accounting for human psychology and learning strategies.Source: §4 · p. 10
• Function approximation, like off-equilibrium payoffs, can affect the basins of attraction, but the paper does not analyze its role in detail.Source: §4 · p. 10

5 Appendix

• In a Stag Hunt, increasing prosociality lowers the minimum probability of the partner hunting needed to prefer Hunt, enlarging its basin of attraction; the payoff-dominant equilibrium remains while a worse one may disappear.Source: §5 · p. 14
• For symmetric N × N games whose two-strategy restrictions are Stag Hunts, prosociality can remove worse equilibria without removing the best equilibrium.Source: §5 · p. 14

5.1 Details of Markov Game Training

• Grid-world policies use convolutional networks and episodic REINFORCE with RMSProp; Markov Stag Hunt and Harvest train for 200,000 episodes, while Escalation trains for 500,000.Source: §5.1 · p. 14
• In Escalation Pong, paddle hits reward both players, while a scored point ends the game and penalizes its scorer by −a × k; the escalation level is shown in the state.Source: §5.1 · pp. 14–15
• Pixel-based policies use A3C with 42 × 42 normalized inputs and frame differences; the convolutional actor-critic has four layers and omits the library’s LSTM.Source: §5.1 · p. 15