CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

• The mechanisms modify base games without restricting players’ actions or adding utility, while enabling cooperation through reciprocity, delegated decisions, or conditional transfers.Repetition uses direct reciprocity; reputation uses indirect reciprocity; mediation enables conditional cooperation; contracts impose action-contingent payments.Source: Figure 1 · p. 1, §1 · pp. 2–3
• CoopEval is the first comparative study of cooperation mechanisms for LLM agents, spanning four mechanism families, four social dilemmas, and six models.The benchmark evaluates repetition, reputation, mediation, and contracts, alongside an unchanged-game baseline and additional coordination experiments.Source: Figure 1 · p. 1, §1 · p. 3
• Theorem 1 establishes that every studied mechanism can implement any Pareto improvement over a base-game Nash equilibrium in subgame perfect equilibrium.Source: §1 · p. 3
• The framework supports both capability evaluation and protocol design by providing mechanism- and game-specific benchmarks for cooperation-oriented reasoning in mixed-motive interactions.It reports cross-play payoffs, replicator-dynamics outcomes, deviation-based rankings, and analyses of decisions and reasoning justifications.Source: §1 · p. 3
• All tested modern LLMs defect in unmodified social dilemmas, regardless of reasoning capability or model size.Source: §1 · p. 3
• Evolutionary optimization significantly increases cooperation and collective welfare under the mechanisms, demonstrating robustness against strong LLM models.The experiments also find substantially different effectiveness across theoretically sound mechanisms in heterogeneous LLM populations.Source: §1 · p. 3

Normal-form Games

• The dilemmas are finite normal-form games in which players simultaneously choose actions or randomized strategies to maximize expected utility.A strategy profile specifies one strategy for each player, with utilities defined over action profiles.Source: §2 · pp. 3–4

Four Social Dilemmas and Their Solutions

• The four dilemmas span different numbers of actions and players, including asymmetric games.They are presented together in Table 1, with historical context in Appendix B.Source: §2 · p. 4
• In Prisoners and PublicGood, defection strictly dominates cooperation, so rational players defect without additional mechanisms or meta-reasoning.Trust instead requires iterated elimination, while Travelers requires repeated elimination leading to a price of $2 under common rationality.Source: §2 · p. 4

Solving General Games

• When dominance does not determine play, Nash and subgame-perfect equilibrium characterize rational best responses, especially after mechanisms convert games into sequential settings.Formal definitions are deferred to Appendix B.Source: §2 · p. 4

Mechanism Families

• The study evaluates four mechanism families: repeated interaction, reputation-based indirect reciprocity, third-party mediation, and outcome-conditional contracts.These mechanisms are widely deployed in classical multiagent systems, although their viability depends on the application domain.Source: §3 · p. 4

Repetition

• Repetition enables direct reciprocity through observed past actions, with continuation determined by an unknown stopping time and probability δ ∈(0, 1).The section cites Axelrod’s tit-for-tat tournament as an example and notes that players must not know when repetition ends for rational cooperation.Source: §3 · p. 4

Reputation

• Reputation supports equilibrium cooperation when players observe sufficient interaction histories and are likely to encounter future partners, enabling punishment of first-order free riders.Higher-order information can additionally help eliminate higher-order free riders, though the literature disagrees about whether it should be included.Source: §3 · p. 4

Mediation

• Mediation lets players delegate decisions to a public third party whose complete action plan is known in advance.The mechanism applies to settings such as centralized routing, where the mediator acts for players who delegate.Source: §3 · pp. 4–5

Contract

• Contracts use zero-sum side payments to reward costly cooperation, expanding feasible final payoffs but making outcomes sensitive to contract design.Some variants exclude welfare-suboptimal subgame-perfect equilibria, whereas unilateral side payments do not achieve Prisoner’s Dilemma cooperation; transfers may also be infeasible when utility is difficult to share.Source: §3 · p. 5

Implementation Designs

• Reputation variants differ in historical scope, while agents propose mediator or contract designs and approval voting selects one winning proposal for play.Repetition and Reputation- expose co-players’ past rounds; Reputation+ extends this to recursively observed partner histories, with random rematching introducing intra-player variance.Source: §3 · p. 5

Theorem 1

• Theorem 1 establishes that a Pareto-superior outcome can be achieved in subgame-perfect equilibrium under Mediation and Contract, and under Repetition and Reputation+ when δ is sufficiently high.For a normal-form game with Nash equilibrium s∗ and profile a satisfying ui(a) > ui(s∗) for every player, the mechanisms sustain payoff u(a).Source: §4 · p. 5

Implication

• In social dilemmas, the theorem applies when a is the profile where every player cooperates, formalizing how these mechanisms overcome the cooperation dilemma.The cooperative profile need not be rational in the base game.Source: §4 · p. 5

Equilibrium multiplicity

• The theorem does not eliminate other bad equilibria: everyone unconditionally defects, and rejects the contract when applicable, remains a subgame-perfect equilibrium.Thus, cooperation is supportable without being the sole equilibrium outcome.Source: §4 · p. 5

Proof strategy

• The proofs unify known folk-theorem ideas through grim-trigger strategies that prescribe a target path and punish deviations by switching to the less-preferred equilibrium s∗.Mediation and Contract additionally require accounting for players’ proposals and votes; analogous results cover Nash equilibrium and finite history depth k variants.Source: §4 · p. 5

Experimental Setup

• Each agent is instructed to maximize its own total points from the mechanism-modified game using standardized prompts across games and mechanisms.Additional experiment settings and exact prompts are provided in Appendices C and N.Source: §5 · pp. 5–6

LLM Models

• Six LLM configurations are evaluated, with chain-of-thought prompting used for all except the non-reasoning Gemini-B model.The models are Claude Sonnet 4.5, GPT 5.2, Gemini 3 Flash with and without reasoning, GPT-4o, and Qwen3-30B-A3B-Instruct-2507.Source: §5 · p. 6

Mechanism Parameters

• Main Repetition and Reputation experiments use k = 3 past rounds and continuation probability δ = 0.8, with parameter ablations discussed separately.The paper states that these settings are sufficient to sustain cooperation in the tested social dilemmas.Source: §5 · p. 6

Sample Size

• Each mechanism–game–model assignment is repeated three times, producing 8586 decisions per LLM model and more than 50,000 decisions overall.Because individual combinations may lack statistical significance, results are interpreted through aggregated experiments and their strong signals.Source: §5 · p. 6

Performance Metrics

• Performance is measured relative to the tested population using Mean cross-play payoff, Fitness under 1000-step replicator dynamics, and deviation ratings for general-sum games.Mean assumes a uniform tested population; Fitness evaluates utility against the final evolved population, while deviation ratings extend ELO-style ranking beyond zero-sum games.Source: §5 · p. 6

Decision Justification Analysis

• A GPT-5.2 judge checks each chain-of-thought for any of 15 predefined action justifications, excluding Gemini-B because it produces no reasoning or explanations.The analysis uses the LLM-as-a-judge framework of Guzman Piedrahita et al. (2025).Source: §5 · p. 6

RQ1. No Mechanism Baseline

• Without mechanisms, modern LLMs consistently defect across social dilemmas, usually 100% of the time, and evolutionary optimization further reduces welfare by eliminating cooperative agents.GPT-4o cooperates about half the time except in PublicGood, where it free-rides about 80%; non-reasoning Gemini-B and Qwen-30B also fail to cooperate.Source: §6 · pp. 6–7

RQ2. Mechanism Effectiveness

• Contracting recovers 80% of the social optimum, whereas Reputation+ raises collective welfare only from 7% to 23%; cooperation remains highly mechanism-dependent.All mechanisms theoretically sustain cooperation in equilibrium, but LLM performance varies sharply; agents’ justifications often invoke individual utility maximization and strategic equilibrium.Source: §6 · p. 7

RQ3. Evolutionary Dynamics

• Evolutionary pressures produce 90%–100% cooperative outcomes under the mechanisms, although population composition can reverse model rankings.For example, Qwen-30B shifts from second-best in a uniform population to second-worst after replicator dynamics; Repetition achieves this without commitments.Source: §6 · p. 8, Figure 3 · p. 8

RQ4. Comparing LLMs and Games

• Gemini-R and Gemini-B perform comparably best, GPT-4o performs worst, and Prisoners is easiest; PublicGood and Stag Hunt expose multi-player and coordination difficulties.Contract does not resolve Stag Hunt cooperation for GPT-4o and Qwen-30B, suggesting complexity risks for less capable models.Source: §6 · p. 8

RQ5. Repetition and Reputation

• Reputation is significantly less effective than Repetition; Reputation- outperforms Reputation+, lower δ or k improves Reputation, and Repetition is insensitive to both parameters.LLMs also show high first-round free-riding and undercutting in PublicGood and Travelers, while Reputation uniquely elicits uncertainty about other players’ intentions at 58%.Source: §6 · pp. 8–9

RQ6. Mediation and Contract

• One well-designed mediator or contract proposal often establishes cooperation, with contract proposals achieving weak dominance in PublicGood 94% and Prisoners 81% of the time.Delegating to mediators is a Nash equilibrium 80–89% of the time in Trust and Prisoners, while proposal approval typically reaches 70%–90%.Source: §6 · p. 9

Future evaluation scope

• Future evaluations should cover sequential social dilemmas and additional mechanisms that may sustain cooperation in equilibrium, including open-source game playing, preplay, and gifting.These directions were beyond the paper’s scope. Mechanisms cited include open-source game playing (Sistla & Kleiman-Weiner, 2025), preplay, and gifting.Source: §7 · p. 9

Agent robustness

• Future work should test cooperation mechanisms with purpose-built LLM agents, including finetuned agents and agents that rely on scaffolds.This would assess the robustness of the mechanisms under more intentionally designed agent capabilities.Source: §7 · p. 9

Research agenda

• The broader research agenda is to characterize rational and robust cooperation in AI agents, with this paper providing groundwork for that investigation.The paper frames this agenda as understanding what rational and robust cooperation may look like in AI agents.Source: §7 · p. 9

Impact Statement

• Cooperation between agents that disregard others’ utilities can produce collusion and harm collective welfare, so the research should be used cautiously.The impact concern applies specifically when mutually beneficial outcomes are not aligned with broader societal welfare.Source: Impact Statement · p. 10

A. Prior Related Work with Modern Agents

• Multi-agent reinforcement learning has studied cooperation through repetition, reputation, mediation, contracts, and side-payments.Representative work spans repetition (Harper et al., 2017; Foerster et al., 2018; Lu et al., 2022), reputation and indirect reciprocity (Anastassacos et al., 2021; McKee et al., 2023; Smit & Santos, 2024), mediation (McAleer et al., 2021), and contracts or side-payments (Hughes et al., 2020; Kram´ar et al., 2022; Willis & Luck, 2023; K¨olle et al., 2023; Haupt et al., 2024).Source: A. Prior Related Work with Modern Agents · p. 17
• Prior LLM studies examined repeated 2 × 2 games including Prisoner’s Dilemma, iterated Prisoner’s Dilemma, reputation assignment under social norms, and donor-game interaction among LLMs.Vallinder & Hughes (2025) tested LLM models against themselves using only last-round information about co-players and higher-order co-players.Source: A. Prior Related Work with Modern Agents · p. 17
• Mediation had not previously been tested with LLMs, while contracting for LLM agents had been explored in Prisoner’s Dilemma, Public Goods, and sequential social dilemmas.Contracting studies include Yocum et al. (2023) and Yan et al. (2024).Source: A. Prior Related Work with Modern Agents · p. 17
• Other LLM research evaluates cooperation in morally contextualized dilemmas and agent dynamics in societal simulations involving the public goods game.These lines include Backmann et al. (2025), Cobben et al. (2026), Piatti et al. (2024), and Faulkner et al. (2026).Source: A. Prior Related Work with Modern Agents · p. 17
• Theoretical cooperation research also considers open-source game playing, preplay, gifting, disarmament, simulation-based cooperation, similarity-based cooperation, and imperfect-recall formulations.Relevant works include Tennenholtz (2004), Sistla & Kleiman-Weiner (2025), Kalai (1981), Lupu & Precup (2020), Deng & Conitzer (2017; 2018), Kovaˇr´ık et al. (2023; 2024; 2025), Oesterheld et al. (2023), Tewolde et al. (2023; 2024; 2025a), and Berker et al. (2025).Source: A. Prior Related Work with Modern Agents · p. 17
• A related bridge between repetition and reputation lets agents decide whether to continue with a partner or seek another one.This direction is studied by Berker & Conitzer (2024) and Fleischmann et al. (2025).Source: A. Prior Related Work with Modern Agents · p. 17

Four Social Dilemmas

• The benchmark focuses on four social dilemmas that cover varying numbers of actions and players, as well as asymmetry across the players.Source: B. Game Theory Background · p. 17
• Prisoner’s is a concise 2-player action game, while Traveler’s is a 2-player k-action race-to-the-bottom game driven by price competition.In Traveler’s Dilemma, the higher-price seller adjusts to pmin, while the lower-price seller earns pmin + 2.Source: B. Game Theory Background · p. 17
• Public Goods is an n-player contribution game whose pool is multiplied by α ∈(1, n) and redistributed evenly, whereas Trust models investment and return-sharing decisions.The benchmark sets n = 3 and α = 1.5; in Trust, P2’s business multiplies total investments by 4 and may share returns equally.Source: B. Game Theory Background · p. 17

Nash Equilibrium, Sequential Games, Subgame Perfect Equilibrium

• Nash equilibrium requires each player’s strategy to be a best response, while subgame perfect equilibrium requires Nash equilibrium from every decision point onward.The studied mechanisms often create sequential games with perfect information, where players choose behavioral strategies specifying randomized actions at their decision points.Source: B. Game Theory Background · p. 18

Mechanism Non-Examples

• Cheap talk, Stackelberg leadership, and correlated strategies are excluded as cooperation mechanisms because defective action remains dominant under each.These mechanisms respectively provide nonbinding communication, observable strategic commitment, or correlated action recommendations.Source: B. Game Theory Background · p. 18

Further Discussions on the Mechanism Designs

• The reputation experiments use decentralized, bottom-up assessments from observed histories, while directly encoding social norms remains future work.The paper contrasts independently assessed histories with variants that encode norms directly into reputation mechanisms.Source: C. Further Details on the Mechanisms… · p. 18
• Removing voting in mediation or contracting would create severe coordination problems among similar proposals, significantly reducing mechanism effectiveness.This concern is linked to prior work (Tewolde et al., 2025b).Source: C. Further Details on the Mechanisms… · p. 18

Handling the Continuation Probability

• Repeated experiments use T = T_δ fixed rounds and δ-weighted payoffs rather than randomized stopping, avoiding high outcome variance while valuing later rounds less.Unweighted averaging is avoided because it diverges from the described game; with T = 15, truncation contributes at most 4.2% of the base-game payoff range in worst-case additive error.Source: C. Further Details on the Mechanisms… · pp. 18–19

Decision Making Format

• LLMs submit action probability distributions that are sampled externally, addressing the cognition–behavior gap associated with randomized decisions.This format replaces direct pure-action submissions in the base game.Source: C. Further Details on the Mechanisms… · p. 19

LLM settings and Prompting

• The protocol uses temperature 1, anonymized action tags, clear non-strategy-specific prompts, persistent mechanism state across evolutionary steps, and reset repetition history between tournaments.Complete implementation details, prompt examples, and parsing logic are provided in Appendix N.Source: C. Further Details on the Mechanisms… · p. 19

Replicator Dynamics

• Evolutionary evaluation uses discrete replicator dynamics with exponential weight updates at learning rate 0.1.The implementation specifies the exponential-update variant rather than an unspecified replicator-dynamics procedure.Source: C. Further Details on the Mechanisms… · p. 19

Deviation Ratings

• Deviation Ratings ranks general-sum-game agents via a most strict coarse correlated equilibrium, preserving dominance and remaining invariant to added clones.Clone-invariance is useful when tested LLMs may behave similarly, such as Gemini-B and Gemini-R.Source: C. Further Details on the Mechanisms… · p. 19

Theorem 1

• A payoff of u(a) is achievable in subgame perfect equilibrium under Mediation and Contract, and under Repetition and Reputation+ when δ is sufficiently high.This applies when a Nash equilibrium s∗ is Pareto-dominated by a, meaning ui(a) > ui(s∗) for every player i.Source: D. Proof of Theorem 1 · p. 19

Proof strategy

• The common proof prescribes play of a and switches permanently to the inferior Nash equilibrium s∗ after any deviation, making deviation unattractive when δ is sufficiently close to 1.The deviation gain is bounded by M, while continued cooperation yields a sufficiently large discounted payoff advantage.Source: D. Proof of Theorem 1 · pp. 19–20

Reputation

• Reputation implements the same threat through good/bad labels, but requires unbounded history depth and Reputation+’s higher-order information to compute current matchup labels accurately.Agents play their part of a when all co-players are good and switch to s∗ when any co-player is bad.Source: D. Proof of Theorem 1 · p. 20

Mediation and Contract

• Mediation sustains cooperation because unanimous delegation to the designated mediator implements a, whereas nondelegation triggers play of each player’s part of s∗.Contracting instead makes cooperation strictly dominant by rewarding players who play their prescribed actions with M-unit transfers from each other player.Source: D. Proof of Theorem 1 · p. 20

Lemma 1

• Lemma 1 extends the result to Nash equilibrium for Reputation-, and for finite-history Repetition, Reputation+, and Reputation- with sufficiently large k and δ ∈(0, 1).The finite-history grim-trigger strategies yield expected payoff u(a), and sufficiently large k with δ sufficiently close to 1 makes deviation an actual utility loss.Source: D. Proof of Theorem 1 · pp. 20–21

E. Individual Game Tables

• Table 3 reports results for the PrisonersDilemma.Source: Table 3 · p. 21
• Table 4 reports results for the PublicGoods game.Source: Table 4 · p. 22
• Table 5 reports results for the TravellersDilemma game.Source: Table 5 · p. 22
• Table 6 reports results for the TrustGame.Source: Table 6 · p. 22
• Table 7 reports results for the StagHunt game.Source: Table 7 · p. 23

F. Evolutionary Dynamics

• Under replicator dynamics, LLM models that initially perform well in a heterogeneous population can be outcompeted, with significantly degrading relative performance.Source: F. Evolutionary Dynamics · p. 24

Evaluation setup

• GPT-5.2 classifies LLM decision reasoning using the Guzman Piedrahita et al. (2025) framework, selecting any number of 15 justification categories; Gemini-B is excluded because it produces no explanations.For Mediation and Contract, classification covers only the chain-of-thought reasoning at the last decision point.Source: G. Reasoning Evaluations with an LLM… · p. 25

Visualization

• Figures 4–7 visualize justification frequencies by mechanism, aggregating across models and games and separately breaking results down by model and game.Figure 4 focuses on the eight most prevalent categories; Figures 5–7 provide mechanism heatmaps at the aggregate, model, and game levels.Source: G. Reasoning Evaluations with an LLM… · p. 25, Figure 4 · p. 26, Figure 5 · p. 27

Justification taxonomy

• The justification categories span individual utility maximization, strategic equilibrium focus, social welfare maximization, inequity aversion, reciprocity, and strategic influence.The framework also includes trust evaluation, competitiveness, uncertainty evaluation, social norm conformity, rule misunderstanding, exploration-exploitation trade-off, risk aversion, strategy legibility, and multidimensional reasoning.Source: G. Reasoning Evaluations with an LLM… · p. 25

H. Ablations on Mechanism Parameters

• Table 8 reports Repetition and Reputation ablations for the Prisoners Dilemma, varying window size k and continuation probability δ.The table presents the mechanism-parameter ablations.Source: Table 8 · p. 30, H. Ablations on Mechanism Parameters · p. 30

I. Action Frequencies

• Figures 8–9 summarize average action probabilities across mechanisms overall and broken down by LLM model.These figures provide pooled and model-specific action-frequency views.Source: Figure 8 · p. 31, Figure 9 · p. 32

J. Action Frequencies Conditioned On Previous Actions of Co-players in Repetition and Reputation

• Figures 10–13 condition action frequencies on the co-player’s previous-round action across Repetition and Reputation in four social dilemmas.The figures cover Prisoners Dilemma, Public Goods, Travellers Dilemma, and Trust Game.Source: Figure 10 · p. 33, Figure 11 · p. 34, Figure 12 · p. 35

K. Statistics about Voting and Adoption in Mediation and Contracting

• Figures 14–17 report voting and adoption statistics for Contracting and Mediation in Prisoners Dilemma, Public Goods, Travellers Dilemma, and Trust Game.Each figure covers one of the four social dilemmas.Source: Figure 14 · p. 37, Figure 15 · p. 37, Figure 16 · p. 37

L. Quality of Proposed Mediators and Contracts

• Figure 18 evaluates when proposed mediator and contract designs make cooperative outcomes game-theoretically stable, using Nash-equilibrium or weak-dominance tests.For Travelers and Trust, no mediator design achieves the cooperative outcome in weakly dominant strategies because such a design is theoretically impossible.Source: Figure 18 · p. 38

M. Match-Up Payoff Figures

• Figures 19–24 display payoff vectors in metagames where each player selects an LLM model, with cell colors indicating Player 1’s payoff relative to base-game Nash-equilibrium and cooperative-action-profile payoffs.Light red represents the payoff Player 1 would receive under the base game’s Nash equilibrium, while light green represents the payoff under its cooperative action profile.Source: Figure 19 · p. 39, Figure 20 · p. 40, Figure 21 · p. 41

N.1. Instruction Prompts

• The instruction prompts require probability distributions over actions, valid JSON output with exact action-name keys, integer percentage values, and values summing to 100.Separate prompts also request step-by-step reasoning or direct output without additional explanation.Source: §N.1 · p. 59

N.2. Game Prompts

• The game prompts define simultaneous probability choices, sampled final actions, player objectives, payoff rules, and player positions for Prisoners Dilemma, Public Goods, Travellers Dilemma, and Trust Game.The prompts specify two actions for Prisoners Dilemma, Public Goods, and Trust Game, and four numbered actions for Travellers Dilemma.Source: §N.2 · pp. 59–60

N.3. Mechanism Prompts

• Mechanism prompts operationalize reputation, mediation, and contracting through repeated-game histories, delegated actions, approval voting, and outcome-conditional payments.Reputation exposes prior play and varying opponents; mediators act for delegating players based on delegation counts; contracts assign action-specific payments and activate only with unanimous acceptance.Source: §N.3 · pp. 61–63

N.4. LLM Judge Prompts

• The LLM judge prompt classifies decisions across 15 strategies, spanning utility, equilibrium, welfare, reciprocity, trust, uncertainty, norms, risk, and multidimensional reasoning.The taxonomy also includes inequity aversion, competitiveness, rule misunderstanding, exploration-exploitation trade-offs, and strategy legibility.Source: §N.4 · p. 65
• The expected-payoff derivation gives E = 2q + 1 - p, so Player 1 maximizes payoff by setting p = 0 and choosing A1 with 100% probability.The prompt explicitly maps this to 0% probability for A0 and 100% for A1.Source: §N.4 · p. 66
• A1 is the dominant strategy because it yields higher payoffs than A0 for Player 1 regardless of Player 2’s choice.The prompt contrasts 3 vs 2 when Player 2 chooses A0 and 1 vs 0 when Player 2 chooses A1, despite higher mutual-cooperation payoffs.Source: §N.4 · p. 66
• The judge must output valid JSON containing the reasoning, confidence, and justification type, with no explanatory text outside the object.The instructions require confidence as a decimal between 0 and 1 and multiple justification types as a comma-separated string.Source: §N.4 · p. 66