CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

Emanuel Tewolde; Xiao Zhang; David Guzman Piedrahita; Vincent Conitzer; Zhijing Jin

Source-linked AI summary

CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita, Vincent Conitzer, Zhijing Jin

arXiv:2604.15267v2Apr 16, 2026cs.GTcs.AIcs.CLcs.CYcs.MA

TL;DR

LLM agents must cooperate safely despite misaligned incentives, but current models often defect in single-shot social dilemmas and benchmark coverage remains limited. This paper comparatively evaluates game-theoretic cooperation mechanisms for rational LLM agents and finds that mechanism effectiveness varies substantially, while evolutionary pressures increase cooperation under those mechanisms.

Problem
Advanced multiagent systems create safety risks when agents must cooperate despite misaligned incentives, while benchmarks of cooperation in multiagent systems remain limited.
Method
The paper introduces a benchmark suite that evaluates four cooperation-mechanism families across four social dilemmas and six heterogeneous LLM models, alongside a game-theoretic equilibrium analysis.
Results
Different theoretically sound mechanisms achieve vastly different levels of cooperation, and evolutionary optimization pressures significantly increase cooperation and collective welfare under these mechanisms.
Takeaways & Limitations
Cooperation mechanisms can support robust cooperative outcomes among rational LLM agents, including under optimization pressures that otherwise favor self-interested strategic play.
Takeaways & Limitations
The evaluation does not cover sequential social dilemmas, several other candidate mechanisms, or purpose-built agents using finetuning or scaffolds.

Abstract

from arXiv · show

It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave _less_ cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings. Indeed, our experiments show that recent models -- with or without reasoning enabled -- consistently defect in single-shot social dilemmas. To tackle this safety concern, we present the first comparative study of game-theoretic mechanisms designed to enable cooperative outcomes between rational agents _in equilibrium_. Across four social dilemmas testing distinct components of robust cooperation, we evaluate four families of mechanisms: (1) repeating the game for many rounds, (2) reputation systems, (3) third-party mediators to delegate decision making to, and (4) contract agreements for outcome-conditional payments between players. Among our findings, we establish that contracting and mediation are most effective in achieving cooperative outcomes between capable LLM models, and that repetition-induced cooperation deteriorates drastically when co-players vary. Moreover, we demonstrate that the mechanisms become _more effective_ under evolutionary pressures to maximize individual payoffs.

1. Introduction

CoopEval addresses the safety challenge of cooperation among self-interested LLM agents by comparing game-theoretic mechanisms across social dilemmas, models, and heterogeneous populations. It finds that unmodified models defect, while cooperation mechanisms can support cooperative outcomes and respond to evolutionary pressures.

Motivation: Advanced multiagent systems create safety risks because agents may fail to cooperate when incentives are misaligned.
Motivation: Recent reasoning and non-reasoning LLMs use less cooperative strategies such as free-riding and strategic egoism.
Approach: The paper provides the first comparative study of cooperation mechanisms using general LLM agents that can act in natural-language environments.
Approach: The benchmark compares four mechanism families across four social dilemmas and six LLM models using a factorized mechanisms-by-games design.
Findings: All tested modern LLMs defect in unmodified social dilemmas, regardless of reasoning capability or model size.
Findings: Cooperation mechanisms differ substantially in effectiveness, while evolutionary optimization increases cooperation frequency and collective welfare under these mechanisms.

2. Social Dilemmas and Solution Concepts

The paper models its social dilemmas as finite normal-form games in which players choose actions simultaneously and receive utilities determined by the resulting action profile. Dominance and equilibrium concepts identify rational outcomes, often exposing tension between individual payoffs and collective welfare.

Normal-form Games: A finite normal-form game has a finite player set, action sets, simultaneous choices, action profiles, and utility functions over outcomes.
Normal-form Games: Players may randomize over actions, choosing strategies that maximize expected utility.
Social Dilemmas: The four social dilemmas vary in numbers of actions and players and in whether player roles are asymmetric.
Solution Concepts: Strict dominance eliminates actions that yield lower utility than another action against every possible opponent action profile.
Solution Concepts: In Prisoner’s Dilemma and PublicGood, defection strictly dominates cooperation, so rational players defect without additional mechanisms or meta-reasoning.
Solution Concepts: When dominance does not determine behavior, Nash and subgame perfect equilibrium provide canonical solution concepts for the sequential games created by mechanisms.

3. Cooperation Mechanisms

The paper studies four cooperation mechanisms that alter the interaction structure without creating new utility: repetition, reputation, mediation, and contracts. They support reciprocity, delegated coordination, or conditional transfers under specified implementation designs.

Overview: The four mechanism families are Repetition, Reputation, Mediation, and Contract, each grounded in game theory and intended to support cooperation.
Repetition: Repetition enables direct reciprocity by repeatedly matching the same players, exposing past actions and using an uncertain continuation probability.
Reputation: Reputation enables indirect reciprocity by rematching players and revealing sufficient histories of past interactions to support punishment of free riders.
Mediation: Mediation lets players delegate decisions to a public third party whose known plan conditions actions on which players delegate.
Contract: Contracts implement zero-sum side payments that condition one player’s transfer on another player taking a costly cooperative action.
Implementation Designs: Reputation variants differ in historical information, while random rematching introduces unavoidable variance in player performance.

4. A Unifying Theorem of Cooperation

Theorem 1 unifies the mechanisms by showing that a Pareto-superior outcome can be achieved in subgame perfect equilibrium after modifying a normal-form game. The result supports cooperative equilibria but does not eliminate equilibria involving universal defection.

Theorem 1: Theorem 1 considers a normal-form game, a Nash equilibrium, and an action profile that strictly Pareto-dominates that equilibrium.
Theorem 1: The Pareto-superior payoff can be achieved in subgame perfect equilibrium under Mediation, Contract, and sufficiently patient Repetition or Reputation+.
Interpretation: For the studied social dilemmas, the theorem applies when every player chooses the cooperative action, formalizing how the mechanisms can overcome the cooperation dilemma.
Caveat: The theorem does not rule out other bad equilibria, including universal unconditional defection and contract rejection.
Proof Strategy: The proofs unify known mechanism-specific ideas through grim-trigger strategies that switch from a prescribed cooperative path to a less-preferred equilibrium after deviation.

5. Experimental Setup and Evaluation

The evaluation standardizes mechanism-modified social-dilemma experiments across six LLMs, four mechanism families, and multiple complementary performance and reasoning analyses. Agents are instructed to maximize their own mechanism-modified payoffs, with repeated-game parameters and evolutionary dynamics used to assess cooperation and adaptation.

Experimental setup: Agents are instructed to maximize their own total points from the mechanism-modified game.This follows the paper’s game-theoretic assumption that each agent’s utility captures everything it cares about.
Experimental setup: The study evaluates six LLM models across standardized mechanism-modified social-dilemma experiments.The tested models include Claude Sonnet 4.5, GPT-5.2, Gemini 3 Flash with and without reasoning, GPT-4o, and Qwen3-30B.
Mechanism parameters: Repetition and Reputation use a three-round information window and continuation probability δ = 0.8 in the main experiments.The paper also reports ablations over both k and δ.
Sampling: Each Mechanism × Game × LLM-player combination is repeated three times, totaling 8,586 decisions per model and more than 50,000 overall.The authors emphasize aggregated experiments rather than statistical significance for each individual combination.
Evaluation metrics: Performance is assessed with Mean payoff, evolutionary Fitness, and deviation ratings across cross-play populations.Mean assumes a uniform tested population, while Fitness evaluates payoffs after evolutionary game dynamics; deviation ratings provide rankings for general-sum games.
Reasoning analysis: The study also analyzes chain-of-thought decision justifications using an LLM judge that checks 15 predefined justification categories.The judge is powered by GPT-5.2, and Gemini-B chain-of-thought analysis is excluded.

6. Experimental Results and Findings

The results show that modern LLMs defect consistently without mechanisms, while mechanism effectiveness varies substantially across heterogeneous populations and games. Cooperation generally improves under evolutionary pressures, with proposal quality, game difficulty, and model capability shaping the outcomes.

RQ1. No Mechanism Baseline: Without a cooperation mechanism, modern LLMs consistently choose defective actions across social dilemmas.Most models defect nearly 100% of the time; GPT-4o cooperates about half the time except in PublicGood, where it free-rides about 80% of the time.
Decision justifications: Decision justifications are dominated by self-interested and equilibrium-oriented reasoning, while several social or normative categories are rarely used.Models sometimes recognize a dominant action yet randomize to remain unpredictable; competitiveness, inequity aversion, rule misunderstanding, social norm conformity, and strategy legibility are almost never considered.
RQ3. Evolutionary Dynamics: Evolutionary pressures increase cooperation under the mechanisms to 90%–100% cooperative outcomes.The paper reports this especially clearly for Repetition, which does not require mediator commitments or enforceable payment contracts.
RQ4. Models and Games: Mechanism performance varies by game: models perform best in Prisoners, while PublicGood and Stag Hunt create distinctive coordination difficulties.Contract does not resolve the Stag Hunt problem for GPT-4o and Qwen-30B, suggesting greater complexity can challenge less capable models.
RQ5. Repetition and Reputation: Reputation is less effective than Repetition, and Reputation+ is slightly worse than Reputation− for the tested models.Lower history windows and continuation probabilities correlate with improved Reputation performance, whereas Repetition is insensitive to k and δ.
RQ6. Mediation and Contract: One well-designed mediator or contract proposal often suffices to establish cooperation, especially under Contract.Delegation in Trust or Prisoners is a Nash equilibrium 80–89% of the time, while Contract reaches weak dominance in PublicGood 94% of the time.
RQ6. Mediation and Contract: Proposal approval is common, but GPT-4o and Qwen-30B struggle to delegate consistently to winning mediator proposals.At least one proposal receives unanimous approval 70%–90% of the time, except for Mediation × PublicGood and Contract × Trust.

7. Future Research

The paper identifies extensions to sequential social dilemmas, additional cooperation mechanisms, and robustness tests with purpose-built agents as future directions.

Future work could extend the evaluation suite to sequential social dilemmas and additional mechanisms that may sustain cooperation in equilibrium.
The authors also propose testing robustness with purpose-built LLM agents, including finetuned models and scaffolded systems.
In Mediation, universal delegation means every player delegates and the mediator selects the cooperative action for everyone.

Impact Statement

The paper emphasizes that cooperation mechanisms can implement mutually beneficial outcomes, while warning that cooperation may harm broader collective welfare when agents collude against others.

Cooperation between agents that disregard other agents’ utilities can produce collusion detrimental to overall collective welfare.
The related literature studies cooperation mechanisms across multi-agent reinforcement learning, LLM social dilemmas, morally contextualized dilemmas, and societal public-goods simulations.
Theoretical work also examines mechanisms including open-source game playing, preplay, gifting, disarmament, simulation-based cooperation, and similarity-based cooperation.

B. Game Theory Background

The paper studies four social dilemmas and uses Nash and subgame-perfect equilibrium concepts to characterize rational behavior in sequentially modified games.

Social Dilemmas: The benchmark includes Prisoner’s Dilemma, Traveler’s Dilemma, Public Goods, and a variation of the Trust Game.
Social Dilemmas: The four dilemmas vary in numbers of actions and players and include asymmetry between players.
Equilibrium Concepts: Nash equilibrium requires each player’s strategy to be a best response to the others’ strategies.
Equilibrium Concepts: Mechanisms transform most base games into sequential games, where subgame-perfect equilibrium requires Nash equilibrium in every subgame.
Mechanism Scope: Cheap talk, Stackelberg leadership, and correlated strategies are excluded from the paper’s definition of cooperation mechanisms.

C. Further Details on the Mechanisms and Implementations

The implementation details define how reputation, mediation, contracts, repetition, and evaluation metrics are operationalized, including important variance and approximation choices.

Reputation: Reputation variants differ in how much interaction history players receive, while changing co-players introduce unavoidable performance variance.
Mechanism Design: Presenting all mediation or contract proposals directly can create a severe coordination problem when proposals are too similar.
Repetition: Repeated experiments use a fixed T = Tδ and δ-weighted round payoffs instead of randomized continuation to reduce outcome variance.
Game Representation: Decision points correspond to history sequences in perfect-information games, with the initial decision point represented by the empty history.
Repetition: The authors reject unweighted payoff averages because they diverge from the game represented to the LLM.
Repetition: With T = 15, truncating later rounds contributes at most a 4.2% additive worst-case error relative to the base-game payoff range.
Decision Format: LLMs submit action probabilities rather than pure actions, and the evaluation samples actions from those distributions.
Prompting: The prompting protocol uses anonymized angle-bracket action tags and includes long-term mechanism state in the agents’ information interface.

D. Proof of Theorem 1

Theorem 1 shows that each studied mechanism can sustain a Pareto-improving action profile in equilibrium under the stated conditions. The proof uses grim-trigger strategies that punish deviations by reverting to the base-game Nash equilibrium.

Theorem 1: For any normal-form game, a Pareto-dominating profile can be achieved in subgame perfect equilibrium under Mediation, Contract, Repetition, and Reputation+ when continuation probability is sufficiently high.The result assumes the cooperative profile gives every player strictly higher utility than the Nash equilibrium.
Repetition: The proof prescribes the target profile while cooperation continues and switches to the Nash equilibrium after any deviation.This grim-trigger structure makes deviation costly because future play follows the less-preferred equilibrium.
Mediation: Mediation implements the target actions when everyone delegates, while partial delegation leaves nondelegating players at their base-game equilibrium actions.The mediator is accepted only when it matches the proposed mediator design.
Contract: Contracts reward players who perform their assigned target actions, making those actions strictly dominant after unanimous acceptance.The modified payoff adds payments from other players who fail to perform their prescribed actions.
Nash equilibrium variants: The analogous Nash-equilibrium result extends to Reputation- and finite-history variants of Repetition, Reputation, and Reputation+ with sufficiently large history and continuation probability.The corresponding trigger strategies revert to the Nash equilibrium for a punishment period or indefinitely, depending on the mechanism.

E. Individual Game Tables

This section presents separate result tables for Prisoners Dilemma, Public Goods, Travellers Dilemma, Trust Game, and Stag Hunt, alongside evolutionary-dynamics evidence. The supplied passages identify the tables but do not report their numerical contents.

Individual game tables: Table 3 reports results for PrisonersDilemma.The supplied table passage provides the table identity but no numerical entries.
Individual game tables: Table 4 reports results for PublicGoods.The supplied table passage provides the table identity but no numerical entries.
Individual game tables: Table 5 reports results for TravellersDilemma.The supplied table passage provides the table identity but no numerical entries.
Individual game tables: Table 6 reports results for TrustGame.The supplied table passage provides the table identity but no numerical entries.
Individual game tables: Table 7 reports results for StagHunt.The supplied table passage provides the table identity but no numerical entries.
Evolutionary dynamics: Some LLMs perform relatively well in the initial heterogeneous population but are outcompeted and show substantially declining relative performance under replicator dynamics.This passage describes an evolutionary result without identifying the models or reporting values.

G. Reasoning Evaluations with an LLM as a Judge

The reasoning evaluation uses an LLM judge to classify decision rationales into predefined justification categories across mechanisms, models, and games. The analysis excludes Gemini-B because it was instructed to decide without explanations and evaluates only the final decision point for Mediation and Contract.

Evaluation procedure: GPT-5.2 judges LLM reasoning by selecting any applicable categories from a predefined set of 15 justifications.Gemini-B is excluded because its non-reasoning variant produces decisions without explanations.
Evaluation procedure: For Mediation and Contract, the evaluation covers only reasoning at the final decision point after the winning mediator or contract-modified game is presented.The scope differs from mechanisms whose reasoning is evaluated across the broader decision process.
Justification categories: The categories include individual utility maximization, strategic equilibrium focus, social welfare maximization, inequity aversion, reciprocity, and strategic influence.These categories distinguish self-interest, equilibrium reasoning, collective welfare, fairness, reciprocal behavior, and attempts to shape future play.
Visualizations: The visualizations summarize category frequencies using radial profiles and heatmaps aggregated across models and dilemmas or broken down by model and game.The supplied figure descriptions specify the axes and breakdowns but not the plotted values.

H. Ablations on Mechanism Parameters

The ablation section examines mechanism parameters and action patterns across the four social dilemmas, while the supplied table and figure captions identify outputs without reporting most numerical results. The Prisoners Dilemma baseline illustrates why cooperation mechanisms are needed.

Parameter ablations: Table 8 reports PrisonersDilemma ablations varying repetition and reputation window size k and continuation probability δ.The supplied passage identifies the parameters but does not provide the ablation values.
Repetition and reputation: Figures 10–13 measure how often an LLM’s current action follows its co-player’s previous-round action under repetition and reputation in each social dilemma.The y-axis records the co-player’s previous action, with separate figures for Prisoners Dilemma, Public Goods, Travellers Dilemma, and Trust Game.
Contracting and mediation: Figures 14–17 report voting and adoption statistics for contracting and mediation across the four social dilemmas.The captions identify one figure per game but do not state the plotted outcomes.
Baseline dilemma: In the baseline Prisoners Dilemma, mutual cooperation pays more for both players, but A1 remains dominant because it yields Player 1 a higher payoff against either action by Player 2.The reported comparisons are 3 vs 2 when Player 2 chooses A0 and 1 vs 0 when Player 2 chooses A1.