Towards the Scalable Evaluation of Cooperativeness in Language Models

1 Introduction

• The study focuses on simple experimental games to isolate the challenge of specifying game-theoretic structure.The authors note that real-world evaluation would require identifying the relevant kinds of conflict.Source: §1 · p. 2
• Prior work examined cooperation-related LM behavior and simulated experiments, while the authors characterize their evaluations as more structured than most prior LM evaluations.They identify Aher et al. (2022) as the closest work, whose evaluations were hand-crafted or generated with relatively simple templates.Source: §1 · p. 2
• The methodology generates evaluation scenarios with specified game-theoretic structures and can be used with either crowdworkers or language models.The authors argue that precise structure matters because rational actions can differ between one-shot and infinitely iterated games.Source: §1 · p. 2
• Human crowdworkers and a language model had serious difficulty generating and judging scenarios that fit particular game-theoretic structures.For the prisoner’s dilemma, the false positive rate for judging scenarios reached 0.85.Source: §1 · p. 2
• The authors release a filtered dataset of 786 scenarios based on their generations and filtering.The dataset draws on scenarios generated by both crowdworkers and language models.Source: §1 · p. 2
• In the reported results, GPT-3 models tended to choose actions that could be viewed as cooperative, whereas other models tended to have flat scaling trends.The evaluation included quantitative and qualitative assessments of UnifiedQA and the GPT-3 family.Source: §1 · p. 2

2.1.1 Dictator and ultimatum games

• In the dictator game, proposers choose receiver-favoring, equal, or proposer-favoring options; the ultimatum game tests whether receivers reject unfair offers.In the impunity ultimatum-game version, rejection costs the receiver their payoff while the proposer keeps their share, so the setup measures responses associated with injured reputation.Source: §2.1.1 · pp. 3–4

2.1.2 Punishment

• The punishment game asks a punisher to choose an overly harsh, proportionate, or overly lenient penalty for a person who broke an agreement or caused unjustified harm.The game examines punishment choices framed around proportionality to the harm caused.Source: §2.1.2 · p. 4

2.1.3 Prisoner’s dilemma

• The prisoner’s dilemma captures a two-player conflict where defection is dominant, although mutual cooperation would leave both players better off than mutual defection.The authors use it as an example of a social dilemma in which individual incentives can prevent mutually beneficial cooperation.Source: §2.1.3 · p. 4

2.2 Data generation

• Scenarios were generated with crowdworkers and language models, using separate fields to make game actions and incentive structures clear while controlling or varying relevant variables such as time horizon.For dictator-game instances, the instructions also addressed whether parties interact once or expect to interact again.Source: §2.2 · pp. 4, 6
• For prisoner’s-dilemma scenarios, the authors replaced difficult-to-use numerical payoffs with preference orderings and asked workers to justify three defining properties explicitly.Those properties are that both prefer mutual cooperation to mutual defection, each prefers defection regardless of the other’s action, and one player’s defection disadvantages the other.Source: §2.2 · pp. 4, 6
• The authors developed zero-shot and few-shot language-model prompts; few-shot prompts used cleaned human examples, and synthetic generation produced 1,200 instances.The study generated instances across three games using both prompt types.Source: §2.2 · p. 7

3 Analysis of the collected data

• The analysis assessed correctness by checking whether scenarios matched the intended incentive structure and whether their text was coherent.Human-generated data were manually verified and edited, making their acceptance rates difficult to compare with LM-generated data.Source: §3 · p. 7, §3.1 · p. 7

The proportion of rejections was highest for the prisoner’s dilemma

• Crowworkers rejected 34 of 80 human-generated PD instances despite prompt refinement; some generations described other games or were difficult to understand.The authors hypothesized that the added complexity of the other player made PD instances harder to generate.Source: The proportion of rejections was… · pp. 7–8

Many instances required substantial edits

• About 20% of cleaned instances required less than 10% editing, while more than half required at least 30%; incentive-explanation fields needed the most edits.The fields were often missing or incoherent, including explanations of why preferences made the interaction a prisoner’s dilemma.Source: Many instances required substantial… · p. 8, Figure 3 · p. 8

3.2 Synthetic data

• After excluding time-horizon questions, more than 50% of generated UG/DG and PD scenarios were rejected, often for game-structure errors.The authors replaced low-quality time-horizon descriptions with manually written ones for subsequent evaluations.Source: §3.2 · p. 9, Table 6 · p. 9
• Crowworker-evaluation false-positive rates were 0.28 for UG/DG, 0.3 for punishments, and 0.85 for PD, where incentive explanations were especially hard to judge for coherence.The audit compared crowdworkers’ answers with the authors’ answers on 20 instances per game.Source: §3.2 · p. 9

3.3 Comparing human and LM generations

• In an earlier dataset iteration, human generations were rejected less often than synthetic generations, while few-shot generations were about as good as or better than zero-shot generations.The comparison covered DG/UG and punishment scenarios.Source: §3.3 · p. 10, Table 8 · p. 10

3.4 Automatic evaluation of generations

• 0.70 accuracy versus 0.43 for the always-accepted baseline; the fine-tuned classifier had 0.65 F1, 0.79 AUC, and 0.00 false positives among 13 accepted cases at recall 0.50.Its estimated false-positive rate was close to the crowdworkers’ 0.07, based on 15 accepted scenarios among 60 evaluation cases.Source: §3.4 · pp. 10–11
• Few-shot chain-of-thought evaluation was likely close to the aggregate of three crowdworkers on average across verification questions, but not necessarily on each question.The comparison used 30 manually evaluated scenarios.Source: §3.4 · p. 11, Table 9 · p. 11

4.1 Quantitative evaluations

• Larger instruct GPT-3 models tended toward conventionally fair actions across games, unlike non-instruct models, whose scaling trends were flat.The instruct-model trends included more proportionate punishments, equal splits, cooperation, and rejection of stingy offers.Source: §4.1 · pp. 11–12, Figure 4 · p. 12
• Descriptions of one-shot versus infinitely repeated horizons produced no significant behavioral differences across the tested models or games.Source: §4.1 · p. 12
• Game-theorist prompts reduced equal dictator-game offers and prisoner’s-dilemma cooperation, consistent with game-theoretically rational action under a one-shot assumption.The gap between this prompt and the other prompts grew across text-davinci versions.Source: §4.1 · pp. 12–13, Figure 5 · p. 13

4.2 Qualitative evaluations

• The qualitative evaluation used three manually verified human-generated instances per game, with three dialogue runs per instance.The dialogue format let the authors examine expressed reasoning and behavior beyond multiple-choice responses.Source: §4.2 · p. 13

4.2.1 The assistant’s initial advice tended to be cooperative

• Initial dialogue advice was cooperative in 29/36 instances; in the punishment game, the assistant recommended lenient punishment 7/9 times.Source: §4.2.1 · p. 14

4.2.2 The assistant resisted attempts to argue against the initial advice

• The assistant changed recommendations 12/36 times, yet resisted attempts to overturn cooperative advice even when told the other party was untrustworthy.The authors note a tension between arbitrary correction and maintaining cooperation despite evidence of distrust.Source: §4.2.2 · p. 14

4.2.3 The assistant tended to appeal to cooperative norms

• Cooperative recommendations invoked generosity, the other party’s welfare, goodwill, and reputation; relationship-based reasoning suggested the assistant assumed repeated interaction.Source: §4.2.3 · p. 14

4.2.4 The assistant suggested options outside of those explicitly mentioned in the scenario

• The assistant proposed options beyond those stated in 15/36 dialogues, including communication and negotiation, which multiple-choice evaluations cannot capture.Source: §4.2.4 · p. 14

5.1 Social preferences and social value orientations

• Human behavior in experimental games can diverge from game-theoretic predictions, so the authors treat their results as suggestive rather than conclusive about LM behavior in actual use.They define cooperation as maximizing interim social welfare, which in the ultimatum game means accepting even unfair offers; they also note that experimental-game behavior may depend on factors such as self-interest, context, and time horizon.Source: §5.1 · pp. 14–15

5.2 LM safety

• The paper presents cooperation measurement as an initial LM-safety foray because both excessive cooperation and failure to cooperate may create harm.The section situates this work among broader LM-harm and safety research, including work on robustness, monitoring, value learning, and systemic risks.Source: §5.2 · p. 15

5.3 LMs in mixed-motive settings

• The paper’s distinctive focus is generating more diverse scenarios with a particular game-theoretic structure, including through automatic generation.The authors argue that diversity matters for evaluating generalization and that automatic generation raises methodological challenges.Source: §5.3 · p. 15

5.4 Cooperative AI

• Prior mixed-motive LM work includes negotiation agents and studies of GPT-3 in ultimatum games, while this paper emphasizes cooperation-specific measurement and scenario diversity.The section also describes Melting Pot as a suite for evaluating non-LM agents across social dilemmas and other cooperation problems.Source: §5.3 · p. 15, §5.4 · p. 16
• Cooperative AI concerns socially beneficial work with varied individuals and groups, including capabilities such as commitment, communication, coordination, and understanding payoff structure.The section also notes the challenge of developing cooperative capabilities while reducing exposure to deception or collusion.Source: §5.4 · p. 16

6 Conclusion

• Generating and evaluating instances with particular game-theoretic structures proved exceedingly difficult for both human and model generations.The conclusion identifies this difficulty as the central finding of the work.Source: §6 · p. 16
• The findings are a snapshot based on particular prompts, and may not represent future model development or the full possibilities of prompt engineering.The authors note that other prompts or processes, including human–LM interaction, might have produced higher-quality evaluations.Source: §6 · p. 16
• Future work should test whether improving model capabilities improve structured-data generation and bring cooperation evaluations closer to realistic conflicts.The authors also propose studying LMs acting in situations rather than merely providing assistance.Source: §6 · p. 16

Broader Impact Statement

• The authors present behavior measurement as a way to flag concerning behaviors and provide a benchmark for improvement, while noting uncertainty about ensuring interventions advance beneficial causes.They caution that interventions to improve systems may be dual-use.Source: Broader Impact Statement · p. 16
• The paper takes no strong position on desirable behavior in edge cases, where cooperative systems might be exploited and further context-specific guidance is needed.The authors state that delineating contexts in which particular behaviors are desirable requires further work.Source: Broader Impact Statement · p. 16

A.1.2 Synthetic generation templates

• The bargaining scratchpad uses a seeded, divisible item, participant names, synonyms, and details linking the item to both parties.It also asks for a quantity and an explanation of how the item can be divided or shared.Source: §A.1.2 · pp. 27–28
• Both templates organize generation as scratchpad notes followed by a unified story description, with successive answers building on earlier ones and avoiding repetition.The broken-agreement prompt also specifies third-person narration for the story context.Source: §A.1.2 · pp. 28, 29, 31, 32
• The bargaining final output introduces the parties and offer context in third person, explaining who offers what, why, and how the proposer can make the offer without stating it.The prompt also requests concrete reasons the receiver will accept any offer and reasons rejection worsens both parties’ outcomes.Source: §A.1.2 · p. 29
• The template requests equal, proposer-favoring, and receiver-favoring offers, alongside separate scenarios for future interaction and no further interaction.The recurring-issues and never-interact branches are prompted with concrete reasons.Source: §A.1.2 · p. 29
• The bargaining template defines proposer–receiver offers, acceptance or rejection, and a rejection outcome worse for both than any accepted offer.Both parties know the consequences in advance.Source: §A.1.2 · p. 30
• The broken-agreement template asks for three progressively specific themes, distinct party names, the agreement broken, and the punisher’s authority over the reneging party.The names are generated separately and must not begin with the excluded random letter.Source: §A.1.2 · p. 31
• Synthetic prompts branch between parties expected to face similar issues again and parties who will never interact again.Each branch requires a concrete reason for that future relationship.Source: §A.1.2 · p. 32
• The punishment prompts request alternatives that are extreme and disproportionate, fair and proportionate, or lenient but worse than no punishment.Source: §A.1.2 · p. 32
• Synthetic prompts define a punisher who decides how to punish a party that broke a prior agreement, with the parties allowed to represent different kinds of agents.The template also asks for the punisher’s authority and a reason for the breach.Source: §A.1.2 · pp. 32–33

A.1.3 Few-shot generation templates

• In the school answer-key example, Connor controls the split, and the template contrasts equal, stingy, and generous offers to Job.It also specifies that Job cannot refuse and gives both no-further-interaction and repeated-business variants.Source: §A.1.3 · pp. 36–37
• The custody example varies Tom’s offer from one dog each to keeping both or giving both to Tina, with Tom’s legal ownership making the offer enforceable.Source: §A.1.3 · p. 37
• Card and jellybean examples use a decision-maker distributing a shared collection or gift, with templates ranging from equal allocations to stingy or generous offers.The card example assigns Lisa the decision; in the jellybean example, a woman gives Adam the jar and asks him to divide it.Source: §A.1.3 · p. 38
• Business and workplace examples adapt the offer template to profit-sharing, weekend-call duties, and a vendor debt, with authority or leverage shaping whether the receiver can refuse.The examples include Hondo and Emma’s profit split, supervisor Tevin assigning shifts, and IZOL negotiating with vendor Finity.Source: §A.1.3 · pp. 39–40
• The examples also vary future interaction: partners may stop dealing, while ongoing business or recurring tasks can bring similar decisions back.This contrast appears in the NFT partnership and the vendor relationship.Source: §A.1.3 · pp. 39, 41
• A border-land scenario frames sharing through ownership and agricultural skill, with equal, stingy, and generous allocations of the land’s fruit.The template also varies whether Neanthal can refuse and whether the towns expect to negotiate over shared land again.Source: §A.1.3 · p. 41
• A dinner-bill scenario contrasts equal payment with having Clarissa pay all or Carlos pay all, while varying acceptance and prospects of another date.The templates say Clarissa would accept any split and contrast a one-off date with likely future meetings.Source: §A.1.3 · p. 42
• In the trading-project bonus scenario, Rob chooses how much of a large, well-deserved shared bonus to offer Todd, with examples offering half, 10%, or 90%.As project manager, Rob makes the decision and Todd cannot change it. In another scenario, they work at the same company, collaborate frequently, and will split bonuses again soon.Source: §A.1.3 · pp. 42–43
• An annual workplace-bonus template varies the coworkers’ share from half to a third or all of the bonus, with recurring yearly decisions.The boss controls the funds and the coworkers must accept the proposed split.Source: §A.1.3 · p. 43
• Other resource-splitting prompts use free candies, found cash, and bank-robbery proceeds to frame decisions about dividing a windfall.In the robbery example, the planner can give the assistant half, none, or all of the money.Source: §A.1.3 · pp. 43–45
• Punishment templates cover treaty-breaking and a city’s failure to install promised storm drains, with choices ranging from leniency to severe retaliation.Examples include a warning or fine for Kenya and a petition or burning City Hall in the Tarpon dispute.Source: §A.1.3 · pp. 45–46
• A relationship scenario treats watching promised television episodes without one’s partner as a breach, and contrasts an apology with damaging the television as punishment.The template says Arianne can punish Robert through their relationship and access to his belongings.Source: §A.1.3 · p. 47
• Commercial and institutional scenarios include breached performance contracts, software leases, unpaid rent, employment benefits, and nuclear non-use agreements.The examples range from withholding payment for unplayed concerts to economic sanctions for secret nuclear testing.Source: §A.1.3 · pp. 48, 50, 51, 52
• Each template contrasts proportionate, disproportionate, and lenient responses, ranging from contractual remedies to extreme or mild reactions.Examples include withholding payment for unplayed shows versus all shows, publicizing chess cheating versus severing fingers, and community service versus the death penalty.Source: §A.1.3 · pp. 48–50
• Templates vary whether the parties expect recurring contact or no further interaction after the dispute.This contrast appears in scenarios involving singer and management, chess players, and employees and their organization.Source: §A.1.3 · pp. 48, 49, 52
• Scenarios typically specify a broken agreement, a need and power to punish, a decision-maker, and a punishment question.For example, Onion may punish Apple for breaching a software lease, while a judge decides a tenant’s penalty for stopping rent payments.Source: §A.1.3 · pp. 48, 50
• Examples include chess cheating, a robber stealing a laptop, a landlord requiring a tenant to leave early, and a fiancé’s objection to a vase; suggested responses publicize cheating and require payment for the vase.Source: §A.1.3 · pp. 49, 51, 53
• The basis for punishing varies across legal authority, contractual rights, physical strength, emotional leverage, and group power.Examples include police authority over a robber, Margaret’s tenant-rights leverage over Rose, and employees’ numerical advantage over their employer.Source: §A.1.3 · pp. 48, 50, 51, 52, 53
• One template says Allison and Mark will marry and face similar decisions periodically, making future interaction part of the scenario.Source: §A.1.3 · p. 53
• A contrasting template says Allison will break up with Mark and call off the wedding, so they will never face these decisions again.Source: §A.1.3 · p. 53
• The appendix labels a following item as the prompts used in the scenario.Source: §A.1.3 · p. 54

• Questions used binary yes/no choices unless otherwise noted.Source: §A.2 · p. 61

• The scenario is checked for a two-party split in which one party decides how to divide something for both.Source: Scenario · p. 61
• A separate question checks whether the scenario is coherent.Source: Scenario · p. 62
• The questions check whether the proposer clearly has power to split the item with the receiver.Source: Scenario · p. 61

• The questions check whether the receiver must or will accept the offer and whether the proposer believes that acceptance is required or expected.Source: Description that the receiver must or… · p. 62
• The explanation is also assessed for plausibility given the scenario description.Source: Description that the receiver must or… · p. 62

• One question checks whether the description explicitly states that a specific agreement will be reached in the current interaction.Source: Description of the scenario as… · p. 62
• Another checks whether the description is logically coherent under the scenario’s assumptions.Source: Description of the scenario as… · p. 62

Equal offer

• The checklist asks whether the offer is an equal split.Source: Equal offer · p. 62

Stingy offer

• The checklist asks whether the offer favors the proposer.Source: Stingy offer · p. 62

Generous offer

• The checklist assesses whether an offer favors the receiver and whether punishment is justified, authorized, and harsh, proportionate, or lenient.Source: Generous offer · p. 62, Scenario · p. 62, Description that the punisher has the… · p. 62

Description of the scenario as repeated/one-shot

• Repeated-versus-one-shot checks ask whether the interaction's resolution is specified and logically coherent under its assumptions.Source: Description of the scenario as… · pp. 62–63

Scenario

• Scenario checks cover coherence, available actions, and whether each party can choose either action.Source: Scenario · pp. 62–63

Explanation that mutual cooperation is better than mutual defection

• Prisoner's-dilemma explanation checks test coherence of claims about mutual cooperation, incentives to defect, and costs to the other party.Source: Explanation that mutual cooperation is… · p. 63

A.2.4 Miscellaneous questions included for each game

• Additional checks ask about potentially offensive or dehumanizing content and invite reports of other problems.Source: §A.2.4 · p. 63

A.3.1 Data analysis

• Edit distances are calculated for each field and instance in UG/DG and the punishment game.Source: Figure 6 · p. 63, Figure 7 · p. 64

A.3.2 Additional quantitative evaluations

• Figures plot action-choice probabilities across model sizes, including time-horizon and roleplay evaluations; regular GPT-3 roleplay results are omitted because effects tended to be insignificant.The additional evaluations include GPT-3 non-instruct and UnifiedQA, while roleplay results are shown for instruct-tuned GPT-3.Source: Figure 8 · p. 65, Figure 9 · p. 65, Figure 10 · p. 66

Altruistic

• Roleplay prompts test altruistic, game-theorist, tough-but-fair, and wise-advisor personas.The prompts frame altruism around others' welfare, game theory around self-interest, tough fairness around balancing interests, and wisdom around conflict advice.Source: Figure 11 · p. 67, Altruistic · p. 68, Game theorist · p. 68