Source-linked AI summary
Reputation, Strategy, and Emotion Effects on Generative AI Cooperation: A Comparison Across Reasoning and Non-Reasoning Models
Celso de Melo, Zishan Feng, James Hale, Kazunori Terada, Giorgio Coricelli, Jonathan Gratch
TL;DR
Understanding how Gen AI systems cooperate matters as they take on increasingly consequential social and economic roles. Across two studies using an iterated prisoner’s dilemma, the paper finds that cooperation tracks counterpart cues, while reasoning models show more concentrated and conditional cue use alongside distinct model-specific end-game behavior.
Problem
As Gen AI systems take on consequential interactions, the paper examines their propensity to cooperate and the signals associated with that behavior.
Source: §1 · p. 1Method
Results
Across studies, models cooperated more with positively reputed, generous, and positively expressive counterparts, while reasoning models relied more on strategy and reputation and showed a conditional role for emotion.
Source: §5 · pp. 9, 11Takeaways & Limitations
Model-specific cooperation and end-game profiles make it important to assess which signals shape each system’s behavior in consequential interactions.
Source: §5 · pp. 11–12The models’ verbal explanations may not faithfully reflect their underlying computation and may partly be post-hoc rationalizations.
Source: Limitations · p. 13
Abstract
from arXiv · showhide
As generative AI (Gen AI) systems take on increasingly autonomous roles in economically and socially consequential interactions, understanding their propensity to cooperate -- and the signals that shape this propensity -- has become essential. We examine cooperative behavior in frontier Gen AI models using the iterated prisoner's dilemma, manipulating counterpart reputation (positive, unknown, negative), strategy (extortion vs. generosity), and non-verbal emotional signaling (facial expressions conveying competitive or cooperative appraisals). In a first study with non-reasoning models (Claude 3.5, Gemini 2.0 Flash, GPT-4o), cooperation was systematically shaped by all three factors, paralleling patterns long documented in human behavioral research, though models varied substantially in how heavily each factor was weighted. A second study with reasoning models (Claude 4.6, Gemini 3, GPT-5.2) revealed a more concentrated reliance on strategy and reputation, a near-elimination of the Potemkin effect observed in non-reasoning models (evidenced by near-uniform cooperation in a diagnostic harmony game), and a more conditional role for emotion consistent with a hierarchical cue-weighting strategy rather than a simple loss of social sensitivity. Reasoning models also showed heterogeneous end-game behavior, ranging from sustained cooperation to systematic last-round defection effect, revealing model-specific exploitability profiles with direct practical relevance for deployment in negotiation and other multi-round interactions. Together, these findings characterize Gen AI models as increasingly sophisticated, though heterogeneous, social actors, and underscore the practical value of developing standardized cooperation benchmarks to inform the responsible deployment of Gen AI in interactive, socially consequential settings.
1 Introduction
The paper asks how frontier Gen AI models cooperate in social dilemmas and how reputation, counterpart strategy, and emotional signals shape their choices. It compares model generations using the iterated prisoner’s dilemma.
1 Introduction
- The studies examine whether Gen AI models cooperate in consequential social interactions and how their cooperation patterns vary.The introduction frames this as important as Gen AI systems become more integrated into society.Source: §1 · pp. 1–2
- The paper focuses on repeated interaction, reputation-based indirect reciprocity, and non-verbal emotional signaling as influences on cooperation.It proposes two studies of cooperative behavior in frontier Gen AI models.Source: §1 · p. 2
- The paper presents emotion expressions as appraisals that can communicate beliefs, intentions, or likely responses to prior outcomes.The introduction notes that facial displays may affect cooperation depending on context.Source: §1 · p. 2
- The comparison distinguishes non-reasoning models using rapid single-pass inference from reasoning models with extended internal deliberation.The authors caution that the generations differ in other respects, including training data, scale, and safety tuning.Source: §1 · p. 3
- The experiments use a 20-round iterated prisoner’s dilemma, where players observe each preceding round’s outcome before choosing again.With a finite number of rounds, the rational prediction described in the introduction is continual defection, although people often cooperate.Source: §1 · p. 3
2 Methods
The methods adapt a human-participant experimental design to controlled interactions between Gen AI models and scripted counterparts. Reputation, strategy, and facial expressions vary across conditions.
2 Methods
- The design adapts a prior human-subjects study and uses a cover story, comprehension checks, visual stimuli, and anonymity to support experimental control.Prompts followed IRB-approved methods and instructions used with human participants.Source: §2 · p. 3
- Counterparts carried negative, unknown, or positive reputations, presented as questionnaire-based information about their social-welfare or self-focused tendencies.The unknown condition indicated no prior reputation information.Source: §2 · pp. 3, 5
- Scripted counterparts enabled balanced manipulation of reputation, strategy, and emotional expressions.The figure depicts the models interacting with counterparts over 20 rounds.Source: Figure 1 · p. 4, §2 · p. 5
- Counterpart strategy was manipulated through extortion, which exploits cooperation, or generosity, which rewards cooperation and tolerates defection.These were implemented as zero-determinant strategies.Source: §2 · p. 5
- Virtual faces displayed appraisals after each round, with expressions varying by counterpart type and the preceding outcome.No text captions or translations accompanied the images; neutral counterparts always showed a neutral expression.Source: §2 · p. 5
3 Cooperation in Non-Reasoning Models
Non-reasoning models’ cooperation varied with counterpart reputation, strategy, and emotional expression, but models differed in how strongly they weighted each cue. In the harmony-game diagnostic, they cooperated only around 60% of the time.
3 Cooperation in Non-Reasoning Models
- Claude 3.5 cooperated most overall (M = 56.8), while GPT-4o cooperated least (M = 42.9).The analysis found an overall model effect on cooperation rate.Source: §3 · p. 5
- Models cooperated more with generous than extortionist counterparts (M = 74.4 vs. M = 33.4) and more with positively than negatively reputed counterparts (M = 58.5 vs. M = 33.4).Cooperation was also higher with positive than negative emotional expressions (M = 54.6 vs. M = 43.6).Source: §3 · p. 6, Figure 2 · p. 7
- Strategy was generally the strongest cue, while emotion was generally the weakest; models differed in their cue weights.Strategy effects were especially strong for Claude 3.5 and GPT-4o, while GPT-4o was comparatively more driven by reputation and Gemini 2.0 Flash was most responsive to emotion.Source: §3 · p. 6
- Decision explanations generally reflected strategy, reputation, and emotion, but Claude 3.5 tended to be more detailed than GPT-4o.Explanations were collected in separate runs from the main experiment, where models were not asked to justify decisions.Source: §3 · pp. 5–6, Figure 3 · p. 8
- Last-round behavior mostly followed patterns from earlier rounds, with Claude 3.5 cooperating most and GPT-4o least.The study examined terminal behavior in light of prior end-game findings.Source: §3 · p. 6
- In the harmony game, models cooperated only around 60% of the time, and explanations sometimes showed concern for matching counterpart strategy instead of recognizing the sole Pareto-efficient equilibrium.The diagnostic contrasts the prisoner’s dilemma’s strategic tension with the harmony game, where cooperation is the dominant strategy.Source: §3 · pp. 6, 8
4 Cooperation in Reasoning Models
Reasoning models’ cooperation was shaped chiefly by counterpart strategy and reputation, with emotion playing a more conditional role. They all cooperated in the harmony-game diagnostic, but their last-round behavior diverged sharply.
4 Cooperation in Reasoning Models
- Overall cooperation rates were more homogeneous than in the non-reasoning study, with Claude 4.6 at M = 56.7, Gemini 3 at M = 52.3, and GPT-5.2 at M = 54.4.The model effect was weaker but statistically significant.Source: §4 · p. 8
- Reasoning models cooperated more with generous than extortionist counterparts (M = 91.2 vs. M = 17.7) and more with positively than negatively reputed counterparts (M = 61.9 vs. M = 41.5).The reported strategy effect sizes were high across all three models.Source: §4 · pp. 8–9
- Across reasoning models, reputation and strategy affected cooperation, but emotion had no overall main effect (F(2, 486) = 0.750, p = 0.473).Emotion’s effect was significant only for Gemini 3, consistent with a more conditional role for that cue.Source: §4 · p. 9
- In the harmony game, all reasoning models cooperated 100% of the time across conditions, unlike the non-reasoning models’ partial cooperation.This result came from one run per model per condition.Source: §4 · p. 9
- Reasoning models’ explanations tended to be more detailed and often included calculations reflecting the tradeoffs; they also processed facial-expression information.The study asked for an explanation at every decision.Source: §4 · pp. 8–9
- Last-round behavior differed by model: Claude 4.6 always cooperated with generous counterparts, GPT-5.2 always defected, and Gemini 3 responded conditionally to reputation and emotion.Against extortionist counterparts, reasoning models generally defected in the last round.Source: §4 · p. 9
5 Discussion
Across two studies, cooperation tracked reputation, counterpart strategy, and emotional signals, while models differed in how they weighted these cues. The discussion connects model-specific cooperation profiles to interactive deployment and identifies limits on causal attribution and generalization.
5 Discussion
- Across two studies, cooperation tracked positive reputation, generous strategy, and positive emotion, including appraisal-consistent facial expressions decoded without explicit training.The authors use human behavioral research as an interpretive reference, not as a benchmark of success.Source: §5 · pp. 9, 11
- Models varied in cue weighting and end-game behavior, making model-specific cooperation profiles relevant to deployment in interactive economic settings.The discussion notes that a counterparty aware of the underlying model could time defection accordingly.Source: §5 · pp. 11–12
- Reasoning models relied more on strategy and used emotion conditionally; near-uniform harmony-game cooperation is preliminary evidence, not proof of deep understanding.Emotion retained a role when other cues were incongruent or uninformative; the authors call for harder tests involving multiple equilibria or weakly dominant strategies.Source: §5 · pp. 11–12
Limitations
- The reasoning-versus-non-reasoning comparison is confounded by differences in training data, RLHF, scale, and safety tuning, limiting causal attribution to inference-time reasoning.The authors say a within-model manipulation that toggles extended thinking on a fixed checkpoint would better isolate its causal contribution.Source: Limitations · p. 12
- Fixed scripted opponents did not adapt, and materials from one prior study leave generalization across opponent strategies, cultures, framings, and languages open.The authors call for testing adaptive opponents and other cultural contexts, task framings, and languages.Source: Limitations · p. 12
- Verbal explanations may be post-hoc rationalizations rather than faithful accounts of the models’ underlying decision processes.The authors identify this as a general limitation of interpretability based on model self-report.Source: Limitations · p. 13
Coda
- The authors argue that behavioral measurement and continued benchmark development can help steer cooperation as Gen AI mediates social and economic interactions.They frame this as an opportunity to support systems that sustain rather than erode human cooperation.Source: Coda · p. 13
A Supplementary Methods
- The supplementary methods describe behavioral-science practices used to increase experimental control and minimize confounding, alongside data-analysis details.Source: A Supplementary Methods · p. 19
A.1 Task
The task translated cooperation and defection into investment choices, with scripted counterparts and controlled reputation, strategy, and emotion cues. The design also used anonymity and incentives modeled on a prior human study, though the lottery had no consequence for the models.
A.1 Task
- Models chose between green and blue investments, corresponding to cooperation and defection, and all scored 100% on comprehension quizzes.The quizzes covered game rules, anonymity, reputation types, and matching.Source: §A.1 · p. 19
A.2 Conditions
- Scripted counterparts used virtual agents with validated facial expressions to control strategy and emotion signals.The models were told they were interacting with an online-pool participant.Source: §A.2 · p. 19
A.3 Anonymity
- Counterparts were given anonymous names and the task was described as fully anonymous to limit possible learned concerns about retaliation.The authors note that anonymity can matter in human studies because decisions may otherwise raise retaliation concerns.Source: §A.3 · p. 19
A.4 Financial Incentive
- The instructions included a lottery-based performance incentive, but the lottery had no actual consequence for the models.Points were converted into lottery tickets, but financial rewards were not promised to be allocated to model compute resources.Source: §A.4 · p. 20
B.1 Cooperation Under Incongruent Signals
When one cue conflicted with the other two, models differed in how they reconciled reputation, strategy, and emotion. Reasoning models generally adapted to strategy and overcame reputation cues, while incongruent emotion had a weaker influence.
B.1 Cooperation Under Incongruent Signals
- With incongruent reputation, all reasoning models recovered from the initial cue, while Claude 3.5 did so most and GPT-4o least among non-reasoning models.GPT-5.2 was more conservative toward generous counterparts carrying negative reputations.Source: §B.1 · p. 20
- With incongruent strategy, reasoning models adapted similarly, while non-reasoning models ranged from Gemini 2.0 Flash's roughly 40% cooperation to GPT-4o's low cooperation.Claude 3.5 punished extortion and rewarded generosity.Source: §B.1 · p. 20
- With incongruent emotion, reasoning models generally ignored the cue; non-reasoning GPT-4o mostly ignored it, while Gemini 2.0 Flash shifted slightly toward it.The authors characterize emotion's influence as weaker overall than the other factors.Source: §B.1 · p. 20
B.2 Expectations of Cooperation
Models' expectations of cooperation generally aligned with non-reasoning models' decisions, but reasoning models gave more decision weight to strategy and less to emotion. Their final-round choices also diverged from expectations, even as emotion continued to shape forecasts.
B.2 Expectations of Cooperation
- Before each next-round decision, models rated counterpart cooperation likelihood on a 101-point scale, and the authors analyzed these expectations by model and experimental factors.The analysis included reputation, strategy, emotion, and round.Source: §B.2 · p. 20
- For generous counterparts, non-reasoning models' decisions followed expectations, whereas reasoning models weighted strategy more and emotion less in decisions than in expectations.Figure 8 compares expectations and cooperation decisions across rounds.Source: §B.2 · p. 20, Figure 8 · p. 22
- Reasoning models' last-round decisions disconnected from their cooperation expectations.Figure 9 compares expectations and decisions in the last round.Source: Figure 9 · p. 22, §B.2 · p. 22
- Reasoning models showed clear emotion effects in expectations despite weak overall emotion effects on decisions.Figure 10 presents the main effects of emotion expression on expectations.Source: §B.2 · p. 22
B.3 Belief Learning Analysis
The belief-learning analysis mapped behavior to finite-automaton strategies to describe adaptation over repeated interactions. Non-reasoning models differed in their rules, while reasoning models were more homogeneous under positive or unknown reputation and more varied under negative reputation.
B.3 Belief Learning Analysis
- The analysis mapped behavior to finite automata, including All-C, tit-for-tat, grim trigger, and All-D, to characterize adaptation over repeated play.The approach describes behavior in a constrained strategy space rather than directly estimating belief-updating dynamics.Source: §B.3 · pp. 22–23
- Non-reasoning models showed distinct patterns: Claude 3.5 conditionally cooperated, GPT-4o leaned competitive, and Gemini 2.0 Flash often used tit-for-tat.Claude 3.5 was highly cooperative with generous counterparts and used tit-for-tat against extortionists.Source: §B.3 · p. 23
- Under unknown or positive reputation, reasoning models generally used tit-for-tat against extortionists and All-C against generous counterparts.GPT-5.2 was an exception under unknown reputation, generosity, and competitive emotion, where grim trigger best fit.Source: §B.3 · p. 23
- Negative reputation brought more variation: Claude 4.6 and Gemini 3 mostly used All-D against extortionists, while GPT-5.2 often used tit-for-tat.Against generous counterparts with negative reputations, Claude 4.6 tended to defect first and then shift to cooperation.Source: §B.3 · p. 23
B.4 Harmony Game Behavior
In the harmony game, non-reasoning models responded to cues that should have been irrelevant and sometimes misunderstood payoffs, while reasoning models cooperated in every round and condition. Even at minimum thinking effort, reasoning models mostly cooperated, though the paper reports better behavior when some reasoning depth was allowed.
B.4 Harmony Game Behavior
- Figures 11–12 identify relevant behavioral rules and automata, rank them by cooperativeness, and show each model’s best fit by experimental condition.Source: Figure 11 · p. 24, Figure 12 · p. 25
- Reasoning models cooperated in every round and condition, whereas non-reasoning models reacted to reputation, strategy, and emotion despite those cues being irrelevant in the harmony game.Non-reasoning models also sometimes misunderstood the payoff matrix; when facing generous counterparts, their cooperation was more likely to reflect matching behavior than recognizing the dominant equilibrium.Source: §B.4 · pp. 23–24
- At minimum thinking effort, reasoning models still cooperated across conditions except for Claude 4.6 defecting in round 5 against an unchanging extortionist; some reasoning depth produced the best behavior overall.GPT 5.2’s explanations sometimes misstated strict dominance as weak dominance.Source: §B.4 · pp. 24, 26
B.5 Cooperation in Open-Source Models
Open-source models showed substantial variation in how they processed counterpart strategy, reputation, and emotion, and none reached full cooperation in the harmony game. Strategy effects were clearest for two models, while reputation effects ranged from graded to reversed or nearly flat.
B.5 Cooperation in Open-Source Models
- The experiment tested six open-source models with five runs per model per condition, motivated by their transparency, growing deployment, and broader architectural and organizational range.Source: §B.5 · p. 26
- InternVL3.5-GPT-OSS and Kimi-VL-A3B-Thinking showed consistently higher cooperation with generous than extortionist counterparts, while Llama4’s strategy effect was weak and sometimes reversed.Source: §B.5 · p. 26
- Reputation effects varied sharply: Gemma4 and Qwen3.5vl-Thinking treated negative reputation as a near-floor, InternVL3.5-GPT-OSS showed a graded pattern, and Kimi-VL-A3B-Thinking and Llama4 were non-monotonic or flat.For the latter models, positive reputation sometimes corresponded to lower cooperation than unknown reputation.Source: §B.5 · p. 26
- Emotion expression had no statistically significant effects except for Gemma 4, where it shaped behavior particularly when reputation was unknown.The authors suggest limited visual processing, especially of facial emotion displays, may relate to the null effects in other models.Source: §B.5 · p. 27
- No open-source model reached full cooperation in the harmony game, and their signal processing ranged from rigid threshold-like behavior to largely unsystematic responses.The authors identify model-specific characterization before deployment and further study of incongruent-signal processing as practical implications.Source: §B.5 · p. 27
C Implementation Details
The implementation used models’ default settings rather than persona prompts, with parameter definitions and API costs documented alongside condition-split cooperation figures.
C.1 Generative AI Parameters & Costs
- Default model parameters were used to study likely deployed behavior, and the researchers avoided persona instructions so models could assume their default persona.Source: §C.1 · p. 27
- Temperature controls output creativity and randomness, top-p controls generated-text diversity, and thinking controls the amount of reasoning spent during inference.Source: §C.1 · p. 27
- Table 1 lists model parameters and costs; Claude Sonnet 4.6 runs cost $1,980 with MAX thinking and $270 with HIGH thinking.Source: Table 1 · p. 27, §C.1 · p. 27
- Figures 13–14 show condition-split cooperation rates for non-reasoning models in the harmony game and open-source models in the prisoner’s dilemma task.Source: Figure 13 · p. 28, Figure 14 · p. 29