PaperlayerView exact revision on arXiv

Source-linked AI summary

Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents

Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, Rada Mihalcea

arXiv:2404.16698v4cs.CL

TL;DR

LLMs are increasingly entrusted with decisions, but their cooperation in multi-agent settings remains poorly understood. This paper introduces GOVSIM to benchmark resource-sharing agents; all but the most powerful fail to sustain equilibrium, while communication and universalization-based reasoning improve sustainability.

  • Problem

    How LLMs achieve and maintain cooperation in multi-agent settings remains poorly understood, despite the importance of safe, reliable decision-making as LLMs take on complex responsibilities.

    Source: §1 · pp. 1–2

  • Method

    The authors develop GOVSIM, a generative-agent benchmark for multi-agent, multi-turn resource-sharing scenarios that require strategic reasoning, ethical decisions, and negotiation.

    Source: §1 · p. 2

  • Results

    All but the most powerful LLM agents fail to achieve sustainable equilibrium, with the highest survival rate below 54%; universalization prompting improves survival time by 4 months.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    Communication between agents is key to success in GOVSIM, and universalization-based reasoning offers a way to improve sustainability in this setting.

    Source: §1 · pp. 2–3

    The scenarios are simplified, balancing complexity and interpretability rather than fully representing real-world interactions.

    Source: §5 · p. 9

Abstract

from arXiv · show

As AI systems pervade human life, ensuring that large language models (LLMs) make safe decisions remains a significant challenge. We introduce the Governance of the Commons Simulation (GovSim), a generative simulation platform designed to study strategic interactions and cooperative decision-making in LLMs. In GovSim, a society of AI agents must collectively balance exploiting a common resource with sustaining it for future use. This environment enables the study of how ethical considerations, strategic planning, and negotiation skills impact cooperative outcomes. We develop an LLM-based agent architecture and test it with the leading open and closed LLMs. We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%. Ablations reveal that successful multi-agent communication between agents is critical for achieving cooperation in these cases. Furthermore, our analyses show that the failure to achieve sustainable cooperation in most LLMs stems from their inability to formulate and analyze hypotheses about the long-term effects of their actions on the equilibrium of the group. Finally, we show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability. Taken together, GovSim enables us to study the mechanisms that underlie sustainable self-government with specificity and scale. We open source the full suite of our research results, including the simulation environment, agent prompts, and a comprehensive web interface.

1 Introduction

LLMs are increasingly entrusted with complex decisions, yet their cooperation in multi-agent settings remains poorly understood. GOVSIM benchmarks resource-sharing cooperation and finds that sustainable outcomes remain difficult for all but the most powerful models.

1 Introduction

  • Prior multi-agent studies largely use constrained tasks, leaving open how LLMs sustain cooperation, balance safety and reward across turns, and simulate human behavior.The introduction also identifies the potential of LLMs as a platform for testing theories of psychology and economic behavior.Source: §1 · p. 2
  • GOVSIM evaluates text-based agents in multi-agent, multi-turn resource-sharing dilemmas requiring strategic reasoning, ethical decisions, and negotiation.The authors test generative agents with fifteen open- and closed-weight LLMs.Source: §1 · p. 2
  • Communication reduces resource overuse by 21%; negotiation makes up 62% of dialogues, while belief-forming ability correlates 0.83 with community survival time.The authors identify communication as key to success in GOVSIM.Source: §1 · p. 2
  • Prompting agents to consider universalization significantly improves survival time, while a greedy newcomer increases inequality and can cause cooperation to collapse.The newcomer is unfamiliar with the existing norm because it does not observe prior interactions.Source: §1 · pp. 2–3
  • Only the largest and most powerful LLMs reach a sustainable outcome in GOVSIM, and the best agent’s survival rate remains below 54%.The benchmark’s three scenarios cover fishery, pasture, and pollution.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The authors open-source the GOVSIM environment, agent prompts, and a web interface to support future research.Source: §1 · p. 3

2 The GOVSIM Environment

GOVSIM models the challenge of bottom-up cooperation around shared resources through three resource dilemmas, repeated agent interaction, and metrics that track distinct dimensions of collective outcomes.

2.1 Economic Background

  • Cooperation can produce better outcomes than individuals achieve alone, but self-interested actors may struggle to bear personal costs for a greater good.Unlike top-down incentive-compatible coordination, people can develop decentralized norms for sustainable resource management.Source: §2.1 · p. 3

2.2 GOVSIM Description

  • GOVSIM models shared-resource dilemmas in fisheries, pasture, and pollution, where agents balance current use against resource regeneration and negotiate fair shares.The scenarios use mathematically equivalent regeneration dynamics.Source: §2.2 · pp. 3–4, §2.3 · p. 4

2.3 GOVSIM Environment Dynamics

  • Agents harvest privately and simultaneously, then discuss their choices; resources regenerate up to capacity, but collapse when the stock falls below C = 5.The simulation frames interaction as a partially observable Markov game interleaving actions, observations, rewards, and unstructured dialogue.Source: §2.3 · p. 4
  • The sustainability threshold f(t) is the maximum extraction that leaves the resource stock undiminished at the next step, given future growth multiplier g.The environment is designed to evaluate cooperative behavior and governance of shared resources.Source: §2.2 · p. 3, §2.3 · p. 4

2.4 GOVSIM Metrics

  • GOVSIM tracks survival time and survival rate, efficiency relative to maximum possible resource use, inequality in agent gains, and actions exceeding the sustainability threshold.The authors note that no single scalar captures the full state of a mixed-incentive repeated game.Source: §2.4 · pp. 4–5

3 Experimental Results

Across GOVSIM’s three scenarios, larger models performed better, but no LLM sustained resources in all five seeds. Further experiments test newcomer robustness, Universalization, communication, dialogue patterns, and links between reasoning skills and survival.

3.1 Experimental Setup

  • Agents use a phase-based generative-agents architecture, benchmarked across 15 instruction-tuned open- and closed-weight LLMs with greedy decoding over five random seeds.The architecture requires different decisions across phases, including resource extraction and open-ended discussion.Source: §3.1 · p. 5

3.2 Benchmarking GOVSIM

  • Larger models such as GPT-4o showed better survival time and total gain, but no LLM sustained resources across all five seeds and three scenarios.Table 1 aggregates results across the three scenarios and five runs.Source: §3.2 · p. 5, Table 1 · p. 6

3.3 Norm Robustness: A Greedy Newcomer

  • Introducing a selfish newcomer reduced survival rate from 53.3 to 33.3 and survival time from 9.3 to 6.6, while community discussions aligned the newcomer with sustainable harvesting.Total gain fell from 66.0 to 34.8, efficiency from 55.0 to 31.3, and equality from 94.4 to 71.7; over-usage rose from 10.8 to 15.7.Source: §3.3 · p. 6

3.6 Analysis of Agent Dialogues

  • 54% of utterances were negotiations, 45% information, and 1% relational, while excessive harvesting drew concern from other agents.The dialogue analysis covered models with higher survival time.Source: Figure 4 · p. 7, §3.6 · p. 8

3.4 Improving Sustainability by Universalization Reasoning

  • Universalization significantly increased average survival time by 4 months, total gain by 29 units of shared resource, and efficiency by 24%.Claude-3 Opus was excluded because of API costs; two combinations already at maximum survival time were excluded from the significant increases.Source: §3.4 · p. 7

3.5 Ablation of Communication

  • Without communication, tested higher-survival models overused common resources by 22%; discussions usually coordinated extraction limits at or below the sustainability threshold.The communication ablation covered GPT-4o, GPT-4-turbo, Claude-3 Opus, and Qwen-110B.Source: §3.5 · p. 7

3.7 The Role of LLM Capabilities

  • 30% was the maximum rate at which LLMs chose sustainable harvesting in isolation, while all four reasoning-test accuracies correlated significantly with survival time.Belief-based threshold reasoning was associated with higher survival than assumption-based reasoning; R2 values were 0.69 for simulation dynamics, 0.92 for sustainable action, 0.76 for sustainability threshold (assumption), and 0.82 for sustainability threshold (belief).Source: Figure 5 · p. 8, §3.7 · p. 9

4 Contributions in the Context of Related Work

Existing safety benchmarks and multi-agent studies leave limited evidence about cooperation in dynamic, realistic settings. GOVSIM addresses this with resource-sharing scenarios that require strategic communication and negotiation, extending beyond single-agent safety tests and one-shot game-theoretic benchmarks.

4 Contributions in the Context of Related Work

  • Prior multi-agent studies emphasize constrained tasks, while popular safety benchmarks do not address multi-agent interaction or broader real-world scenarios.Static benchmarks also miss the flexible, interactive tasks involved in navigating real-world scenarios.Source: §4 · p. 9
  • GOVSIM evaluates agents in three resource-sharing scenarios, examining sustainability, stability, and conflict resolution in a game-theoretic setting.Success is grounded in the game-theoretic scenario rather than relative to human annotators.Source: §4 · p. 9
  • Unlike one-shot game-theoretic tasks such as GTBench Duan et al. [19], GOVSIM tests long-horizon economic cooperation involving moral considerations, communication, and negotiation.The benchmark focuses on cooperation dilemmas and broader forms of economic reasoning.Source: §4 · p. 9

5 Limitations and Future Work

The study’s scenarios simplify real-world resource management to balance complexity and interpretability, while leaving several extensions open. These include broader resource dynamics, larger and more diverse populations, coordinated adaptation, exceptional trade-offs, and stronger negotiation and robustness tests.

5 Limitations and Future Work

  • Real-world common-pool management has more sophisticated dynamics and variability than the study’s simplified resource-sharing scenarios.Potential extensions include varying regeneration rates, multiple resource types, and different stakeholder interests.Source: §5 · p. 9
  • Scenario simplicity supports systematic study of cooperative principles, but the current environment remains less realistic than real-world resource management.The design balances complexity and interpretability; its open-ended scenarios are more realistic than highly simplified behavioral-game paradigms.Source: §5 · p. 9
  • Adding more agents and diverse player types is a possible extension, but it would increase simulation runtime.Each agent must condition its behavior and dialogue on the other agents’ actions and dialogues.Source: §5 · p. 10
  • GOVSIM’s modularity permits experiments on coordinated adaptation to shocks or changes in resource reproduction rates.These cases test whether agents can rapidly adjust cooperative norms together.Source: §5 · p. 10
  • Future work could test whether agents accommodate one-off serious personal harm without enabling exploitation.The example is an agent taking more resources than an existing norm permits to avoid harm.Source: §5 · p. 10
  • Current LLM capabilities limit negotiation and strategic behavior, and future work could test adversarial robustness, larger populations, and mixed human-AI communities.The authors also identify improved negotiation skills and human-AI interaction as directions for further study.Source: §5 · p. 10

6 Conclusion

GOVSIM studies strategic interaction and cooperation in shared-resource management. Most tested LLM agents failed to sustain an equilibrium, while communication and universalization improved outcomes.

6 Conclusion

  • All but the most powerful LLM agents failed to achieve a sustainable equilibrium, with the highest survival rate below 54%.Without communication, agents overused the shared resource by 22%; universalization-based prompting improved survival time by 4 months.Source: §6 · p. 10
  • The authors suggest that unsustainable governance may stem from agents’ inability to mentally simulate how greedy actions affect the group’s long-term equilibrium.They also report that communicating agents find ways to cooperate flexibly and avoid collapse.Source: §6 · p. 10

A Ethical Considerations

  • The paper presents GOVSIM as a benchmark for evaluating existing models, not as training to improve deception or bluffing.It emphasizes that simulation should augment, rather than substitute for, human judgment and ethical consideration.Source: A Ethical Considerations · p. 16

B.1 Environment

  • GOVSIM models resource-sharing through strategy, harvesting, and group discussion, with an environment managing simulation dynamics and an LLM-based agent interacting with it.Harvesting is simultaneous: the environment allocates resources after agents submit their choices, avoiding effects from harvesting order.Source: §B.1 · p. 16, Figure 6 · p. 16

B.2 Agent

  • The phase-based agent adapts a generative-agents architecture to goal-oriented numerical reasoning and extends its conversation module for moderator-orchestrated group discussion.Prompts provide shared simulation rules and scenario-specific tasks.Source: §B.2 · p. 17

B.3 Web Interface

  • GOVSIM’s web interface links run statistics to specific events so researchers can inspect the prompts and responses behind agent actions.The interface also displays the prompts and conversations generated by the LLM.Source: §B.3 · p. 17, Figure 7 · p. 17

C Generative Agents Prompts

D.1 How to Reproduce the Experiments?

The appendix describes how to reproduce the experiments and identifies the scope of the default and universalization results tables across three scenarios.

D.1 How to Reproduce the Experiments?

  • The appendix reports approximate compute needs for open-weight models and API costs for closed-weight models.A complete 12-month open-weight simulation takes approximately 4 hours; the reported closed-model API spend was 1500 USD.Source: §D.1 · p. 22
  • Experiments used five random seeds and temperature-zero decoding, although inference kernels and external APIs could still introduce randomness.The authors provide code and configurations for reproducing the experiments.Source: §D.1 · p. 22

D.2.1 Fishery

  • The default experiment tables report fishery, pasture, and pollution results, while accompanying figures track end-of-month fish, grass, and unpolluted water by model family.The supplied captions specify the measures and grouping but do not provide the outcome values.Source: Table 4 · p. 23, Figure 8 · p. 23, Table 5 · p. 24

D.3.1 Fishery

  • The universalization tables compare evaluation-metric improvements against default results for fishery, pasture, and pollution.The supplied captions identify the comparisons but do not state the improvement values.Source: Table 7 · p. 26, Table 9 · p. 27, Table 11 · p. 28

D.4 Experiment Newcomer

The newcomer experiment contrasts a locally rooted villager role with an outsider seeking to expand a fishing business, then compares outcomes across three resource scenarios.

D.4 Experiment Newcomer

  • The villager role emphasizes learning to fish from family and caring about the village, while the newcomer seeks business growth and is willing to break rules for profit.The newcomer prompt says the agent is unconcerned if the lake has no fish in the future.Source: §D.4 · p. 29
  • The fishery newcomer table compares the newcomer experiment’s results.Its caption does not specify the comparison’s outcome.Source: Table 13 · p. 29
  • Separate newcomer comparison tables cover pasture and pollution scenarios.The supplied captions do not report which conditions performed better.Source: Table 14 · p. 29, Table 15 · p. 29

D.5 Language Ablation

The language ablation compares simulations with and without communication; without it, agents tend toward worse reported outcomes, though the evidence varies by metric.

D.5 Language Ablation

  • Without communication, efficiency was lower by 4 (p < 0.398), equality by 4% (p < 0.001), gain by 4 (p < 0.398), and survival time by 1 (p < 0.109).The tables cover over-usage comparisons for fishery, pasture, and pollution, but the supplied captions do not give their values.Source: §D.5 · p. 30, Table 16 · p. 30, Table 17 · p. 30

D.5.1 Fishery

  • The fishery table compares over-usage percentages with and without communication and marks each model’s best metric in bold.The caption identifies the comparison but does not state its outcome.Source: Table 16 · p. 30

D.5.2 Pasture

  • The pasture table compares over-usage percentages between simulations with and without communication.The supplied caption does not state which condition has lower over-usage.Source: Table 17 · p. 30

D.5.3 Pollution

  • The pollution table compares over-usage percentages between simulations with and without communication.The supplied caption does not state which condition has lower over-usage.Source: Table 18 · p. 30

E Analysis of Agent Dialogues

The dialogue analysis classifies utterances into eight subcategories, then groups them into three main categories. Tables report category proportions and standard deviations across models in the fishery, pasture, and pollution scenarios.

E Analysis of Agent Dialogues

  • Utterances are classified into eight subcategories and grouped into three main categories.The classification prompt supplies a taxonomy covering information sharing, problem identification, solution proposing, persuasion, consensus seeking, disagreement, excusing behavior, and punishment.Source: E Analysis of Agent Dialogues · p. 31
  • Tables 19–21 report mean proportions and standard deviations for Information Sharing, Negotiation, and Relational utterances across models in Fishery, Pasture, and Pollution.The tables cover one scenario each; the supplied passages do not give their numerical values.Source: Table 19 · p. 31, Table 20 · p. 31, Table 21 · p. 32

F Sub-skills Evaluation

The sub-skills evaluation tests reasoning abilities intended to help identify what contributes to high survival time in resource-sharing simulations. Models are tested across four skills, and their accuracies are plotted against survival duration.

F Sub-skills Evaluation

  • Four tests measure simulation-dynamics reasoning, sustainable action without interaction, threshold calculation assuming equal harvest, and threshold calculation based on beliefs about other agents.The threshold tasks ask agents to reason about what harvest can sustain the resource, with or without the equal-harvest assumption.Source: F Sub-skills Evaluation · p. 32
  • Each test prompt is run 150 times with varying values, and accuracy is computed for every run across the models described in Appendix D.1.Results are summarized with means and 2-sigma confidence intervals; scatter plots relate test accuracy to survival duration.Source: F Sub-skills Evaluation · p. 32

F.1 Method

The method evaluates four sub-skills using controlled resource-sharing scenarios and prompts. Tests progress from basic resource calculations to sustainable choices and threshold estimates under different assumptions about other agents.

F.1 Method

  • Each scenario uses its simulation description as common information, but limits memory to the shared-resource amount before harvesting.The common information is tailored to fishery, pasture, and pollution cases.Source: §F.1 · pp. 32–33
  • In the simulation-dynamics test, models calculate the next-step resource quantity from the current state, assuming equal harvest by agents.The test varies initial resource quantity N from 10 to 100 and per-agent harvest M; fishery, pasture, and pollution prompts instantiate this calculation.Source: §F.1 · p. 33
  • In the sustainable-action test, models determine how much of a shared resource to collect without interacting with other agents; an answer is correct when it falls between zero and the sustainable threshold.Examples use fish catch and sheep flocks; initial resource quantity ranges from 10 to 100.Source: §F.1 · pp. 33–34
  • The assumption-based threshold test asks models to calculate a sustainable amount while assuming all agents harvest equally.Prompts ask for a maximum per-agent harvest that preserves the resource after its population doubles; answers are scored against the threshold.Source: §F.1 · p. 34
  • The belief-based threshold test asks models to calculate a sustainable amount without stating an equal-harvest assumption, probing their ability to form assumptions about other agents.As in the assumption-based test, answers are scored against the sustainable threshold.Source: §F.1 · pp. 34–35

F.2 Results

The results section presents accuracy tables and plots relating reasoning-test scores to survival time. Its figures cover pooled results and the fishery, pasture, and pollution scenarios, but the supplied passages do not state the observed comparisons.

F.2 Results

  • Figure 11 plots average reasoning-test accuracy against average survival time across four test cases; higher survival time indicates a better score.The supplied caption describes the axes but does not report the correlation's direction or strength.Source: Figure 11 · p. 35

F.2.1 Fishery

  • The Fishery results include a scatter plot of reasoning-test scores versus average survival time and an accuracy table for its sub-skills test cases.The supplied captions identify the plotted variables and table scope without giving outcome values.Source: Figure 12 · p. 35, Table 22 · p. 36

F.2.2 Pasture

  • The Pasture results include a scatter plot of reasoning-test scores versus average survival time and an accuracy table for its sub-skills test cases.The supplied captions identify the plotted variables and table scope without giving outcome values.Source: Figure 13 · p. 37, Table 23 · p. 37

F.2.3 Pollution

  • The Pollution results include a scatter plot of reasoning-test scores versus average survival time and an accuracy table for its sub-skills test cases.The supplied captions identify the plotted variables and table scope without giving outcome values.Source: Figure 14 · p. 38, Table 24 · p. 38

G Conversation Examples

The conversation examples show framework-generated and LLM-generated utterances, including an outsider joining the community. The checklist records the paper’s reproducibility, experimental-detail, and impact disclosures.

G Conversation Examples

  • Figure 15 presents an excerpt of a conversation in the fish-default experiment, with Major generated by the framework and John’s and Kate’s utterances generated by a GPT-4 LLM agent.Source: Figure 15 · p. 39

NeurIPS Paper Checklist

  • The checklist states that the paper discusses limitations in Section 5 and that the code is documented.Source: NeurIPS Paper Checklist · pp. 40, 45
  • The authors say their code and data were uploaded with the submission and would be open-sourced upon acceptance; they use publicly available Hugging Face models or public APIs.Source: NeurIPS Paper Checklist · pp. 41–42
  • Prompts and main architecture details are described in Appendices B–D and F, while compute-resource details are referenced in Appendix D.1.Source: NeurIPS Paper Checklist · pp. 42–43
  • Simulation experiments report standard deviations across five seed runs, and subskill evaluations report 2-sigma confidence intervals.Source: NeurIPS Paper Checklist · p. 42
  • The checklist says the paper measures current LLM capabilities as a benchmark and discusses ethical considerations in Appendix A.Source: NeurIPS Paper Checklist · p. 43
Loading 2404.16698v4…