Open Problems in Cooperative AI

Purpose and scope

• Cooperative AI focuses AI research on helping individuals, humans and machines, find ways to improve their joint welfare and on improving groups’ ability to cooperate.The proposal contrasts this emphasis on social intelligence with AI research focused primarily on individual intelligence and algorithms.Source: §1 · pp. 3–4

Research program

• The field should connect multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and social tools through focused cooperation-oriented research conversations.This integration is intended to support a unified theory and vocabulary and stronger links with natural, social, and behavioural sciences.Source: §1 · p. 3
• Cooperative AI research should develop training environments, tasks, and domains where cooperative capabilities are crucial, non-trivial, learnable, and measurable.The broader agenda should also address common-interest and mixed-motive settings, including trust, deception, commitment, human preferences, norms, and ethics.Source: §1 · pp. 3–4

Purpose and scope

• The proposal recommends a technically precise, problem-defined scope to distinguish Cooperative AI from adjacent concepts such as aligned, trustworthy, and beneficial AI.For a given situation and set of agents, the target problem is framed as relatively well defined and unambiguous.Source: §1 · p. 4

Cooperative capabilities

• Core cooperative capabilities include understanding other agents, communicating, forming commitments, and creating institutions that promote cooperation.These capabilities span beliefs, incentives, capabilities, shared language, mistrust, deception, norms, and legal systems.Source: §1 · p. 4

Potential downsides

• Research on Cooperative AI should investigate risks because cooperative competence can enable exclusion, coercion, and competition, and should develop ways to anticipate and mitigate these downsides.The proposal treats understanding and mitigating these risks as an important part of the research agenda.Source: §1 · p. 4

2.1. Vignettes: Self-Driving Vehicles and COVID-19

• Self-driving vehicles illustrate cooperation problems involving prediction, communication, credible commitments, and institutions that could improve welfare among many human and machine drivers.Relevant mechanisms include modeling others’ goals, beliefs, capabilities, conventions, and intentions; communicating hazards and coordination signals; and designing congestion pricing, information sharing, and incentive-compatible policies.Source: §2.1 · pp. 4–5
• Pandemic preparedness shows how cooperation failures can cause millions of deaths and tens of trillions of dollars in damage, despite gains from vaccines, surveillance, data sharing, and coordination.Achieving these gains requires fair investment agreements, sensitive communication, reliable commitments during crises, and institutions balancing conflicts of interest, legitimacy, and technical competence.Source: §2.1 · p. 5
• Cooperation problems are ubiquitous and diverse, spanning interpersonal to interstate settings, two agents to millions, small to global stakes, and humans, machines, and organizations.Cooperative competence therefore requires increasingly general social intelligence for dynamic problems, while AI systems must participate cooperatively in a shared civilization of humans, organizations, and machines.Source: §2.1 · pp. 5–6
• Cooperation is a fundamental scientific problem across biology and the social sciences, so framing AI research around it could improve cross-disciplinary consilience through shared vocabulary and exchange.The authors recommend adding cooperation and Cooperative AI to directed AI programs such as safety [AOS+16], fairness [HPS16], interpretability, and robustness.Source: §2.1 · p. 6

2.2. Prior Work, Machine Learning, and Timeliness

• Cooperative AI draws on decades of multi-agent research covering distributed interaction, markets, negotiation, institutions, learning, communication, social choice, and multi-agent robotics.Prior work evolved from simulations combining game theory and emergence theories toward architectures for distributed problems and agents that learn to communicate and collaborate.Source: §2.2 · p. 7
• Recent advances in learning from images, language, and reinforcement learning make it timely to reinvigorate Cooperative AI, while mixed-motive research addresses cooperation beyond rivalrous games.Earlier work often favored tractable two-player zero-sum games, but current directions include common-interest, team, and mixed-motive settings involving trust and commitment; early coordination also matters before AI deployments lock in suboptimal equilibria.Source: §2.2 · pp. 7–8

3. Cooperative Opportunities

• The section organizes cooperative opportunities along four dimensions: common versus conflicting interests, agent types, individual versus planner perspectives, and Cooperative AI’s relationship to related fields.These dimensions structure the character of cooperation problems and the associated research.Source: §3 · p. 8

3.1. Common and Conflicting Interests

• Cooperation is possible in pure common-interest and mixed-motive settings, while pure conflicting-interest games are the only cases where cooperation is impossible.Pure conflicting-interest situations are rare in strategic games and likely rare in the real world, despite receiving disproportionate machine-learning attention.Source: §3.1 · pp. 10–11

3.2. Who is Cooperating: Humans, Machines, Organizations

• Cooperation dynamics differ across humans, machines, organizations, and hybrid groups, requiring attention to cognition, preferences, communication, culture, and strategic behavior.Artificial agents may produce solutions unlike those observed in human communities, while human-machine cooperation must account for human behavior such as assertiveness and cultural context.Source: §3.2 · pp. 11–12

3.3. The Individual Perspective and the Planner Perspective

• Cooperative AI distinguishes individual and planner perspectives: individuals improve local cooperative capabilities, while planners shape population welfare through policies, institutions, and centralized mechanisms.Understanding both perspectives can help planners intervene effectively and help individuals identify group-level changes that move populations toward the Pareto frontier.Source: §3.3 · p. 12

3.4. Scope

• Cooperative AI bets on an overarching compass, unified vocabulary, and deliberate cross-community conversations to connect AI research with the broader science of cooperation.It is not simply the union of existing fields, but a focus on productive conversations spanning AI and adjacent disciplines.Source: §3.4 · p. 13
• Cooperative AI emphasizes horizontal coordination among multiple principals rather than treating alignment and control in vertical principal-agent relationships as its primary focus.It therefore directs attention toward bargaining, credible communication, trust, and commitment, while treating aligned human-machine dyads as pure common-interest cases.Source: §3.4 · p. 13

Understanding

• Understanding other agents’ payoffs, beliefs, capabilities, and intentions can prevent mutually harmful choices and enable agents to select a mutual best outcome.In the one-sided assurance game, Row may choose D from uncertainty, but learning Column’s preferences leads both to choose C.Source: §4 · p. 14, Table 1 · p. 14

Communication

• In games with multiple equilibria, communication can coordinate intentions and beliefs by making an announcement self-committing when it induces the partner’s best response.In Stag Hunt, saying “Stag” can lead Column to believe Row intends Stag and choose it as well.Source: §4 · p. 14, Table 2 · p. 14
• Conflicting incentives can make communication unreliable, because agents may misrepresent, ignore, or pretend not to understand others’ intentions.In Chicken, cheap talk may fail to prevent collision even when players have abundant opportunity to communicate.Source: §4 · p. 14, Table 3 · p. 14

Commitment

• Credible commitments can resolve coordination failures and sustain cooperation by constraining future behavior or making conditional actions believable.Examples include physical constraints, binding contracts, sunk costs, repeated interaction, and reputational collateral.Source: §4 · pp. 14–15

Institutions

• Institutions structure behavior through conventions, norms, and formal rules, selecting cooperative equilibria or changing games and payoffs.In the Prisoner’s Dilemma, infinitely repeated play with discounting supports cooperative subgame-perfect equilibria such as tit-for-tat; Fearon calls institutional design the harder and more important cooperation problem.Source: §4 · p. 15

4.1. Understanding

• Understanding includes predicting action consequences, other agents’ behavior, and contributing factors such as beliefs and preferences, whether predictions are explicit or implicit.This framework includes model-based reasoning, search, adaptation, evolutionary selection, and model-free reinforcement learning.Source: §4.1 · p. 15

4.1.1. The World

• Agents can improve cooperation by inferring private information, but without communication or common interest they may need costly signals; trusted AI advisors could support joint learning.Credibly sharing private information can reveal whether a joint venture is mutually beneficial, while costly-to-fake actions can communicate when utterances are untrustworthy.Source: §4.1.1 · p. 16

4.1.2. Behaviour

• Cooperation requires anticipating others’ strategies, but moving to a new cooperative equilibrium demands deeper understanding than sustaining an established one.Model-free methods can discover coordination through experience, whereas model-based reasoning can identify larger jumps in strategy space while incurring model-learning costs and error risks.Source: §4.1.2 · pp. 16–17

4.1.3. Preferences

• Understanding preferences is difficult because they can be computationally intractable, inaccessible even to their holders, or strategically misrepresented, motivating preference-learning research for AI alignment.Relevant approaches include inverse reinforcement learning, cooperative IRL [HMRAD16], and assistance games.Source: §4.1.3 · p. 18

4.1.4. Recursive Beliefs

• Recursive beliefs and common knowledge support coordination, but deeper reasoning can also undermine cooperation by enabling backward-induction defection or overriding intuitive cooperation.Public-goods participants contributed more under time pressure, suggesting deliberation can reduce heuristic cooperation.Source: §4.1.4 · pp. 18–19

4.2. Communication

• Explicit communication helps agents infer intentions and preferences, coordinate division of labour, and discover, converge on, and maintain Pareto-optimal equilibria.Compressed abstractions can make cooperation more efficient and flexible than transmitting low-level instructions.Source: §4.2 · p. 19

4.2.2. Bandwidth and Latency

• Limited bandwidth requires context-dependent compression, whereas high latency increases the need for autonomous behavior, planning, and prediction in cooperative systems.Emergent communication can support machine cooperation under minimized bandwidth, but high-latency settings such as Mars rover control require high-level guidance rather than low-level direction.Source: §4.2 · p. 19, §4.2.2 · p. 21

4.2.3. Teaching

• Teaching and learning from others expand opportunities for joint gains, but effective teaching requires suitable curricula, theory of mind, and feedback despite delayed evaluation.Teaching machines to instruct humans also requires interface design, pedagogy, summarization, and knowledge distillation beyond interpretability [SHS+17].Source: §4.2 · p. 19, §4.2.3 · p. 22

4.2.1. Common Ground

• Common ground—shared background knowledge, vocabulary, or representations—is necessary for meaningful communication and requires an initial investment that reduces later communication costs.Its optimal complexity depends on fixed construction costs and variable use costs, while developing human-machine common ground remains a major challenge involving language, games, and preference elicitation.Source: §4.2.1 · pp. 20–21

4.2.4. Mixed Motives

• Under mixed motives, communication faces incentives to deceive, but incentive-compatible mechanisms, structured transparency, privacy-preserving learning, and negotiation can support cooperative outcomes.Costly signals can make communication credible, while automated mechanism design and cryptographic information architectures can promote truthful revelation and privacy-preserving cooperation.Source: §4.2.4 · p. 23

4.3. Commitment

• Commitment problems can cause cooperation failure independently of informational problems, including in the perfectly informed Prisoner’s Dilemma where conditional commitment can sustain mutual cooperation.Informational and commitment problems are often intertwined in practice, and their solutions can be substitutes.Source: §4.3 · p. 24, §4.3.1 · p. 24

4.3.1. Commitment Solutions: Unilateral vs Multilateral; Unconditional vs Conditional

• Commitment devices compel promised behavior by changing incentives or pruning actions, and may be unilateral or multilateral, unconditional or conditional.Conditional commitments are harder to construct but can support precise promises or threats; multilateral devices include legal contracts.Source: §4.3.1 · pp. 24–25
• Cooperative AI can develop commitment-contract languages, optimize commitments, predict responses, and identify welfare-improving commitment changes in domains such as disarmament.This work benefits from interdisciplinary research spanning law, politics, economics, psychology, and physics.Source: §4.3.1 · p. 25

4.3.2. Devices: Reputation, Delegation, Contracts, Hardware

• Reputation, delegation, contracts, and hardware provide distinct commitment mechanisms, while cryptographic protocols, trusted hardware, and smart contracts enable sophisticated conditional commitments without central authorities.Reputation can collateralize cooperation in transient encounters, and social planners or central authorities can mediate decisions and enforce contracts.Source: §4.3.2 · pp. 25–26

4.4. Institutions

• Institutions support cooperation by shaping permissible actions and outcomes, resolving coordination problems, aligning incentives, and scaffolding complex inter-agent behaviours.They range from conventions and norms to thicker entities that allocate roles, responsibilities, power, and resources.Source: §4.4 · p. 26

4.4.1. Decentralized Institutions and Norms

• Decentralized institutions emerge through agent interactions without a trusted central authority, with norms constraining behaviour through expectations, sanctions, and equilibria.Open problems include designing distributed institutions, predicting effective norms, handling non-compliance, and generalizing norms across circumstances and human-machine groups.Source: §4.4.1 · p. 27

4.4.2. Centralized Institutions

• Centralized institutions use authority, social choice, and mechanism design to aggregate preferences, address fairness, and align self-interested agents’ incentives with socially desirable outcomes.Outstanding directions include combining axiomatic and data-driven methods, designing mechanisms for welfare and fairness, and exploring institutional structures beyond auctions for complex coordination problems.Source: §4.4.2 · pp. 28–29

4.4.3. Trust and Reputation Systems

• Trust and reputation systems align incentives by making past behaviour informative and reputation valuable as collateral, supporting cooperation, coordination, community formation, and human-machine acceptance.Further research is needed to understand the relative benefits of different reputation mechanisms.Source: §4.4.3 · p. 29

4.4.4. Cooperative AI Problems and Institutions

• Cooperative AI could use machine learning for institutional design, allowing participants to specify desiderata while AI discovers mechanisms that incorporate richer information, including natural-language opinions.Such methods may improve performance while sacrificing parsimony, interpretability, and closed-form provability, raising questions about transparency and accountability in social systems.Source: §4.4.4 · p. 30

5.1. Exclusion and Collusion

• Cooperative competence can harm excluded groups and undermine pro-social competition through collusion, creating an open question about which forms of cooperation produce net social benefit.Examples include criminal cooperation, rent seeking, auction collusion, and AI-enabled price coordination without explicit communication; social planners may need to incentivize beneficial forms of cooperation.Source: §5.1 · pp. 30–31

5.2. Coercive Capabilities

• Capabilities useful for cooperation—including understanding, honest communication, deception detection, commitment, and institutional design—can also improve coercion, threats, manipulation, and obedience.Coercion can transfer value through threats or force without guaranteeing welfare improvements, unlike cooperation within the cooperating set.Source: §5.2 · p. 31

5.3. The Role of Coercion and Competition in Cooperation

• Coercion and competition are often integral to cooperation: punishment sustains cooperation, while inter-group competition drives learning and evolutionary and cultural transitions, making capabilities difficult to separate.Legal contracts illustrate cooperative punishment, and strong competitive pressures may train cooperative skills alongside coercive and competitive skills.Source: §5.3 · p. 31

5.4. Understanding and Mitigating the Downsides

• The section proposes the Hypothesis that Broad Cooperative Competence is Beneficial: large, broadly distributed increases in cooperative competence tend to be broadly welfare improving on net.The rationale is that first-order cooperative gains may outweigh second-order exclusion harms, mutual coercive gains may cancel through balance or deterrence, and cooperation can be reinforcing.Source: §5.4 · p. 32
• Mitigation research should seek cooperation-biased capabilities and evaluate how to separate cooperative gains from coercive ones through communication, trusted institutions, multilateral commitments, and tests of AI deception and threats.The section highlights cheap talk, trusted mediators, reputation systems, verifiable hardware, norms against lying, and legal contracts as possible cooperation-biased mechanisms, while calling for broader theories linking welfare to power and capability distributions.Source: §5.4 · pp. 32–33

• Cooperation shaped major evolutionary transitions, underpins human history and well-being, and becomes harder as cooperative achievements grow.These features make cooperation an attractive target for research on intelligence.Source: §6 · p. 33
• AI can provide scientific tools for understanding social systems and devising cooperative structures, while its deployment must promote human cooperation.Source: §6 · p. 33
• Cooperative AI spans intersections with biology, sociology, psychology, anthropology, economics, history, international relations, and computer science.Its success therefore depends on bridging disciplines, developing a common vocabulary, and agreeing on achievable cooperative goals.Source: §6 · p. 33
• As AI pursues increasingly capable machine agents, research must attend to the intelligence humanity most needs, including cooperative intelligence.Source: §6 · p. 33

Aims and Focus

• Cooperative AI would study cooperation as a distinct AI research focus, building agents capable of cooperation and using mechanism design and mediation to foster cooperation in human and machine populations.The agenda spans cooperation problems from everyday interactions to global challenges and motivates AI research because machines increasingly affect human lives.Source: §8 · p. 34
• The proposed capability agenda includes understanding, communicating, committing, bargaining, and institution-building, supported by challenging training environments where cooperative skills are learnable and essential.Relevant fields include reinforcement learning, multi-agent systems, game theory, mechanism design, social choice, language learning, interpretability, trusted hardware, and cryptography.Source: §8 · p. 34
• Cooperative AI must account for human preferences, norms, and ethics because agents often act for humans and can produce consequential outcomes.The agenda also calls for studying exclusion, collusion, and coercion, while directing cooperative capabilities toward improving human welfare.Source: §8 · p. 34
• The workshop planned to convene scholars from diverse backgrounds to examine how AI research could contribute to the broader scientific and social study of cooperation.This reflects the agenda’s intended connection between machine learning, natural and social sciences, and efforts to solve coordination problems.Source: §8 · p. 34