Source-linked AI summary
Open Problems in Cooperative AI
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, Thore Graepel
TL;DR
Cooperation problems are ubiquitous and important, yet agents often cannot easily realize available joint welfare gains. The paper proposes Cooperative AI as a focused research program spanning AI and related disciplines, and concludes that cooperative capabilities should be developed alongside attention to their entanglement with coercion.
Problem
AI research lacks an explicitly focused, problem-defined program for helping agents improve joint welfare through cooperation.
Method
The paper organizes Cooperative AI around cooperative opportunities, capabilities, cross-disciplinary conversations, and tools for agents and populations.
Results
The paper concludes that AI can contribute scientific tools, social mechanisms, infrastructure, and agents relevant to understanding and promoting cooperation.
Takeaways & Limitations
Cooperative AI should connect AI research with broader natural, social, and behavioural science research on cooperation.
Takeaways & Limitations
Cooperative capabilities can also support coercion, and coercive mechanisms may be intertwined with cooperation, including through punishment and contracts.
Abstract
from arXiv · showhide
Problems of cooperation--in which agents seek ways to jointly improve their welfare--are ubiquitous and important. They can be found at scales ranging from our daily routines--such as driving on highways, scheduling meetings, and working collaboratively--to our global challenges--such as peace, commerce, and pandemic preparedness. Arguably, the success of the human species is rooted in our ability to cooperate. Since machines powered by artificial intelligence are playing an ever greater role in our lives, it will be important to equip them with the capabilities necessary to cooperate and to foster cooperation. We see an opportunity for the field of artificial intelligence to explicitly focus effort on this class of problems, which we term Cooperative AI. The objective of this research would be to study the many aspects of the problems of cooperation and to innovate in AI to contribute to solving these problems. Central goals include building machine agents with the capabilities needed for cooperation, building tools to foster cooperation in populations of (machine and/or human) agents, and otherwise conducting AI research for insight relevant to problems of cooperation. This research integrates ongoing work on multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and the construction of social tools and platforms. However, Cooperative AI is not the union of these existing areas, but rather an independent bet about the productivity of specific kinds of conversations that involve these and other areas. We see opportunity to more explicitly focus on the problem of cooperation, to construct unified theory and vocabulary, and to build bridges with adjacent communities working on cooperation, including in the natural, social, and behavioural sciences.
1. Introduction
Cooperative AI frames cooperation as a distinct AI research problem: helping agents improve joint welfare and developing the social intelligence needed for effective cooperation. It proposes focused, cross-disciplinary research organized around cooperative opportunities, capabilities, and potential downsides.
- The field shifts attention beyond individual intelligence toward groups’ ability to cooperate in solving shared problems.
- Research should connect multi-agent systems, game theory, social choice, alignment, natural-language processing, and social tools through focused conversations on cooperation.
- Cooperative AI studies how individuals, humans, and machines can find ways to improve their joint welfare.
- Cooperative opportunities can be analyzed by strategic context, interests, participating entities, and whether the focus is individual competence or social planning.
- Core cooperative capabilities include understanding, communication, commitments, and institutions such as norms and legal systems.
- Cooperative AI research should also investigate how cooperative competence can exclude others, enable coercion, or remain difficult to disentangle from competition.
2. Why Cooperative AI?
Cooperation problems span everyday interactions and global challenges, involving diverse agents, scales, stakes, and institutional settings. Cooperative AI is motivated by opportunities to use AI tools, agents, and scientific insights to understand and improve these problems while connecting AI with broader cooperation research.
- Vignettes: Cooperation problems range from self-driving vehicles and interpersonal interactions to pandemic preparedness, conflict, commerce, and other global challenges.
- Self-Driving Vehicles: Self-driving vehicles require agents to understand other drivers’ goals, beliefs, capabilities, and local driving conventions.
- Self-Driving Vehicles: Vehicle cooperation also depends on communication, credible commitments, and institutions that can improve outcomes while preserving participation incentives, fair pricing, and privacy safeguards.
- Pandemic Preparedness: Pandemic preparedness may require agreement on investment, reliable communication about outbreaks and medical systems, and solutions to commitment problems.
- Commonalities: Cooperation problems vary by scale, number and type of agents, stakes, and the clarity of interests, norms, and institutions.
- Broader Motivation: AI can contribute tools, infrastructure, agents, and scientific models that facilitate cooperation and connect AI research with natural, social, and behavioural sciences.
3. Cooperative Opportunities
Cooperative opportunities span different agents, incentive structures, and perspectives, from human and machine interactions to population-level institutions. The paper proposes Cooperative AI as a focused, cross-disciplinary research program organized around these dimensions.
- Cooperative opportunities vary by common versus conflicting interests, the kinds of agents involved, and whether research takes an individual or social-planner perspective.
- 3.1. Common and Conflicting Interests: Pure common-interest, mixed-motive, and pure-conflict games form a spectrum, with cooperation possible in principle except under pure conflicting interest.
- 3.1. Common and Conflicting Interests: In a simple class of two-player, two-strategy games, there are 144 distinct games, and the overwhelming majority contain some common interest.
- 3.1. Common and Conflicting Interests: Multi-agent research has focused heavily on pure-conflict environments, although techniques from these settings can also inform mixed-motive cooperation.
- 3.4. Scope: Cooperative AI is framed as a bet that unified theory, vocabulary, and deliberate cross-community conversations can connect disparate research threads around cooperation.
- 3.4. Scope: The field differs from alignment and human-machine interaction by emphasizing horizontal coordination among multiple principals rather than vertical coordination with a human principal.
4. Cooperative Capabilities
The paper organizes cooperative capabilities around understanding, communication, commitments, and institutions. Strategic-game examples show how these capabilities address uncertainty, coordination failures, conflicting incentives, and equilibrium selection.
- Understanding: In assurance games, uncertainty about another player’s payoffs can prevent selection of the mutual best outcome despite its being the unique equilibrium.
- Understanding: Understanding payoffs, beliefs, capabilities, and intentions helps agents predict behavior and identify mutually beneficial outcomes.
- Communication: Communication can coordinate intentions and beliefs in games with multiple equilibria, as a self-committing announcement can make the efficient action a best response.
- Communication: When incentives conflict, communication may fail because players can misrepresent or ignore messages, as in Chicken.
- Commitments: Credible commitments can avert disaster in Chicken and support conditional cooperation in the Prisoner’s Dilemma.
- Institutions: Institutions structure behavior through rules, conventions, norms, and stronger interventions that can select equilibria or alter games.
4.1. Understanding
Understanding in Cooperative AI concerns predicting action consequences, other agents’ behavior, and relevant beliefs or preferences so agents can achieve mutually beneficial outcomes. It becomes more demanding when agents must discover new cooperative equilibria, infer private information, or reason about recursive beliefs.
- 4.1. Understanding: Understanding includes predicting action consequences, other agents’ behavior, and factors such as their beliefs and preferences.The paper also includes implicit predictions reflected in adapted behavior, not only deliberate reasoning.
- 4.1. Understanding: Agents may need to infer private information from another agent’s actions when direct communication is unavailable.With insufficient common interest, signals may be untrustworthy, making costly signals a possible alternative.
- 4.1. Understanding: In multi-agent settings, anticipating other agents’ actions and responses is particularly important for cooperation.
- 4.1. Understanding: Sustaining cooperation requires mutual understanding of strategies, while discovering a new cooperative equilibrium often requires jointly imagining or finding a new pattern of behavior.Existing experience supports sustaining an equilibrium, but not necessarily establishing that a new one is beneficial and robust.
- 4.1. Understanding: Model-based reasoning can identify novel cooperative equilibria across a large strategy space, but learned-model errors add learning costs and risks.
- 4.1. Understanding: Eliciting preferences can be difficult because broad human preferences may be computationally intractable or lack adequate conscious access.
4.2. Communication
Communication supports cooperative understanding, coordination, and the discovery or maintenance of mutually beneficial equilibria, but it depends on common ground and channel constraints. Mixed motives add risks of deception and motivate incentive-compatible mechanisms, cryptographic information architectures, and teaching capabilities.
- 4.2. Communication: Communication can reveal agents’ behavior, intentions, and preferences, supporting coordination on and maintenance of Pareto-optimal equilibria.
- 4.2. Communication: Limited bandwidth requires efficient context-dependent compression, while high latency demands communication that remains effective despite message delays.
- 4.2. Communication: Teaching and learning are cooperative communication problems involving curriculum design, theory of mind, opponent modeling, shaping strategies, and student sample efficiency.Machines that can teach humans could help people learn from high-performing algorithms such as AlphaZero.
- 4.2. Communication: Meaningful communication requires common ground, such as shared world knowledge, vocabulary, or representations.Building these representations imposes fixed costs that can make different communication complexities optimal in different settings.
- 4.2. Communication: Human-machine communication faces a major common-ground challenge, spanning natural language and actions with communicative content.
- 4.2. Communication: Conflicting preferences increase incentives to deceive, whereas costly signals can sometimes make communication credible under mixed motives.AI research can also develop incentive-compatible mechanisms, trustworthy mediation, and structured transparency for controlled information sharing.
4.3. Commitment
Commitment addresses cooperation failures that persist even when agents have complete information, because agents may lack credible promises or threats. Cooperative AI can study commitment devices, contracts, reputation systems, delegation, and technologies that support sophisticated conditional commitments.
- 4.3. Commitment: Commitment problems are cooperation failures caused by the inability to make credible threats or promises, distinct from purely informational failures.The paper notes that informational and commitment problems are often intertwined in practice.
- 4.3. Commitment: In the Prisoner’s Dilemma, perfect information does not produce cooperation because each player has a unilateral incentive to defect.A conditional commitment to play C if and only if the other plays C can overcome this dilemma.
- 4.3. Commitment: Commitment devices compel fulfillment through incentive changes or by removing actions from the available choice set.They may be unilateral or multilateral, with legal contracts as an example of the latter.
- 4.3. Commitment: Conditional commitments are harder to construct than unconditional ones but can support precise promises or threats needed for cooperation.The paper uses the Prisoner’s Dilemma to illustrate that conditional, unlike unconditional, commitment can be sufficient.
- 4.3. Commitment: AI research can specify commitment contracts, reason about their strategic effects, predict responses, and identify welfare-improving commitment changes.
- 4.3. Commitment: Reputation systems create collateral for cooperative behavior in transient encounters, while delegation lets trusted planners or authorities mediate decisions.Emerging cryptographic protocols, trusted hardware, and smart contracts can support sophisticated commitments without a central authority.
4.4. Institutions
Institutions provide social structures that shape agents’ actions and outcomes, ranging from emergent decentralized norms to designed centralized mechanisms. Cooperative AI can study and design these structures using insights from bargaining, social choice, mechanism design, and machine learning.
- Institutions are systems of beliefs, norms, or rules that determine the rules of the game and shape actions and outcomes.
- Conventions coordinate common-interest games, norms reinforce mixed-motive interactions through rewards and sanctions, and institutions allocate roles, power, and resources.
- Decentralized institutions: Institutions may emerge through repeated interaction or trial and error before becoming more formal, designed, or centralized.
- Decentralized institutions: Bargaining institutions provide methods and protocols for negotiating welfare-improving arrangements, with challenges including formal specification, tractability, and social welfare.
- Centralized institutions: Centralized institutions use an authority to shape participants’ rules and constraints, drawing on social choice theory, game theory, and mechanism design.
- Centralized institutions: Open questions include combining axiomatic and data-driven design, identifying useful social-choice characterizations, setting incentives for welfare or fairness, and integrating communication with social choice.
- Machine learning and institutional design: Machine learning may design institutions from human-specified desiderata, but possible performance gains may trade off against parsimony, interpretability, and closed-form provability.
5. The Potential Downsides of Cooperative AI
Cooperative AI can produce harms through exclusion, collusion, coercion, and interactions with competition, so research must distinguish beneficial cooperation from harmful coordination and investigate mitigation. The paper hypothesizes that broad, widely distributed gains in cooperative competence tend to improve welfare overall, while identifying major open questions about when that holds.
- Cooperative competence can harm excluded groups, support criminal cooperation or rent seeking, and undermine pro-social competition through collusion.
- Coercive capabilities: Many cooperation capabilities also support coercion, including understanding vulnerabilities, distinguishing honesty from deception, making commitments, and designing institutions.
- Coercive capabilities: Punishment can sustain cooperation, as legal contracts facilitate cooperation by exposing parties to punishment for breach.
- Competition and cooperation: Inter-group competition has driven major biological and cultural transitions and may provide a motivating, scalable curriculum for learning cooperative skills.
- Understanding and mitigating downsides: The paper offers the hypothesis that large, broadly distributed increases in cooperative competence tend to be broadly welfare improving on net.
- Historical examples suggest larger-scale cooperative structures have been more effective than smaller parasitic or rivalrous ones.
- Understanding and mitigating downsides: Open problems include identifying cooperation-biased capabilities, building tests for coercive dispositions, and separating gains in cooperative skills from coercive skills.
6. Conclusion
Cooperation has shaped evolutionary and human development and remains central to human well-being, making cooperation an attractive target for intelligence research. AI can contribute scientific tools and cooperative structures, but its deployment should promote human cooperation across disciplines.
- Cooperation contributed to major evolutionary transitions, underpinned human history, and remains critical for human well-being.
- Because cooperation problems are complex and become harder at larger scales, they provide an attractive target for research on intelligence.
- AI provides tools for understanding social systems and devising cooperative structures, while its deployment as tools, infrastructure, and agents should promote human cooperation.
- Cooperation research is dispersed across natural, engineering, and social sciences, so many open problems arise at the intersection of AI with these disciplines.
- Cooperative intelligence is among the kinds of intelligence humanity most needs as AI develops increasingly capable machine agents.
8. Appendix A: Cooperative AI Workshop - NeurIPS 2020
The Cooperative AI workshop agenda frames cooperation as a broad AI research problem involving joint welfare, capable agents, and tools for human-machine cooperation. It organizes work around cooperative capabilities, suitable learning environments, human preferences and norms, and potential harms.
- Aims and focus: Cooperative AI studies how agents can jointly improve welfare across everyday interactions and global challenges.
- Aims and focus: The agenda calls for AI and machine learning to build cooperative agents and tools such as mechanism design and mediation that foster cooperation in human and machine populations.
- Capabilities and methods: Research can be organized around understanding, communication, cooperative commitments, bargaining, and institutions.
- Capabilities and methods: Machine-learning research needs training environments and tasks where cooperative skills are crucial to success, learnable, and non-trivial.
- Human considerations and risks: Cooperative AI must address human preferences, norms, and ethics because artificial agents often act for particular humans in consequential settings.
- Human considerations and risks: The agenda includes studying exclusion, collusion, and coercion and how cooperative skills can most improve human welfare.