Source-linked AI summary
Cooperating with Machines
Jacob W. Crandall, Mayada Oudah, Tennom, Fatimah Ishowo-Oloko, Sherief Abdallah, Jean-François Bonnefon, Manuel Cebrian, Azim Shariff, Michael A. Goodrich, Iyad Rahwan
TL;DR
Many autonomous systems must cooperate with people and machines despite conflicting preferences, but human coordination mechanisms such as cheap talk are difficult for AI to reproduce. This paper introduces S#, which augments S++ with mechanisms for generating and responding to signals; across repeated interactions, S# established cooperation with people and machines at levels rivaling human cooperation. Its evidence is limited to five human-S# repeated games, although broader algorithmic results provide some confidence that the findings may generalize.
Problem
Autonomous machines need to cooperate with people and other machines despite conflicting goals, but human cooperation relies on signals and other mechanisms difficult to encode for arbitrary contexts.
Method
S# couples S++ with mechanisms that generate speech acts from expert strategies and use partners’ signals to guide expert selection.
Results
S# established effective cooperative relationships with people and other machines across a wide variety of repeated interactions, with cooperation rivaling human cooperation.
Takeaways & Limitations
Flexible human-machine cooperation is computationally achievable with a non-trivial but ultimately simple combination of algorithmic mechanisms.
Takeaways & Limitations
Human-S# studies covered five repeated games, so future work should test additional scenarios.
Abstract
from arXiv · showhide
Since Alan Turing envisioned Artificial Intelligence (AI) [1], a major driving force behind technical progress has been competition with human cognition. Historical milestones have been frequently associated with computers matching or outperforming humans in difficult cognitive tasks (e.g. face recognition [2], personality classification [3], driving cars [4], or playing video games [5]), or defeating humans in strategic zero-sum encounters (e.g. Chess [6], Checkers [7], Jeopardy! [8], Poker [9], or Go [10]). In contrast, less attention has been given to developing autonomous machines that establish mutually cooperative relationships with people who may not share the machine's preferences. A main challenge has been that human cooperation does not require sheer computational power, but rather relies on intuition [11], cultural norms [12], emotions and signals [13, 14, 15, 16], and pre-evolved dispositions toward cooperation [17], common-sense mechanisms that are difficult to encode in machines for arbitrary contexts. Here, we combine a state-of-the-art machine-learning algorithm with novel mechanisms for generating and acting on signals to produce a new learning algorithm that cooperates with people and other machines at levels that rival human cooperation in a variety of two-player repeated stochastic games. This is the first general-purpose algorithm that is capable, given a description of a previously unseen game environment, of learning to cooperate with people within short timescales in scenarios previously unanticipated by algorithm designers. This is achieved without complex opponent modeling or higher-order theories of mind, thus showing that flexible, fast, and general human-machine cooperation is computationally achievable using a non-trivial, but ultimately simple, set of algorithmic mechanisms.
1 Introduction
Autonomous machines must cooperate with people and other machines whose goals may conflict, but existing AI has focused mainly on competition and zero-sum tasks. Effective cooperation requires generality, flexibility, and communication mechanisms that support human coordination.
- Autonomous machines increasingly interact with people and machines that may have conflicting goals and preferences.
- Most AI milestones emphasize competing with humans, although many real-world interactions require cooperation despite exploitation risks.
- A successful cooperative algorithm must generalize across scenarios and learn relationships without prior knowledge of its associates’ behavior.
- Human cooperation relies on intuition, norms, emotions, signals, and evolved cooperative dispositions that are difficult to encode for arbitrary contexts.
- Cheap talk helps people coordinate quickly and create shared representations, making understandable communication an important requirement for repeated-interaction algorithms.
2 Results
The paper develops S#, a learning system that adds signal generation and response mechanisms to S++ and evaluates it in interactions with people and machines. Across repeated games, the system forms effective cooperative relationships, while its signaling, expert set, and expert-selection mechanism explain its performance.
- 2 Results: The paper’s primary contribution is a learning system that couples machine learning with mechanisms for generating and responding to signals.
- 2.1 An Algorithm that Cooperates with People and Other Machines: Among 25 representative algorithms, only S++ performed consistently at the top across all metrics and game lengths with machine associates, yet it did not consistently cooperate with people.
- 2.1 An Algorithm that Cooperates with People and Other Machines: S# augments S++ by comparing partner speech acts with experts’ philosophies and generating game-generic speech from selected experts’ state machines.
- 2.1 An Algorithm that Cooperates with People and Other Machines: Cheap talk doubled mutual cooperation in Human-Human and Human-S# pairings across three representative games when compared with conditions without cheap talk.
- 2.1 An Algorithm that Cooperates with People and Other Machines: Participants often could not distinguish S# from a human player, and combined behavioral and signaling strategies were as effective as human strategies across the studied games.
- 2.2 Distinguishing Algorithmic Mechanisms: S#’s success reflects understandable signaling, a rich expert set, and expert selection aligned with recognition-primed decision making rather than conventional exploration mechanisms.
- 2.3 Repeated Stochastic Games: S# also cooperated with people in repeated stochastic games, where cheap talk substantially increased payoffs in a turn-taking block-sharing scenario.
3 Discussion
The studies suggest that S# can form cooperative relationships with people and machines across representative repeated games, while the human-partnership evidence remains limited in scope. The authors therefore treat broader generalization as plausible but requiring future evaluation.
- Scope and generalization: Human-S# studies covered five repeated games representing different classes from the periodic table of games.The authors identify broader scenario coverage as a direction for future work.
- Scope and generalization: S# consistently performed highly across all classes of 2x2 games and various repeated stochastic games when interacting with other algorithms.This machine-machine performance contributes to the authors’ confidence that the human-partnership findings may generalize.
- Core conclusion: Human-machine and machine-machine cooperation was achieved through expert strategies, a meta-strategy for selecting experts, and simple signaling.The authors characterize these mechanisms as non-trivial but ultimately simple.
- Core conclusion: The work presents an extensive demonstration that autonomous machines can learn cooperative relationships in repeated interactions rather than only compete in zero-sum encounters.The discussion frames cooperation as necessary when machines interact with humans or machines amid conflicting interests and exploitation threats.
4 Methods
The methods benchmark cooperation across canonical repeated games, compare 25 algorithms using six metrics, and describe S++ and its speech-enabled extension S#.
- Interacting with Other Machines: The evaluation compares 25 representative algorithms across the periodic table of 2×2 games using six performance metrics and multiple interaction lengths.Algorithms include classical, evolutionary, machine-learning, belief-based, and expert approaches.
- Interacting with Other Machines: Only S++ was top-performing across all metrics and game lengths when paired with other algorithms.S++ also maintained high performance across game classes and associate classes.
- S++ and Expert Selection: S++ computes diverse experts from the game description, filters them by potential relative to an aspiration level, selects one satisficingly, and updates learning after each epoch.Its reduced expert set contains strategies expected to produce satisfactory payoffs.
- Human Cooperation: S++ effectively forms profitable long-term relationships with machines, motivating evaluation of whether S# can establish cooperative relationships with people.The paper frames this question within arbitrary repeated games and signal-based cooperation.
- S#: A Machine-Learning Algorithm that Talks: S# extends S++ by using partners’ speech-act proposals to filter experts and by generating speech that conveys each selected expert’s strategy state.If no desirable expert matches the partner’s proposal, S# falls back to the original S++ filtering procedure.