Source-linked AI summary
The Hanabi Challenge: A New Frontier for AI Research
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, Iain Dunning, Shibl Mourad, Hugo Larochelle, Marc G. Bellemare, Michael Bowling
TL;DR
The paper asks whether Hanabi can expose multi-agent challenges involving cooperation, imperfect information, and reasoning about teammates’ beliefs and intentions. It proposes Hanabi as a benchmark, releases an open-source environment and evaluation framework, and assesses existing methods. Current learning techniques remain below hand-coded agents in self-play and fail to collaborate reliably in ad-hoc teams.
Problem
Hanabi combines cooperative gameplay, imperfect information, and limited communication, requiring agents to reason about teammates’ beliefs and intentions.
Method
The paper introduces Hanabi as an AI benchmark, releases the Hanabi Learning Environment, proposes self-play and ad-hoc evaluation protocols, and benchmarks rule-based and reinforcement-learning agents.
Results
Current learning techniques remain below the best hand-coded agents in self-play and fail to collaborate effectively with unknown teammates in ad-hoc play.
Takeaways & Limitations
Hanabi provides a research domain for studying cooperative learning, communication, and theory-of-mind reasoning in self-play and ad-hoc teams.
Takeaways & Limitations
The study reports that some agent combinations in ad-hoc play were not evaluated because ACHA caused SmartBot to crash, and assumes other combinations would perform similarly without further evidence.
Abstract
from arXiv · showhide
From the early days of computing, games have been important testbeds for studying how well machines can do sophisticated decision making. In recent years, machine learning has made dramatic advances with artificial agents reaching superhuman performance in challenge domains like Go, Atari, and some variants of poker. As with their predecessors of chess, checkers, and backgammon, these game domains have driven research by providing sophisticated yet well-defined challenges for artificial intelligence practitioners. We continue this tradition by proposing the game of Hanabi as a new challenge domain with novel problems that arise from its combination of purely cooperative gameplay with two to five players and imperfect information. In particular, we argue that Hanabi elevates reasoning about the beliefs and intentions of other agents to the foreground. We believe developing novel techniques for such theory of mind reasoning will not only be crucial for success in Hanabi, but also in broader collaborative efforts, especially those with human partners. To facilitate future research, we introduce the open-source Hanabi Learning Environment, propose an experimental framework for the research community to evaluate algorithmic advances, and assess the performance of current state-of-the-art techniques.
1. Introduction
Hanabi is proposed as a cooperative, imperfect-information benchmark that foregrounds reasoning about other agents’ beliefs and intentions. The paper introduces evaluation resources and finds current learning methods remain substantially behind hand-coded agents and unreliable with ad-hoc teammates.
- Motivation: Multi-agent decision making requires agents to account for other agents’ stochastic, changing, or privately informed behavior.The paper connects this requirement to theory of mind: reasoning about others’ perspectives, beliefs, and intentions.
- Motivation: Hanabi combines cooperation, imperfect information, limited communication, and two to five players, creating distinctive multi-agent learning challenges.Players cannot see their own cards and must coordinate through grounded hints with limited information tokens.
- Why Hanabi: Hanabi differs from adversarial two-player zero-sum games because cooperative play makes coordination across potentially multiple equilibria consequential.The paper emphasizes that different conventions and policies can matter because teammates must respond compatibly.
- Human play: Human Hanabi play appears to use theory of mind from the first game, including reasoning about teammates’ perspectives and intentions when signaling and acting.Experienced players also use conventions whose rationale explicitly involves other players’ beliefs and intentions.
- Contributions: Current learning techniques achieve reasonable self-play performance but generally trail the best hand-coded agents and learn brittle policies for ad-hoc teams.The paper identifies substantial room for advances in both settings, especially as the number of players increases.
2. Hanabi: The Game
Hanabi is a cooperative card game in which players act with incomplete knowledge of their own hands and limited grounded communication. Success requires strategically selecting hints and interpreting teammates’ actions as implicit signals.
- Game structure: Hanabi supports two to five players who build five ordered color stacks from cards ranked 1 to 5.Players hold cards they cannot see, while seeing their partners’ hands.
- Actions: Players choose among hinting, playing, and discarding; hints identify all cards matching a chosen rank or color and consume information tokens.Discarding recovers an information token when fewer than eight remain, while unsuccessful plays cost a life and discard the card.
- Basic strategy: Hints must be selected for contextual value rather than information quantity alone because tokens cannot fully reveal every playable card.The game contains too few tokens to provide complete rank-and-color information for all 25 target cards.
- Information limits: 29 hints is the maximum during a game, and fewer are available as the number of players increases.The bound follows from eight initial tokens, recovered tokens, and the limited number of useful discards.
- Implicit communication: Players can communicate implicitly because the rationale for choosing an action may reveal information beyond its direct game effect.A color hint can signal playability, while a finesse exploits teammates’ inferences about one another’s intended interpretations.
3. Hanabi: The Challenge
The paper establishes Hanabi as a benchmark for cooperative, partially observed multi-agent learning and defines self-play and ad-hoc evaluation protocols. It also releases an open-source environment and shows that current learning agents struggle against hand-crafted rules, especially with more players.
- Benchmark motivation: Hanabi combines partial observability, cooperative rewards, and communication embedded in environment actions, unlike standard adversarial or separate-channel benchmarks.These properties motivate its use as a challenging multi-agent learning benchmark.
- Evaluation challenges: The benchmark evaluates self-play, where agents learn a high-utility joint policy, and ad-hoc play, where agents cooperate with unknown partners after limited interaction.Ad-hoc play focuses on recognizing intent and adapting across diverse teammate strategies.
- Research infrastructure: The Hanabi Learning Environment provides an open-source Python and C++ interface for generating observations, rewards, and transitions from agent actions.The environment is designed to support consistent, comparable, and reproducible research.
- Evaluation framework: The proposed evaluation framework includes sample-limited and unlimited regimes plus reporting of training curves, score distributions, perfect-game rates, and performance variance.Self-play results should be reported over at least 1000 full-game trials for the best agent.
- Empirical challenge: Current learning agents have difficulty approaching hand-crafted rules in four-player games and fall far short in three- and five-player games, even with extensive computation.The paper also states that learning techniques fail to collaborate effectively with unknown teammates in ad-hoc play.
- Ad-hoc evaluation: Canonical pools of pre-trained or hard-coded agents are proposed for training and hold-out sets to enable consistent ad-hoc comparisons.The evaluation measures performance with previously unseen teammates selected from diverse pools.
4. Hanabi: State of the Art
State-of-the-art learning agents remain substantially behind strong hand-coded strategies in Hanabi self-play, while ad-hoc play exposes severe incompatibilities between learned conventions. The experiments also reveal training instability, convention diversity, and scaling challenges as player count increases.
- Self-play: Learning agents fall between competitiveness with hand-coded rules and significant underperformance, with neither Rainbow nor ACHA matching the best hand-coded agents beyond two players.Experienced human teams are considered stronger than these bots, leaving an even larger gap to superhuman self-play.
- Self-play: BAD achieves the best reported two-player performance, but its evaluation on only two-player Hanabi and a small synthetic game limits general claims about algorithmic improvement.The authors find its tracking of all agents’ beliefs suggestively associated with marked performance improvement.
- Self-play: ACHA outperforms Rainbow across all player counts, using over 20 billion experience steps per learner versus Rainbow’s 100 million, while both decline as player count increases.The authors note that ACHA’s advantage may reflect greater training experience or Rainbow’s lack of action-history memory; ACHA drops precipitously with five players.
- Self-play: Independent ACHA runs converge to many difficult local minima, with roughly one-third of two-player agents below 15 points and no longer improving.Parameter evolution can hide the extent of this local-minima problem.
- Self-play: ACHA agents with similar final scores learn different conventions, including distinct uses of color and rank hints for discarding or playing cards.These findings show that comparable performance does not imply a shared communication protocol.
- Ad-hoc team play: In ad-hoc teams, performance drops sharply when agents use different learned conventions, with some pairings scoring essentially zero and combinations scoring only slightly more than zero overall.Because agents do not learn from ad-hoc teammates, performance depends on protocol compatibility.
5. Hanabi: Related Work
Hanabi research spans rule-based, search-based, complexity, reinforcement-learning, emergent-communication, and agent-modelling approaches. The paper positions Hanabi as a setting for studying cooperation, imperfect information, communication, and reasoning about other agents.
- Prior Hanabi AI: Earlier Hanabi work included intention-recognizing strategies, the hat strategy, and competition agents using genetic algorithms or Monte Carlo tree search.The 2018 competition included Mirror and Mixed tracks, with reported average scores of 17.52 and 20.57 for two agents.
- Prior Hanabi AI: Prior Hanabi agents used fixed rules, Monte Carlo planning, search, and models of other players to improve play.Reported approaches include manually tuned rules, Monte Carlo tree search, predictor models, and search combined with the hat strategy.
- Complexity: Optimal Hanabi gameplay is NP-hard even for a centralized cheating player with perfect information of the cards and deck order.
- Reinforcement learning: The paper identifies a gap in learning to play Hanabi directly from experience using only the game rules, framed as a reinforcement-learning problem.Earlier work primarily used rule-based and search-based players exploiting domain knowledge.
- Related machine-learning problems: Imperfect-information games require agents to reason about information they do not know, while emergent-communication work studies learned communication for cooperative tasks.Hanabi differs because communication is embedded in the game rather than provided as a separate cheap-talk channel.
- Agent modelling and communication: Hanabi is presented as a domain for studying communication with artificial agents and human partners, including theory-of-mind-based pragmatics.Formal belief-and-intention models exist, but exact solutions can become intractable for complex problems.
6. Conclusion
The paper presents Hanabi as a cooperative imperfect-information challenge for multi-agent learning and evaluates current methods in self-play and ad-hoc teams. It finds substantial shortcomings relative to hand-coded play and motivates standardized evaluation through an open-source framework.
- Hanabi combines cooperative gameplay and imperfect information into a research challenge centered on multi-agent learning.
- State-of-the-art deep reinforcement-learning agents generally fail to surpass hand-coded bots in self-play and fail to collaborate with unknown teammates.
- The paper argues that theory of mind may be important for human Hanabi play and for agents learning to collaborate with people and unfamiliar teammates.
- The authors provide open-source Hanabi code and propose evaluation methodology to support consistent comparison among techniques.
Appendix A.1. FireFlower Details
FireFlower uses conventions that coordinate hints, discards, protection, and inferred play across teammates. These conventions let actions communicate information beyond their immediate literal content.
- FireFlower generally uses hints to indicate playable cards, prioritizing newer cards.
- Hints can chain across players, allowing one player’s indicated play to support a subsequent hint for the next card.
- Players discard provably useless cards or otherwise discard the oldest unprotected card.
- Hints about an oldest unprotected card generally protect it, although exceptions can make the hint mean play instead.
- Hints about garbage cards signal protection for cards older than the indicated garbage card.
- Deliberately discarding a known playable card signals that the partner holds a copy and may identify it precisely enough to play without hints.
- Hints about previously hinted cards can change teammates’ beliefs and alter their future actions.
Appendix A.2. Conditional Probability Tables
This appendix presents conditional probability summaries of learned policies from ACHA and Rainbow runs in the two-player game. The tables cover multiple Rainbow and ACHA agents.
- The tables summarize conditional action probabilities for learned ACHA and Rainbow policies in the two-player game.
- Rainbow agents: Tables A.2, A.3, and A.4 report conditional action probabilities for three Rainbow agents.
- ACHA agents: Tables A.5 and A.6 report conditional action probabilities for ACHA agents.