Source-linked AI summary
Suphx: Mastering Mahjong with Deep Reinforcement Learning
Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, Hsiao-Wuen Hon
TL;DR
Mahjong’s multi-player imperfect information and complex scoring and rules make it a challenging game-AI problem. The paper introduces Suphx, a deep reinforcement-learning system using global reward prediction, oracle guiding, and run-time policy adaptation. Suphx reached 10 dan and surpassed most top human players on Tenhou, while online testing omitted run-time policy adaptation and the authors note further room for improvement.
Problem
Mahjong is a challenging multi-player imperfect-information game because of its complex scoring and rules and rich hidden information.
Method
Suphx combines supervised learning from professional-player logs with self-play deep reinforcement learning, global reward prediction, oracle guiding, and run-time policy adaptation.
Results
Suphx reached 10 dan on Tenhou and surpassed most top human players in stable rank.
Takeaways & Limitations
Suphx was the first Mahjong AI reported to surpass most top human players on Tenhou.
Takeaways & Limitations
Run-time policy adaptation was not integrated during Tenhou testing because the platform’s per-action time constraint made it time consuming.
Abstract
from arXiv · showhide
Artificial Intelligence (AI) has achieved great success in many domains, and game AI is widely regarded as its beachhead since the dawn of AI. In recent years, studies on game AI have gradually evolved from relatively simple environments (e.g., perfect-information games such as Go, chess, shogi or two-player imperfect-information games such as heads-up Texas hold'em) to more complex ones (e.g., multi-player imperfect-information games such as multi-player Texas hold'em and StartCraft II). Mahjong is a popular multi-player imperfect-information game worldwide but very challenging for AI research due to its complex playing/scoring rules and rich hidden information. We design an AI for Mahjong, named Suphx, based on deep reinforcement learning with some newly introduced techniques including global reward prediction, oracle guiding, and run-time policy adaptation. Suphx has demonstrated stronger performance than most top human players in terms of stable rank and is rated above 99.99% of all the officially ranked human players in the Tenhou platform. This is the first time that a computer program outperforms most top human players in Mahjong.
1 Introduction
Mahjong poses a demanding test for game AI because its multi-round scoring, hidden information, and irregular action structure complicate learning and established search methods. Suphx combines supervised learning, self-play reinforcement learning, and three techniques tailored to these challenges, then achieves strong performance against top human players.
- Challenges: Mahjong combines multi-round scoring, extensive hidden information, and complex action interruptions, making strong play difficult for AI.Final game ranking depends on accumulated round scores; each decision point has more than 10^48 indistinguishable hidden states on average, and the irregular game tree prevents direct use of several established methods.
- Approach: Suphx trains deep convolutional-network policies first on professional-player logs and then through self-play reinforcement learning.The system uses policy gradients for self-play and introduces techniques to address Mahjong’s challenges.
- Approach: Global reward prediction uses future game outcomes to provide learning signals beyond individual round scores and adds look-ahead features for winning hands and scores.The predictor estimates final reward from current and previous rounds, while look-ahead features represent possible winning hands and their round scores.
- Approach: Oracle guiding accelerates reinforcement learning by gradually removing an oracle agent’s access to opponents’ private tiles and wall tiles until it becomes a normal observable-information agent.The oracle sees perfect state information during training, whereas the normal agent receives only observable information.
- Approach: Parametric Monte-Carlo policy adaptation updates the offline policy during play as more public information becomes available, improving run-time performance.The method adapts to a specific round rather than relying on a regular game tree, whose construction is obstructed by Mahjong’s complex rules.
- Evaluation: Suphx reached 10 dan on Tenhou and achieved a stable rank surpassing most top human players.The evaluation used Tenhou, a competitive Mahjong platform with more than 350,000 active users.
2 Overview of Suphx
Suphx organizes Mahjong play around specialized models and a rule-based winning model, while encoding observable game information and look-ahead features for convolutional networks. Its decision flow handles draws, discards, melds, and interruptions through situation-specific action models.
- Decision Flow: Suphx uses five learned models—discard, Riichi, Chow, Pong, and Kong—plus a rule-based winning model to handle Mahjong’s complex situations.The learned models decide discards and whether to take specific actions, while the winning model applies simple game-state rules.
- Decision Flow: The winning model declares a winning hand except when the last-round result would leave the accumulated game score lowest among the four players.Outside the last round, a legal winning hand is declared; in the last round, the model suppresses a win only under the stated lowest-score condition.
- Decision Flow: During a draw, Suphx checks winning and Kong options, then Riichi, and otherwise uses the discard model before other players act.The flow returns to drawing after a ClosedKong and proceeds to discarding after Riichi or an ordinary draw.
- Decision Flow: After another player discards, Suphx checks winning, Chow, Pong, and Kong possibilities and selects the highest-confidence proposed action unless a higher-priority action interrupts it.If no action is suggested, the turn passes to other players or the round ends when no wall tiles remain.
- Features and Model Structures: Suphx encodes tiles, discards, categorical variables, and bucketed integer features into channels suitable for deep convolutional networks.Private tiles use four channels across 34 tile types, while categorical and integer features use all-zero/all-one channels or bucket channels.
- Features and Model Structures: Look-ahead features estimate winning probabilities and round scores after hypothetical discards and replacements, capturing Mahjong’s many possible winning hands.The representation supplements directly observable information with prospective hand and score information.
- Features and Model Structures: The models share similar network structures but differ in input and output dimensions, with discard predicting among 34 tiles and action models using two outputs.Chow, Pong, and Kong models also receive information about the tiles involved in the candidate action.
3 Learning Algorithm
Suphx combines distributed policy-gradient reinforcement learning with entropy control and specialized methods for Mahjong’s delayed rewards, hidden information, and irregular play. It also adapts its policy at run time using simulations tailored to the current hand.
- 3 Learning Algorithm: Suphx trains five models with supervised learning, then improves them through distributed self-play reinforcement learning using policy gradients and importance sampling.Asynchronous trajectory generation is decoupled from policy updates through self-play workers, GPU inference engines, a replay buffer, and a parameter server.
- 3.1 Distributed Reinforcement Learning with Entropy Regularization: Entropy regularization stabilizes exploration by dynamically adjusting the trade-off coefficient toward a target policy entropy.Too little entropy limits self-play improvement, whereas too much makes training unstable and produces high-variance policies.
- 3.2 Global Reward Prediction: A global reward predictor assigns round-level learning signals by predicting final game reward from the current and previous rounds.The predictor is trained on top-human Tenhou logs and uses a recurrent network with two GRU layers followed by fully connected layers.
- 3.3 Oracle Guiding: Oracle guiding accelerates learning by gradually dropping perfect-information features until an oracle-trained policy becomes a normal agent using observable information.The dropout probability for perfect features decays from 1 to 0 during training.
- 3.4 Parametric Monte-Carlo Policy Adaptation: Parametric Monte-Carlo policy adaptation fine-tunes the offline policy for a fixed private hand and generalizes simulation-derived updates to unseen states.The method is motivated by Mahjong’s irregular play order, which prevents direct application of ordinary Monte-Carlo tree search.
4 Offline Evaluation
Offline experiments evaluate Suphx’s supervised initialization, reinforcement-learning components, global reward prediction, oracle guiding, and run-time policy adaptation. The results show progressive gains from the learning components and a 66% win rate for adapted RL-2 against its non-adapted version.
- 4.1 Supervised Learning: 76.7%, 85.7%, 95.0%, 91.9%, and 94.0% are the supervised test accuracies for the discard, Riichi, Chow, Pong, and Kong models, respectively.The models were trained separately from human professional state-action pairs; the reported values are not directly comparable with previous work because datasets and architectures differ.
- 4.2 Reinforcement Learning: RL-basic improves over SL, RL-1 outperforms RL-basic, and RL-2 adds further gains, demonstrating value from reinforcement learning, global reward prediction, and oracle guiding.Each agent was trained using 1.5 million games, with the comparison based on stable-rank evaluation.
- 4.3 Run-time Policy Adaptation: 66%: the run-time-adapted RL-2 agent defeats its non-adapted version in the tested setting.The technique was evaluated on hundreds of initial rounds because roll-outs and online learning are time consuming.
- 4.3 Run-time Policy Adaptation: Run-time adaptation makes the policy more aggressive in late-game situations when simulations indicate that winning the current round alone would still yield fourth place.The method is intended to tailor decisions to the agent’s fixed private hand.
5 Online Evaluation
Suphx was evaluated against AI and human players on Tenhou using stable-rank comparisons and additional rank, win, and deal-in statistics. It achieved leading performance among the compared systems and players, while the evaluation excluded run-time policy adaptation.
- Tenhou evaluation: Suphx reached 10 dan in record rank and 8.74 dan in stable rank after playing more than 5000 expert-room games.It is reported as the first and only AI in Tenhou to achieve 10 dan in record rank.
- Evaluation limitation: Run-time policy adaptation was not integrated during Tenhou testing because its time cost conflicted with the platform’s per-action constraint.The authors state that integrating adaptation could further improve Suphx, but this was not tested in the online evaluation.
- Comparison with AI and human players: Suphx is about 2 dan better in stable rank than Bakuuchi and NAGA, and ranks above 99.99% of human Tenhou players.The comparison uses stable rank rather than record rank because record rank can mask differences in true playing strength.
- Additional statistics: Suphx has a very low 4-th rank rate, which is important for achieving a high stable rank under Tenhou’s scoring rules.The broader statistics also characterize Suphx as strong at defense with a very low deal-in rate.
- Playing style: Suphx’s playing style includes keeping safe tiles to balance future attack and defense, a behavior illustrated by the supplied evaluation example.The paper reports that this style was recognized by top human players.
6 Conclusion and Discussions
Suphx combines deep reinforcement learning with global reward prediction, oracle guiding, and run-time policy adaptation to address Mahjong’s complex rewards, hidden information, and irregular game tree. It surpassed most top human Mahjong players on Tenhou while remaining open to further improvement.
- Suphx became the first Mahjong AI to surpass most top human players on Tenhou.net and is described as the strongest Mahjong AI system to date.
- Global reward prediction estimates final game rewards from current and previous rounds, while look-ahead features represent winning possibilities and scores.
- Oracle guiding gradually removes perfect information from an oracle agent, accelerating learning before producing a normal observable-information agent.
- Run-time policy adaptation modifies the offline policy as more tiles become observable, with continued adaptation intended to improve performance.
- The authors argue that Suphx techniques may benefit real-world problems sharing Mahjong’s complex rules and imperfect information, while acknowledging substantial room for improvement.
Appendix A: Rules of Mahjong
Japanese Mahjong uses 136 tiles, multiple rounds, complex winning-hand and scoring rules, and interruptions caused by melds or winning declarations. These rules create a large action and outcome space for four players.
- Japanese Mahjong has 136 tiles, four players, multiple rounds, and ends when a losing or winning condition is triggered.
- Each player begins with 13 private tiles; 14 tiles form the dead wall and 70 form the live wall, which players draw from during play.
- A complete winning hand contains four melds and a pair, with melds formed as Pongs, Kongs, or Chows.
- Yaku patterns determine round scores, and a hand may contain multiple yaku whose values accumulate, with dora providing bonus points.
- Play can be interrupted by actions such as Kong, declaring a winning hand, or robbing a Kong, complicating the regular turn order.
- Final ranking points depend on the player’s level and accumulated round-score rank over the game’s multiple rounds.
Appendix B: Tenhou Ranking Rules
Tenhou ranks players through levels from rookie and kyuu ranks to dan ranks, with ranking points changing according to game results, player level, and room. Rank levels rise or fall as cumulative points cross thresholds.
- Tenhou’s ranking system runs from rookie and 9 kyuu through 1 kyuu, then from 1 dan to 10 dan.
- Players earn or lose ranking points based on game result, current level, and room, with winning rewards varying across rooms while losing penalties are consistent.
- A player’s rank increases when total points reach the next-level requirement and decreases when points fall to zero.
- Record rank denotes the highest rank ever achieved because rank levels can change over time.
Appendix C: Stable Rank
Stable rank is Tenhou’s measure of long-term average performance in the expert room, based on the distribution of finishing ranks. Hidden-information and opponent-allocation randomness make it variable, requiring many games for reliable estimation.
- Stable rank evaluates a player’s long-term average performance in Tenhou’s expert room.
- Its calculation uses the numbers of games finished first, second, third, and fourth by accumulated round score.
- Stable rank can vary substantially because outcomes depend on hidden tiles and randomly allocated opponents, so several thousand games are generally assumed necessary for reliable estimation.