Source-linked AI summary
Towards Playing Full MOBA Games with Deep Reinforcement Learning
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, Yinyuting Yin, Bei Shi, Liang Wang, Tengfei Shi, Qiang Fu, Wei Yang, Lanxiao Huang, Wei Liu
TL;DR
Existing MOBA AI systems struggle to scale beyond restricted hero pools because expanding heroes creates enormous, difficult-to-learn lineup combinations. The paper combines curriculum self-play, policy distillation, off-policy adaption, multi-head value estimation, and Monte-Carlo tree search, achieving 95.2% win-rate over 42 matches against professionals in Honor of Kings.
Problem
Scaling self-play reinforcement learning to full MOBA games is difficult because expanding the hero pool causes a huge increase in lineups and can produce degraded performance or learning collapse.
Method
The paper develops a deep-reinforcement-learning paradigm combining curriculum self-play, multi-teacher policy distillation, off-policy adaption, multi-head value estimation, and Monte-Carlo tree search.
Results
95.2% win-rate over 42 matches against professionals was achieved in Honor of Kings, with 40 wins.
Takeaways & Limitations
The approach supports a 40-hero-or-more MOBA AI and defeats top human players with statistical significance in the reported Honor of Kings evaluation.
Abstract
from arXiv · showhide
MOBA games, e.g., Honor of Kings, League of Legends, and Dota 2, pose grand challenges to AI systems such as multi-agent, enormous state-action space, complex action control, etc. Developing AI for playing MOBA games has raised much attention accordingly. However, existing work falls short in handling the raw game complexity caused by the explosion of agent combinations, i.e., lineups, when expanding the hero pool in case that OpenAI's Dota AI limits the play to a pool of only 17 heroes. As a result, full MOBA games without restrictions are far from being mastered by any existing AI system. In this paper, we propose a MOBA AI learning paradigm that methodologically enables playing full MOBA games with deep reinforcement learning. Specifically, we develop a combination of novel and existing learning techniques, including curriculum self-play learning, policy distillation, off-policy adaption, multi-head value estimation, and Monte-Carlo tree-search, in training and playing a large pool of heroes, meanwhile addressing the scalability issue skillfully. Tested on Honor of Kings, a popular MOBA game, we show how to build superhuman AI agents that can defeat top esports players. The superiority of our AI is demonstrated by the first large-scale performance test of MOBA AI agent in the literature.
1 Introduction
MOBA games combine multi-agent competition and cooperation, imperfect information, complex control, and enormous state-action spaces. This paper proposes a scalable deep-reinforcement-learning paradigm for full MOBA games and evaluates it through large-scale professional matches.
- MOBA games are challenging AI testbeds because they combine multi-agent competition and cooperation, imperfect information, complex action control, and enormous state-action spaces.Honor of Kings can have a state and action space of magnitude 10^20000 even after significant discretization.
- OpenAI Five advanced 5v5 MOBA play but supported only a subset of 17 heroes and simplified some regular-game rules.The hero-pool restriction limits coverage of the hero-varying and team-varying mechanics central to MOBA games.
- 213,610,453,056 lineups arise for 40 heroes versus 4,900,896 for 17 heroes, making random presentation of combinations prone to learning collapse.The paper reports degraded performance and unacceptably slow training when expanding OpenAI Five's pool toward 25 heroes, even with thousands of GPUs.
- The proposed paradigm combines off-policy adaption, multi-head value estimation, curriculum self-play, policy distillation, and Monte-Carlo tree search to train and play many heroes.Training starts with fixed lineups, distills teacher policies into a student model, and performs merged training.
2 Related Work
Prior Game AI research spans rule-based, supervised, reinforcement-learning, and hybrid systems across RTS and MOBA games. Existing MOBA systems differ in scope, from macro guidance or 1v1 micro-control to 5v5 self-play with a restricted hero pool.
- RTS games: AlphaStar combined supervised learning and multi-agent reinforcement learning to achieve grandmaster-level StarCraft 2 play.
- MOBA games: Tencent HMS guides macro movement on the map but does not execute agents' actions, so it is not a complete MOBA AI solution.
- MOBA games: Tencent Solo masters MOBA combat micro-control with reinforcement learning but studies only 1v1 solo games rather than multi-agent 5v5 play.
3 Learning System
The learning system combines a unified neural architecture, off-policy actor-critic training, curriculum self-play, policy distillation, and drafting search to address MOBA complexity and large hero pools.
- 3.1 Architecture: The unified policy network encodes multimodal observations, maintains history with LSTM, and maps representations to action distributions.It uses scalar and spatial features, including attributes, game statistics, opponent information, and local-view map channels.
- 3.1 Architecture: Hierarchical action heads decompose control into action type, target, and discretized execution details such as movement direction.This structure addresses the enormous action space by predicting what to do, whom to target, and how to act.
- 3.2 Reinforcement Learning: Off-policy actor-critic training uses replayed experiences, Dual-clip PPO for policy updates, and multi-head value estimation to improve stability and value accuracy.The value function can use hidden game-state information during training, while reward decomposition produces multiple value heads whose weighted sum estimates total value.
- 3.3 Multi-agent Training: Large hero pools create combinatorial lineups, non-stationary self-play, and drafting trees too large for complete search methods.For 40 heroes, the cited lineup count is 213,610,453,056, and complete Minimax search is described as computationally intractable.
- 3.3 Multi-agent Training: Curriculum self-play progresses from fixed-lineup training through multi-teacher policy distillation to continued learning with randomly selected lineups.Phase transitions are based on Elo-score convergence, and the distilled student inherits policy and value knowledge from fixed-lineup teachers.
- 3.4 Learning to draft: MCTS handles drafting through selection, expansion, simulation, and backpropagation, with a value network replacing random rollouts during simulation.The value network is trained using simulated drafting data and lineup win-rate predictions derived from a match dataset.
4 Evaluation
The evaluation uses Honor of Kings with a 40-hero pool and unrestricted game rules, testing the AI against professionals and top-ranked public players. Training comparisons and ablations examine scalability, component contributions, and drafting strategies.
- Experimental Setup: The AI was trained for a 40-hero pool covering all hero roles, with no restrictions on item builds or summoner abilities.The pool was 2.4x larger than previous MOBA AI work and produced 2.1 × 10^11 more agent combinations.
- AI Performance: 95.2% win rate: the AI won 40 of 42 matches against professional Honor of Kings esports teams.The matches were conducted weekly over 10 weeks, with professionals encouraged to use skilled heroes and varied team strategies.
- AI Performance: 97.7% win rate: the AI won 627,280 of 642,047 matches against top-ranked public players.Only players around the High King level could participate, and repeated play was allowed.
- AI Performance: The 20-hero version achieved a 100% win rate in 30 matches against professional teams, while training toward a complete 101-hero pool continued.The 20-hero model used the same training paradigm as the 40-hero model.
- Ablations: Ablations found benefits from CSPL, invisible opponent information, multi-head value estimation, dual-clip PPO, and MCTS-based drafting.The drafting comparison used 1,000 matches for each pair of strategies; MCTS outperformed random drafting and highest-win-rate drafting.
5 Conclusion and Future Work
The paper presents a deep-reinforcement-learning paradigm for full-MOBA play and reports statistically significant victories over top human players. Future work targets complete hero-pool support and more efficient training.
- The proposed paradigm combines off-policy adaption, multi-head value estimation, curriculum self-play, multi-teacher policy distillation, and Monte-Carlo tree search.
- The Honor of Kings AI can defeat top human players with statistical significance.
- The system supports a 40-hero pool, which the paper identifies as the first reinforcement-learning-based MOBA AI program to reach that scale or more.
- Future work will pursue complete hero-pool support and more efficient training methods to shorten the MOBA AI learning process.
6 Broader Impact
The paper frames unrestricted MOBA mastery as a major AI challenge and describes implications for research, game development, and esports. Its methodology is presented as potentially relevant to similar multiplayer domains, while its AI has already influenced game design and professional play.
- The proposed methodology uses general-purpose machine-learning components applicable to other similar multiplayer domains.
- The AI has found real-world applications in Honor of Kings and is changing how MOBA game designers, particularly balance designers, work.
- The AI uses playing styles that differ from normal human esports strategies, including alternative resource allocation and hero-role choices.
- Professional feedback reports that some AI playing styles increased gold and experience during certain game phases and suggested marksman heroes can play middle.
8 Supplementary Materials
The supplementary materials describe the infrastructure, game interface, hero pools, lineup organization, action and reward designs, and feature representations used in the experiments. They also document substantial computational requirements and a fixed game version for fair comparison.
- 8.1 Infrastructure Design: KaiWu comprises an AI Server, Inference Server, distributed RL Learner, and circular-queue Memory Pool.
- 8.2 Game Environment: The game UI exposes movement and ability controllers, a local screen view, dashboard game states, and a global mini-map view.
- 8.3 Hero Pool: Table 2 summarizes fixed lineups for the 20-hero and 40-hero settings used in Phase 1 of curriculum self-play learning.
- The action space, reward design, and feature details are documented in Tables 3, 4, and 5.
- Features include scalar and spatial inputs, while invisible opponent information is used only by the value network during training.
- Continuous features are normalized to [0,1], whereas discrete features use one-hot representations.