Source-linked AI summary
Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, Lanxiao Huang
TL;DR
MOBA 1v1 games pose a complex reinforcement-learning problem because their state, action, and combat-control demands exceed those of traditional 1v1 benchmarks. The paper develops a scalable deep reinforcement-learning system and actor-critic algorithm with several specialized control strategies, and Tencent Solo defeats top professional players in Honor of Kings full 1v1 games.
Problem
MOBA 1v1 games require reinforcement learning over highly complex state and action spaces, partially observable combat, target selection, and hero-specific controls.
Method
The paper combines a scalable, low-coupling deep reinforcement-learning system with an actor-critic network using control decoupling, action masking, target attention, LSTM skill-combo modeling, and dual-clip PPO.
Results
Tencent Solo defeats top professional human players across different hero types in Honor of Kings 1v1 games.
Takeaways & Limitations
The framework supports effective training and deployment of an AI agent for complex action control in MOBA 1v1 games.
Abstract
from arXiv · showhide
We study the reinforcement learning problem of complex action control in the Multi-player Online Battle Arena (MOBA) 1v1 games. This problem involves far more complicated state and action spaces than those of traditional 1v1 games, such as Go and Atari series, which makes it very difficult to search any policies with human-level performance. In this paper, we present a deep reinforcement learning framework to tackle this problem from the perspectives of both system and algorithm. Our system is of low coupling and high scalability, which enables efficient explorations at large scale. Our algorithm includes several novel strategies, including control dependency decoupling, action mask, target attention, and dual-clip PPO, with which our proposed actor-critic network can be effectively trained in our system. Tested on the MOBA game Honor of Kings, our AI agent, called Tencent Solo, can defeat top professional human players in full 1v1 games.
Introduction
The paper targets MOBA 1v1 games as a highly complex reinforcement-learning setting, combining enormous state-action spaces with partially observable, multi-unit combat and hero-specific controls. It proposes a scalable deep reinforcement-learning framework with algorithmic innovations and reports that Tencent Solo defeats professional players.
- Motivation: MOBA 1v1 games present far more complicated environments and controls than traditional 1v1 games such as Go and Atari.Honor of Kings MOBA 1v1 involves state and action magnitudes of 10^600 and 10^18000, compared with 10^170 and 10^360 in Go.
- Motivation: Partially observable combat requires planning, attacking, defending, skill-combo control, target selection, and deception amid many game units.Different heroes also require substantially different control methods, while limited high-quality human data makes supervised learning unfeasible.
- Approach: The framework combines scalable deep reinforcement learning with a neural architecture for multimodal inputs, control-dependency decoupling, exploration pruning, and attack attention.The system and algorithm are designed for efficient exploration at massive scale in multi-agent competitive environments.
- Approach: The actor-critic network uses multi-label PPO, action masking, target attention, LSTM-based skill-combo modeling, and dual-clip PPO.These components address action control, target selection, exploration efficiency, temporal skill combinations, and training convergence.
- Results: Tencent Solo defeats top professional human players across different hero types in Honor of Kings 1v1 games.The evaluation tests the agent against professional and top-amateur players and compares it with existing MOBA 1v1 methods.
Preliminaries
The paper formalizes MOBA 1v1 control as a two-agent discounted Markov-game problem and situates it within competitive multi-agent reinforcement learning. It distinguishes this setting from prior game benchmarks and related strategy-focused environments.
- Problem formulation: The formal setting uses a two-agent Markov game that can be extended to multiple agents.The formulation includes state, observation, action, transition, reward, initial-state distribution, discount factor, and stochastic policy components.
- Problem formulation: Each agent seeks to maximize cumulative discounted reward returns under a stochastic policy mapping observations to actions.The discount factor is defined over an infinite-horizon discounted Markov Decision Process.
- Related work: Prior competitive reinforcement-learning studies commonly use 1v1 games including Go, Atari, Pong, simulated soccer, trivia, and MuJoCo environments.Related cooperative work includes recurrent Q-networks, 3D FPS agents, and supervised macro-strategy modeling for MOBA 5v5 games.
- Related work: This work studies MOBA 1v1 as a more complex competitive setting focused on agents’ fine-grained action control.Unlike StarCraft 1v1, which emphasizes controlling many units simultaneously, MOBA 1v1 emphasizes complex hero action control.
- Related work: The study evaluates full Honor of Kings 1v1 games and tests robustness across multiple hero types.The paper contrasts its scope with prior MOBA work and notes that different heroes can require substantially different playing methods.
System Design
The system separates experience generation, sample handling, and parameter learning into loosely coupled, scalable modules. This architecture supports large-batch distributed training and high-throughput exploration for complex multi-agent control.
- Architecture: The loosely coupled architecture consists of an RL Learner, AI Server, Dispatch Module, and Memory Pool.The design supports large batch sizes needed for high-variance stochastic gradients in complex control problems.
- System implications: The decoupled modules can be flexibly configured, allowing researchers to focus on algorithm design and environment logic.The authors state that the system design is also applicable to other multi-agent competitive problems.
- System modules: The AI Server runs environment-model interaction, generates self-play episodes, samples actions, and forwards them for game execution.It uses mirrored policies for self-play, opponent-policy sampling, and Boltzmann exploration based on extracted game features.
- System modules: The Dispatch Module collects, compresses, and transmits samples, while the Memory Pool stores variable-length samples in a memory-efficient circular queue.Samples include rewards, features, and action probabilities and can be sampled according to generation time.
- Distributed learning: Multiple RL Learners fetch data in parallel, average gradients with ring allreduce, and use shared memory to reduce communication cost.Shared-memory communication delivers a reported 2–3 times speed boost, while trained models are rapidly synchronized to AI servers.
- Architecture: The system separates experience generation from parameter learning to make AI servers and reinforcement-learning learners scalable with high throughput.Models are synchronized from the master learner to AI servers through peer-to-peer communication.
Algorithm Design
The algorithm models MOBA 1v1 control with an actor-critic network that integrates multimodal state inputs, target selection, temporal skill-combo modeling, and decoupled action components. Training efficiency is improved through action masking and dual-clip PPO for large-scale off-policy data.
- State and target modeling: The actor-critic network encodes image, unit, and game-state information, then uses target attention to select among variable numbers of units.Unit features are pooled by type, while the policy uses an LSTM and attention over unit encodings.
- State and action representation: Each MOBA action combines content, including buttons and movement parameters, with a target game unit.The content includes Move X, Move Y, Offset X, and Offset Y, alongside actions such as moving, attacking, and releasing skills.
- Control dependency decoupling: Action components are decoupled so each label is modeled independently, simplifying policy structure and increasing action diversity during exploration.The approach addresses correlations such as those between skill direction and skill type, while independent value channels diversify actions.
- Exploration control: An action mask prunes exploration by incorporating prior knowledge to eliminate physically forbidden or unreasonable action elements.The mask is introduced because decoupling further increases training complexity in MOBA games' large state and action spaces.
- Dual-clip PPO: Large policy-ratio deviations in off-policy training motivate dual-clip PPO, which adds a lower-bound clip when the advantage estimate is negative.The constant c > 1 specifies the lower bound, complementing the standard ratio clip with ϵ.
Experiments
Experiments evaluate Tencent Solo in Honor of Kings against professional and top-amateur human players, track training progression, and test key model and rollout components.
- Evaluation setup: The evaluation deploys the trained AI in Honor of Kings, predicts actions every 133 ms, and compares it with human players and established baseline methods.Baselines include behavior-tree AI, MCTS, and its variants; Elo ratings compare model versions.
- Professional evaluation: Five active professional players compete against the AI in best-of-five matches using heroes they specialize in.The experiment targets the highest level of hero control ability represented by professional players.
- Professional evaluation: The AI defeats professional players across different hero types; against the professional DiaoChan specialist, it wins 3:0.For DiaoChan, the AI averages 5 kills and 1.33 deaths per game, with an average game duration of 6 minutes and 56 seconds.
- Robustness evaluation: 99.81% win rate is achieved across 2,100 public matches against top human players, with only four losses.Five of eight showcased heroes achieve 100% win rates across hundreds of matches.
- Training progression: After about 6 hours, the AI reaches a 100% win rate against internal behavior-tree AI, reaches King Player level after 30 hours, and becomes comparable to professionals after 70 hours.The reported Elo growth rate is inversely proportional to training time.
- Ablation study: Full rollouts raise win rate to 70%∼80% versus partial rollouts, while random initial frames accelerate convergence by 15% but reduce win rate to 40% versus zero-start.The ablation compares full versus partial rollouts and random initial frames versus starting from the beginning of the game.
Conclusion and Future Work
The paper presents a scalable, off-policy DRL system and actor-critic algorithm for complex action control in MOBA 1v1 games. Tested in Honor of Kings, the resulting AI defeated top professional esports players, while future work focuses on broader access and research infrastructure.
- Conclusion: The framework combines scalable, off-policy DRL system architecture with an actor-critic multi-label network and dual-clipped PPO for MOBA 1v1 control.The algorithm includes strategies for modeling MOBA combats and ensuring convergence.
- Conclusion: The resulting AI defeated top professional esports players in full MOBA 1v1 games tested on Honor of Kings.
- Future Work: Future work includes open-sourcing the framework and algorithm, opening Honor of Kings’ game core, and providing some computing resources through a virtual cloud.The stated goal is to facilitate further research on complex games and public use of computing resources.
Appendix
The appendix describes MOBA 1v1 games, their objectives and hero-control demands, and supporting materials including the Honor of Kings interface and reward setup. It also notes limited beta access to the framework by November 21, 2019.
- MOBA 1v1 Games: MOBA is a type of real-time strategy game, with popular examples including Dota, Honor of Kings, and League of Legends.
- MOBA 1v1 Games: In formal mirror 1v1 matches, players select the same hero, control their own hero, and seek to destroy the opponent’s turrets and base while protecting their own.
- MOBA 1v1 Games: Different hero types have different playing methods, and each hero requires complicated esports micro-management.
- Access: By November 21, 2019, a beta version of the framework was open to four universities in China for user feedback.
- Interface: Figure 6 organizes the Honor of Kings interface into a mini-map, dashboard, movement controller, and ability controller.
- Reward: All trained 1v1 heroes use the same zero-sum reward, and the framework supports on-the-fly reward analysis during training.A DiaoChan training case is shown in Figure 7.