PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Games

Zhengyang Li, Qijin Ji, Xinghong Ling, Quan Liu

arXiv:2509.03682v1cs.LG

TL;DR

MARL research in video games spans increasingly complex environments, but evidence and practical frameworks for comparing these applications remain fragmented. This review synthesizes MARL studies across game genres, analyzes challenges and industry barriers, and proposes a reproducible complexity framework. The reviewed applications include strong results such as a 95.2% win rate over professionals in 42 hero-drafting matches and benchmark advances for Rocket League AI.

  • Problem

    Research on MARL in video games spans complex environments, but comparing game complexity objectively is difficult because traditional measures are subjective or production-oriented.

  • Method

    The paper reviews MARL applications across video-game genres, examines implementation and industrial-adoption challenges, and introduces a five-dimension game-complexity classification.

  • Results

    95.2% win-rate over 42 matches against professionals was achieved by MCTS for hero selection, alongside benchmark advances for Rocket League AI.

  • Takeaways & Limitations

    The review provides a practical and reproducible framework for comparing MARL performance across diverse video-game environments.

  • Takeaways & Limitations

    A comprehensive, universally applicable measure of game complexity is beyond the scope of the review.

Abstract

from arXiv · show

Recent advancements in multi-agent reinforcement learning (MARL) have demonstrated its application potential in modern games. Beginning with foundational work and progressing to landmark achievements such as AlphaStar in StarCraft II and OpenAI Five in Dota 2, MARL has proven capable of achieving superhuman performance across diverse game environments through techniques like self-play, supervised learning, and deep reinforcement learning. With its growing impact, a comprehensive review has become increasingly important in this field. This paper aims to provide a thorough examination of MARL's application from turn-based two-agent games to real-time multi-agent video games including popular genres such as Sports games, First-Person Shooter (FPS) games, Real-Time Strategy (RTS) games and Multiplayer Online Battle Arena (MOBA) games. We further analyze critical challenges posed by MARL in video games, including nonstationary, partial observability, sparse rewards, team coordination, and scalability, and highlight successful implementations in games like Rocket League, Minecraft, Quake III Arena, StarCraft II, Dota 2, Honor of Kings, etc. This paper offers insights into MARL in video game AI systems, proposes a novel method to estimate game complexity, and suggests future research directions to advance MARL and its applications in game development, inspiring further innovation in this rapidly evolving field.

I. INTRODUCTION

Video games have evolved toward complex, real-time multiplayer environments, increasing demand for adaptive multi-agent AI. This review surveys MARL applications across game genres, challenges, and complexity dimensions.

  • Video games have shifted from primarily single-player or turn-based titles toward real-time multiplayer formats across MOBA, RTS, and FPS genres.
  • Existing game AI techniques such as finite state machines and behavior trees increasingly face demands for more intelligent cooperative teammates and competitive opponents.
  • Reinforcement learning fits video games because they provide defined environments, actions, and reward structures for learning through interaction.
  • Deep reinforcement learning extends RL with neural networks to handle large state spaces and has achieved strong performance in games and 3D navigation.
  • Because modern games involve multiple interacting agents, researchers use MARL to model cooperative and competitive interactions beyond primarily single-agent DRL applications.
  • The review covers two-agent and multi-agent games across sports, FPS, RTS, and MOBA genres while examining observability, coordination, rewards, and scalability challenges.It proposes estimating game complexity using Observability, State Space, Action Space, Reward Sparsity, and Multi-Agent Scale.

B. Reinforcement Learning (RL)

Reinforcement learning models sequential decision-making through states, actions, transitions, rewards, and policies. Deep and policy-based methods extend this framework to high-dimensional visual inputs, continuous actions, and stable optimization.

  • An MDP represents RL with states, actions, transition probabilities, rewards, and a discount factor governing sequential decision-making.
  • Q-learning estimates state-action values and updates them iteratively to learn an optimal policy.
  • Policy-gradient methods directly learn a parameterized policy by maximizing expected cumulative reward, with the discount factor controlling future-reward importance.
  • Deep reinforcement learning uses neural networks such as CNNs, RNNs, and LSTMs to approximate policies or value functions from visual or sequential data.
  • DQN scales Q-learning to high-dimensional state spaces by estimating Q-values from raw pixels with a deep network and stabilizing learning through experience replay.
  • Actor-critic methods combine policy learning with value estimation, while DDPG targets continuous actions and PPO stabilizes updates with a clipped objective.

D. Multi-Agent Reinforcement Learning (MARL)

MARL extends RL to multiple agents interacting with one another and the environment in competitive, cooperative, or mixed settings. Self-play and centralized training with decentralized execution are prominent approaches for coordination and deployment.

  • MARL models multi-agent environments as Markov Games in which agents follow individual policies while jointly influencing state transitions and rewards.
  • Multi-agent interactions may be competitive, cooperative, or mixed, depending on whether rewards oppose, align, or combine individual objectives.
  • Self-play trains agents against different versions of themselves and has been applied from TD-Gammon to StarCraft II and Dota 2.
  • CTCE uses shared global information during training and execution but is often impractical for real-time systems because of scalability and communication constraints.
  • CTDE trains agents with global or shared information while enabling decentralized execution from local observations, balancing coordination with deployability.
  • Across reviewed games, CTDE is the dominant training methodology because it supports coordinated learning while meeting real-time execution constraints.

III. METHODOLOGY

The review uses a curated literature corpus to systematically analyze MARL in competitive and cooperative video games. Its scope includes two-agent and multi-agent environments while excluding single-agent and non-video-game studies.

  • The review corpus contains 84 reports, including 40 core studies for synthesis and 44 supplementary works for background and methodological context.
  • The methodology systematically identifies, organizes, and analyzes research applying MARL in competitive or cooperative multi-agent video game settings.
  • The review includes both two-agent and multi-agent environments but excludes studies focused solely on single-agent systems or non-video-game environments.

B. Data Sources and Search Strategy

The review constructs its literature corpus through expert-guided database searches, explicit inclusion and exclusion criteria, and documented screening. It acknowledges selection bias from citation chaining and subjectivity from manual filtering.

  • Search Strategy: The literature search combined professional expertise with structured searches across major academic databases for MARL and video-game studies.Search terms combined “multi-agent,” “reinforcement learning,” and “video games,” tailored to each platform’s query syntax.
  • Screening: The review applied tailored inclusion and exclusion criteria to maintain consistency and relevance across reports.The criteria depended on each report’s intended role within the review structure.
  • Inclusion Criteria: Core studies were required to apply MARL in one or more real-time or turn-based video-game environments and provide implementation or experimental details.Eligible reports included model architecture, training algorithm, reward structure, or experimental results.
  • Limitations: Citation chaining may bias selection toward more visible or recent studies, while manual filtering introduces subjectivity.The authors mitigate these risks by documenting sources and selection criteria.

C. Designing Team Incentive Mechanisms

Team-based video games require incentive mechanisms, communication, and credit assignment that align individual behavior with collective objectives. The review frames these coordination problems within a five-dimensional complexity classification for comparing MARL environments.

  • Team Incentives: Team objectives can conflict with individual incentives, so mechanisms must discourage self-benefiting actions that harm overall team success.This issue arises in both competitive team-wise zero-sum games and cooperative multiplayer settings.
  • Communication and Coordination: Communication helps agents share local observations and coordinate joint actions such as attacks, defense, and broader strategies.This is especially important under partial observability.
  • Credit Assignment: Credit assignment becomes difficult when teams execute long-horizon strategies whose success depends on contributions distributed across many agents and time steps.Examples include terrain control and expansion in strategy games.
  • Scalability: As agent counts grow, interaction-management complexity increases exponentially, creating a need for scalable MARL algorithms.The review contrasts Go’s estimated 10^170 state space with MOBA environments reaching 10^20,000.
  • Complexity Framework: The proposed complexity framework evaluates environments along observability, state space, action space, reward sparsity, and multi-agent scale.These dimensions are derived from the MDP framework and target learning-oriented game complexity rather than production measures.

C. Scope and Limitations

The review spans foundational two-agent games and later MARL-relevant achievements, while organizing game complexity for systematic comparison. Its framework is practical and reproducible, but not intended as a universal measure of real-time game complexity.

  • Scope and Limitations: Real-time video games are typically partially observable, multi-agent, stochastic, sequential, dynamic, continuous, and unknown.These properties make them especially challenging learning environments.
  • Scope and Limitations: The proposed complexity framework supports reproducible comparisons across diverse game environments but is not intended as a comprehensive universal complexity measure.The authors apply it to organize the review by increasing complexity.
  • Two-Agent Games: The review begins with two-agent games across cooperative, competitive, and mixed settings before transitioning toward real-time video games.Backgammon and Go provide foundational examples despite not being real-time video games.
  • Backgammon and TD-Gammon: TD-Gammon used an MLP value function and self-play to achieve master-level Backgammon play close to the world’s best human players.The system learned from board configurations and generated training data autonomously through self-play.
  • Backgammon and TD-Gammon: Self-play in TD-Gammon established a foundation for later reinforcement-learning systems, including DQN and AlphaGo.The cited discussion connects this early success to subsequent applications of reinforcement learning and neural networks.
  • Go and AlphaGo: AlphaGo combined supervised learning, reinforcement learning through self-play, and MCTS, defeating the European Go champion 5 games to 0.The approach later influenced systems such as AlphaStar, OpenAI Five, and Honor of Kings.

C. Blade & Soul

The review contrasts real-time, multi-agent game complexity with two-agent and sports-game settings, emphasizing rapid decisions, coordination, and structured complexity dimensions. It surveys MARL implementations spanning Blade & Soul, 3v3 Snake, Google Research Football, Roller Champions, and Rocket League.

  • Blade & Soul: Blade & Soul requires continuous millisecond-scale decisions under imperfect information, with a large 3D action space and highly dependent moves, skills, and strategies.Its one-on-one mode uses an LSTM-based model with four decision-making heads and produced pro-level agents capable of competing with human players.
  • Competitive and Sports Games: 3v3 Snake combines within-team cooperation with competition against another team, using a territory matrix to incorporate human-designed strategic rules into MARL.Its agents consistently outperform both rule-based algorithms and human players.
  • Competitive and Sports Games: Google Research Football supports whole-team control or multi-agent control of individual players, while intermediate checkpoint rewards address sparse scoring feedback.The platform supports IMPALA, PPO, and Ape-X DQN, and includes progressively structured Football Academy scenarios.
  • Competitive and Sports Games: Roller Champions applies MARL to human collaboration, competition, game balance, skill adaptation, and replacing disconnected players in a fast-paced team sport.Rocket League similarly supports team-based coordination through dynamic attacker, receiver, and defender roles that adapt to game states.

B. First-Person Perspective Games

First-person perspective games provide demanding MARL environments with real-time control, visual complexity, and mixed cooperation and competition. Studies span Doom, Minecraft, and Quake III Arena, with agents achieving strong performance through curriculum learning, reward shaping, self-play, and multi-timescale architectures.

  • Game characteristics: FPP games place agents in first-person 3D worlds requiring real-time decisions as single entities.Continuous action spaces and high-dimensional visual inputs further increase difficulty.
  • Doom and ViZDoom: The F1 agent used A3C and curriculum learning, progressively introducing harder maps and stronger opponents in ViZDoom Deathmatch.It achieved state-of-the-art performance, including winning Track 1 of the ViZDoom AI Competition.
  • Minecraft: Minecraft supports mixed-setting MARL because players can cooperate, compete, and change roles within a shared open-world sandbox.Project Malmo provides an API through which multiple agents perceive observations and rewards and act in real time.
  • Quake III Arena: Quake III Arena CTF offers procedurally generated environments and adjustable map sizes for studying strategic multi-agent cooperation and competition.Teams capture the opponent’s flag while defending their own in 3D maze-like levels.
  • Quake III Arena: FTW agents captured an average of 16 more flags per game than humans on previously unseen procedurally generated maps.The system combined self-play, population-based training, distributed actor-learner training, reward shaping, and fast and slow LSTMs.
  • Quake III Arena: FTW agents achieved 258-millisecond reaction times and 80% tagging accuracy, compared with 559 milliseconds and 48% for humans.They also exceeded human performance across varied CTF scenarios, with over 70% win probability in most reported conditions.

C. RTS and MOBA Games

RTS and MOBA games expose MARL to simultaneous decisions, partial observability, large action spaces, and demanding team coordination. AlphaStar, OpenAI Five, and Honor of Kings demonstrate progress from strategic self-play to superhuman performance and larger hero pools.

  • Genre characteristics: RTS games involve simultaneous real-time play, resource management, base building, and control of multiple units, whereas MOBAs emphasize one hero and team coordination.Both genres require strategic decisions under partial observability.
  • StarCraft II: StarCraft II limits information through local camera views, a high-level minimap, and fog of war, making exploration necessary to infer opponents’ states.The minimap provides battlefield overview, movement tracking, navigation, and team communication through pings.
  • StarCraft II: AlphaStar used the StarCraft II Learning Environment and Prioritized Fictitious Self-Play, including main agents, main exploiters, and league exploiters.The approach targets difficult opponents and weaknesses while facing possible non-transitive strategy loops among three countering races.
  • Dota 2: OpenAI Five’s Dota 2 success depended on team coordination, macro strategies, and sequential decision-making across five-agent teams.Training addressed lane assignments, hero-lineup diversity, and team incentives using penalties, randomized lineups, and a Team Spirit hyperparameter.
  • Honor of Kings: Honor of Kings achieved 95.2% wins in 42 matches against professional esports players and 97.7% wins in 642,047 matches against top-ranked human players.Its system supported 40 heroes, compared with OpenAI Five’s 17-hero limitation, while addressing hero-selection complexity.
  • Honor of Kings: Honor of Kings has 1.56 × 10^16 possible lineup combinations with 113 heroes, making complete Minimax tree search computationally infeasible.The drafting system required substantial resources, including 320 GPUs and 35,000 CPUs.

VIII. DISCUSSION

MARL has achieved notable research results, but its integration into commercial video-game development remains limited by production constraints, design priorities, and control requirements.

  • Industry adoption: Commercial adoption of MARL remains limited because industry requirements differ from academic assumptions.The discussion identifies production constraints, design priorities, and control requirements as central barriers.

A. Industry and Academia Gap

The game industry generally favors predictable, controllable Game AI over research-oriented MARL, while superhuman performance may conflict with enjoyable player experiences. Bridging this gap requires attention to adaptability, cost, and human-like interaction.

  • Industry and academia gap: Most game studios still use handcrafted, predictable, and manageable Game AI rather than advanced MARL.Designer control and predictability take precedence over more powerful but unpredictable behaviors.
  • Industry and academia gap: Advanced AI techniques increase development costs and are often tied to specific game environments rather than reusable across projects.Generalist agents handling multiple roles, modes, or games are suggested as one bridge between research and production.
  • Player experience: Superhuman opponents can be undesirable in games because unbeatable behavior may reduce enjoyment and mental challenge.Human-like agents that adapt to player skill could support cooperation and fair competition.

C. Creating Designer Centric RL

Designer-centric reinforcement learning must balance adaptive behavior with the game industry's need for control, predictability, and manageable costs. The review identifies accessibility and genre breadth as key conditions for broader adoption.

  • Designer constraints: Game designers prioritize player experience, narrative alignment, and predictable behavior over purely optimal solutions.Behavior trees and finite state machines provide control, whereas reinforcement learning offers adaptation but can conflict with design constraints.
  • Genre scope: MARL has demonstrated superhuman performance in Sports, FPS, MOBA, and RTS games, while other genres demand broader management of game systems.Civilization VI, for example, requires diplomacy, economics, construction, military strategy, and resource management.
  • Accessibility: Accessible toolkits and cost-efficient training methods are needed because MARL currently remains concentrated in large studios with extensive resources.Pretrained models, cloud-based platforms, and general-purpose algorithms are proposed to lower entry barriers for smaller studios.

IX. CONCLUSION

The review surveys MARL across game genres and highlights representative methods and outcomes, including strong performance in Rocket League, StarCraft II, and Dota 2-related settings. It concludes that adoption remains constrained by engineering complexity, resource demands, and the industry's preference for controllable AI.

  • Scope: The review covers MARL applications across two-agent and multi-agent games in Sports, FPS, RTS, and MOBA genres.Its scope includes both research advances and challenges in applying reinforcement learning within game development.
  • Challenges and outlook: Industry adoption remains difficult because developers prioritize control, predictability, and budget, while MARL is resource-intensive and often limited to large studios.The review points to accessible tools and future research as ways to broaden deployment across game genres.
  • Competitive and Sports Games: Rocket League research reports performance exceeding Necto and Nexto, establishing benchmarks for Rocket League AI.The reviewed approaches include synchronous and asynchronous modes and vision-based reinforcement learning from raw visual input.
  • RTS and MOBA: League training with PFSP enabled an agent to beat 99.8% of human players.This result is listed alongside PBT, FTW Agent, and Temporal Hierarchy methods.
  • RTS and MOBA: Large-scale distributed self-play and the “Surgery” technique produced agents that beat world champions.The conclusion presents this as a representative multi-agent training achievement.
  • RTS and MOBA: A Dota 2-related system achieved a 95.2% win rate over 42 matches against professionals using MCTS for hero selection.The reviewed methods also include CTDE, Actor-Critic, and Dual-clip PPO.
Loading 2509.03682v1…