PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Deep Reinforcement Learning in Video Games

Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, Dongbin Zhao

arXiv:1912.10944v2cs.MAcs.AIcs.LG

TL;DR

DRL research needs methods that make decisions from high-dimensional inputs and that transfer across increasingly complex video-game settings. This survey compares value-based, policy-gradient, and model-based methods, reviews their game applications, and identifies persistent challenges. It reports human-level or super-human performance across games while emphasizing unresolved issues in exploration, efficiency, generalization, multi-agent learning, imperfect information, and sparse rewards.

  • Problem

    Video-game AI requires agents to make decisions in complex environments, while transfer, exploration, sample efficiency, imperfect information, and multi-agent learning remain challenging.

  • Method

    The paper surveys value-based, policy-gradient, and model-based DRL methods and reviews their applications across Arcade, first-person, and multi-agent real-time strategy games.

  • Results

    DRL agents achieve human-level or super-human performance across games ranging from 2D perfect-information to 3D imperfect-information and single-agent to multi-agent settings.

  • Takeaways & Limitations

    The survey concludes that high-level game AI needs more efficient and robust DRL techniques and novel frameworks, especially for 3D imperfect-information multi-agent games.

  • Takeaways & Limitations

    The field still faces unresolved challenges in exploration–exploitation, sample efficiency, generalization, multi-agent learning, imperfect information, and delayed sparse rewards.

Abstract

from arXiv · show

Deep reinforcement learning (DRL) has made great achievements since proposed. Generally, DRL agents receive high-dimensional inputs at each step, and make actions according to deep-neural-network-based policies. This learning mechanism updates the policy to maximize the return with an end-to-end method. In this paper, we survey the progress of DRL methods, including value-based, policy gradient, and model-based algorithms, and compare their main techniques and properties. Besides, DRL plays an important role in game artificial intelligence (AI). We also take a review of the achievements of DRL in various video games, including classical Arcade games, first-person perspective games and multi-agent real-time strategy games, from 2D to 3D, and from single-agent to multi-agent. A large number of video game AIs with DRL have achieved super-human performance, while there are still some challenges in this domain. Therefore, we also discuss some key points when applying DRL methods to this field, including exploration-exploitation, sample efficiency, generalization and transfer, multi-agent learning, imperfect information, and delayed spare rewards, as well as some research directions.

I. INTRODUCTION

The introduction frames video games as safe, controllable, data-rich environments for game-AI research and presents DRL as an end-to-end approach to high-dimensional decision-making. It highlights challenges involving large state spaces, dynamic policies, and transfer across games.

  • I. INTRODUCTION: Video games offer safe, controllable, fast environments with an effectively continuous supply of data for AI research.Their complexity makes them useful settings for studying interactions between agents and environments.
  • I. INTRODUCTION: Game AI combines perception and decision-making, but strategic games create especially large state spaces.Deep neural networks have been used to model large-scale state spaces through representation learning.
  • I. INTRODUCTION: Learning suitable policies in dynamic, unknown environments is difficult, motivating data-driven approaches such as supervised learning and reinforcement learning.The introduction identifies these methods as feasible solutions to policy-learning difficulty.
  • I. INTRODUCTION: The paper identifies transfer across different games as a core challenge because most game AI is developed for a specified virtual environment.It also calls for more general learning systems.
  • I. INTRODUCTION: DRL combines deep learning and reinforcement learning to make decisions from high-dimensional state spaces in an end-to-end framework.The paper presents this combination as improving the generalization and scalability of traditional reinforcement learning algorithms.

B. Reinforcement learning

Reinforcement learning models sequential decision-making through agent–environment interaction, with policies learned to maximize discounted cumulative reward. The section distinguishes major learning paradigms and introduces value-based, policy-gradient, and actor–critic formulations.

  • B. Reinforcement learning: In an MDP, an agent selects actions from states, receives rewards, and transitions to new states according to the environment.The policy specifies the action distribution π(a_t|s_t).
  • B. Reinforcement learning: The objective of reinforcement learning is to maximize expected discounted cumulative rewards.Discounting weights future rewards in the return objective.
  • B. Reinforcement learning: Off-policy methods learn from a policy different from the behavior policy, whereas on-policy methods use the same policy for both.This distinction concerns how collected experience relates to the learning policy.
  • B. Reinforcement learning: Value-based reinforcement learning learns a value function and derives a policy, while policy-based reinforcement learning learns the policy directly.Q-learning is presented as a typical off-policy, value-based method.
  • B. Reinforcement learning: Policy-gradient methods parameterize the policy and update its parameters using accumulated return.Actor–critic methods augment policy gradients with a value-based critic, such as a value, advantage, or temporal-difference estimate.

III. DEEP REINFORCEMENT LEARNING

The paper surveys DRL as a combination of deep learning and reinforcement learning, emphasizing value-based methods and their extensions for stability, representation, delayed rewards, distributional values, and imperfect information.

  • III. DEEP REINFORCEMENT LEARNING: DRL combines deep learning and reinforcement learning, with the survey organized around value-based, policy-gradient, and model-based methods.The supplied section primarily develops value-based DRL methods.
  • III. DEEP REINFORCEMENT LEARNING: DQN maps raw pixel inputs to value estimates for future rewards using experience replay and a target Q-network to stabilize learning.It is described as learning policies directly from high-dimensional inputs.
  • III. DEEP REINFORCEMENT LEARNING: Double DQN reduces observed overestimation, prioritized replay replays important transitions more often, and Dueling DQN separates state-value and advantage estimation.These extensions target different aspects of DQN performance and representation.
  • III. DEEP REINFORCEMENT LEARNING: DRQN replaces DQN’s first fully connected layer with a recurrent network to address limited memory and imperfect information at each decision point.The recurrent architecture is presented as a response to partially observed game information.
  • III. DEEP REINFORCEMENT LEARNING: Rainbow combines major DQN extensions and measures each component’s contribution, while RUDDER targets finite MDPs with delayed rewards.Ape-X DQfD additionally uses human demonstrations to ease exploration and processes rewards with varying densities and scales.
  • III. DEEP REINFORCEMENT LEARNING: Distributional DRL models the value distribution rather than only the expected return, as exemplified by C51 and QR-DQN.The section presents this as an explicit modeling of return distributions.

B. Policy gradient DRL methods

Policy-gradient DRL directly optimizes parameterized policies, often using actor–critic critics and asynchronous or parallel training. The surveyed extensions target exploration, data efficiency, off-policy learning, and computational efficiency.

  • B. Policy gradient DRL methods: Actor–critic architectures compute policy gradients with a value-based critic that estimates expected future rewards.Asynchronous DRL uses asynchronous gradient descent, and A3C trains agents across multiple environments.
  • B. Policy gradient DRL methods: The actor objective includes an entropy term to encourage exploration.The entropy term is explicitly identified as H_θ(π(s_t)).
  • B. Policy gradient DRL methods: UNREAL improves data efficiency and robustness by learning auxiliary policies for pseudo-rewards including value replay, reward prediction, and pixel control.These auxiliary objectives are trained alongside the main policy.
  • B. Policy gradient DRL methods: PGQ combines policy gradients with Q-learning, while Retrace(λ) combines importance sampling, off-policy Q(λ), and tree-backup(λ).The section associates Retrace(λ) with low variance, safety, and efficiency.
  • B. Policy gradient DRL methods: Reactor is a sample-efficient, numerically efficient off-policy actor–critic agent whose network outputs a target policy, action-value function, and behavioral-policy estimate.Its architecture uses multi-step returns.

1) Trust region method:

Trust-region methods constrain policy updates to improve optimization stability, while related actor-critic methods emphasize simpler implementation or sample efficiency.

  • Trust region method:: TRPO constrains each policy update to ensure a small change in the policy distribution and guaranteed monotonic improvement.
  • Trust region method:: PPO clips the probability ratio in its surrogate objective, providing simpler implementation and better sample complexity than TRPO.
  • Trust region method:: ACER combines stochastic dueling networks, truncated importance sampling, and a trust-region method to improve stability and sample efficiency.
  • Trust region method:: ACKTR uses natural policy gradients and Kronecker-factored approximate curvature with a trust region for actor and critic optimization.

2) Deterministic policy:

Deterministic and entropy-regularized policy methods address continuous control, exploration, and sample efficiency, while model-based planning augments learning with predicted transitions and values.

  • 2) Deterministic policy:: DDPG is an off-policy actor-critic deterministic policy-gradient method that adapts DQN’s success to continuous control.
  • 2) Deterministic policy:: D4PG extends DDPG with distributional updates and distributed workers sharing a replay table, improving performance on difficult continuous-control problems.
  • 2) Deterministic policy:: SAC is an off-policy policy-gradient method bridging DDPG and stochastic policy optimization through clipped double-Q estimation.
  • 2) Deterministic policy:: SAC maximizes a trade-off between entropy and expected return, encouraging exploration and helping prevent convergence to poor local optima.
  • 2) Deterministic policy:: TreeQN and ATreeC combine learned transition models with differentiable tree planning and outperform n-step DQN and value-prediction networks on multiple Atari games.
  • 2) Deterministic policy:: MuZero combines Monte Carlo tree search with a learned model that predicts rewards and action-selection information.

IV. DRL IN VIDEO GAMES

DRL game research spans standardized platforms, competitions, and environments ranging from 2D to 3D and from single-agent to multi-agent settings.

  • IV. DRL IN VIDEO GAMES: DRL agents have achieved impressive progress across video games spanning 2D and 3D environments and single-agent and multi-agent settings.
  • A. Game research platforms: Platforms and competitions support game-AI development and provide settings for evaluating agent intelligence.
  • A. Game research platforms: General platforms include ALE, Gym, OpenAI Universe, Gym Retro, and Unity ML-Agents, supporting broad collections of games or flexible simulation.
  • A. Game research platforms: ALE provides interfaces to many Atari 2600 games and serves as a major evaluation testbed for DRL research.
  • A. Game research platforms: The surveyed methods and competitions are organized in separate reviews covering recent DRL methods and game-AI competitions suitable for DRL research.
  • A. Game research platforms: Specific platforms target environments such as Minecraft, racing, first-person shooters, and StarCraft for specialized DRL research.

B. Atari games

Atari games have served as a central evaluation domain for DRL, with methods achieving strong performance across many games. Montezuma’s Revenge highlights the importance of exploration when rewards are sparse and delayed.

  • ALE evaluates DRL agents across Atari 2600 games and is a widely used testbed for general intelligence.Its evaluation includes difficult tasks such as Montezuma’s Revenge.
  • DQN achieved human-level performance across 49 Atari games, establishing a milestone for DRL in this domain.
  • UNREAL averaged 880% expert human performance, while DQfD exceeded DQN’s average rewards on 27 of 42 games and learned faster with poor demonstrations.
  • Rainbow improved sample efficiency and final performance, while later distributional and distributed methods further improved Atari results.Ape-X DQfD exceeded average human performance on 40 of 42 Atari games using common hyperparameters.
  • Montezuma’s Revenge requires long action sequences before rewards, making efficient exploration crucial for learning an effective policy.

2) TORCS:

The survey presents DRL applications across increasingly complex game environments, including realistic racing, 3D first-person games, Minecraft, and StarCraft.

  • TORCS: TORCS requires agents to control acceleration, braking, and steering while learning realistic vehicle dynamics.
  • TORCS: FIGAR-DDPG completed 20 circuit laps with 10× the total reward of DDPG and smoother policies.
  • Minecraft: Minecraft provides an infinitely varied 3D platform supporting navigation, problem solving, survival, and collaboration scenarios.
  • Minecraft: Hierarchical DRL with reusable skills achieved superior performance and lower learning sample complexity than regular DQN in Minecraft.The approach also showed potential to transfer knowledge between related Minecraft tasks without additional learning.
  • DeepMind Lab: UNREAL achieved a 10× learning speedup over A3C and averaged 87% expert human performance on challenging DeepMind Lab tasks.
  • StarCraft: DRL methods in StarCraft address micromanagement, build-order optimization, relational reasoning, and cooperative multi-agent control.Reported systems include BiCNet, decentralized actor-critic methods, TStarBot, and relational DRL agents.

2) MOBA and Dota2:

The survey describes DRL progress in MOBA games, culminating in successful self-play systems for both small-team and full-team Dota2 scenarios.

  • MOBA and Dota2: DRL self-play succeeded in both 1v1 and 2v2 Dota2 scenarios and later extended to 5v5 play.
  • MOBA and Dota2: OpenAI Five used an LSTM-centered network and PPO with massively distributed training to develop team-fighting and other coordinated abilities.
  • MOBA and Dota2: OpenAI Five defeated human champion OG 2:0.
  • Challenges: DRL for video games remains challenged by exploration, sample efficiency, generalization, multi-agent learning, imperfect information, and delayed sparse rewards.
  • Challenges: Parametric noise, randomized value functions, and hash-based count bonuses are described as exploration strategies for high-dimensional environments.

B. Sample efficiency

The survey identifies data inefficiency as a major DRL limitation and reviews hierarchy, demonstrations, transfer, distillation, and curriculum learning as remedies.

  • Sample efficiency: DRL algorithms usually require millions of samples for human-level performance, making model-free methods especially data inefficient in high-dimensional environments.
  • Hierarchy: Hierarchical reinforcement learning decomposes tasks into subtasks, which can speed training and improve sample efficiency.
  • Hierarchy: The option-critic architecture learns internal policies and option termination conditions without additional rewards or subgoals.
  • Demonstration: Demonstration-based learning can improve sample efficiency but is difficult to optimize jointly and sensitive to noisy demonstrations.
  • Transfer: Multi-task learning and policy distillation target knowledge transfer across environments and related tasks.
  • Transfer: Distral and Mix & Match support transfer by sharing distilled knowledge or progressively training more complex agents from simpler ones.

D. Multi-agent learning

The survey treats multi-agent coordination, imperfect information, memory, and sparse rewards as interconnected challenges for DRL in video games.

  • Multi-agent learning: Cooperative multi-agent learning faces curse-of-dimensionality, communication, and credit-assignment challenges.
  • Multi-agent learning: Centralized training with decentralized policies is becoming a standard paradigm for multi-agent training.
  • Imperfect information: Partially observable and first-person games require agents to handle imperfect information when learning policies.
  • Imperfect information: Memory architectures such as past frames and LSTM layers help agents retain information but may capture only transitory information.
  • Delayed rewards: Sparse and delayed rewards reduce reinforcement-learning sample efficiency.
  • Conclusion: The survey concludes that major problems remain, especially in 3D imperfect-information multi-agent games.
Loading 1912.10944v2…