PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Video Game Playing

Niels Justesen, Philip Bontrager, Julian Togelius, Sebastian Risi

arXiv:1708.07902v3cs.AI

TL;DR

Video-game-playing research must address genre-specific requirements, including generalization across games, sparse rewards, and large decision spaces. The paper reviews deep learning and related approaches across game genres, concluding that simple arcade games can reach above-human performance while harder settings remain challenging.

  • Problem

    Video-game genres impose diverse requirements, while general game playing, extremely large decision spaces, and sparse rewards remain open challenges.

  • Method

    The paper reviews deep learning methods and related evolutionary, supervised, and model-based approaches across multiple video-game genres and environments.

  • Results

    For simple games such as most arcade games, reviewed methods can achieve above-human-level performance, while sparse-reward and broader general-game settings remain difficult.

  • Takeaways & Limitations

    Progress in game playing is uneven across genres, with strong results in simple arcade games but continuing challenges in generalization and reward-guided learning.

Abstract

from arXiv · show

In this article, we review recent Deep Learning advances in the context of how they have been applied to play different types of video games such as first-person shooters, arcade games, and real-time strategy games. We analyze the unique requirements that different game genres pose to a deep learning system and highlight important open challenges in the context of applying these machine learning methods to video games, such as general game playing, dealing with extremely large decision spaces and sparse rewards.

I. INTRODUCTION

The article surveys deep learning for video game playing across genres, platforms, and learning paradigms. It emphasizes genre-specific challenges and the scope limits of its review.

  • Scope: The review examines deep learning for playing Atari/ALE, Doom, Minecraft, StarCraft, and car-racing games.Its focus is playing particular games well from pixels or feature vectors without an existing forward model.
  • Scope: Different game genres pose diverse challenges for human and machine players.The review uses genre comparisons to identify these requirements.
  • Scope limitation: The article excludes many Game AI uses, including believable behavior, player modeling, content generation, and non-deep-learning methods.This scope boundary is explicitly identified as a limitation of the review.
  • Learning approaches: The paper covers supervised, unsupervised, and reinforcement learning, alongside evolutionary and hybrid approaches.These methods are presented as optimization-based approaches commonly applied to game-playing behaviors.
  • Neural architectures: Neural networks provide general function approximators, with feedforward and recurrent architectures supporting game-state representations and temporal context.Recurrent networks can use previous hidden states when a single observation does not fully represent the game state.
  • Supervised learning: Supervised learning can map game states to demonstrated actions, but available play-trace data may be expensive and incomplete.Learning state transitions from action-state pairs can instead produce a model for planning or improved play.

2) Unsupervised Learning

Unsupervised learning is presented as a potential response to sparse-reward games, while reinforcement learning learns behavior through reward-driven interaction. The review explains model-based, temporal-difference, policy-gradient, and actor-critic approaches, including their scalability limits.

  • Unsupervised Learning: Unsupervised learning may help games with sparse rewards by discovering useful data representations without labeled actions.The paper identifies this as an important open deep learning challenge.
  • Unsupervised Learning: Autoencoders compress high-dimensional game data through an encoder-decoder structure, but applications to games remain at an early stage.A low-dimensional hidden representation is learned by reconstructing the input.
  • Reinforcement Learning: Reinforcement learning trains an agent through environment interaction, using rewards such as game-score changes to update behavior.The objective is to maximize accumulated rewards.
  • Reinforcement Learning: Sparse rewards create a credit-assignment problem because rewards must be propagated back to the many actions that preceded them.Open-world games are especially challenging when they lack a clear reward model.
  • Model-based methods: Adaptive Dynamic Programming can handle credit assignment through a learned transition model, but building a probability tree over large game spaces is intractable.The method learns P(s′|s, a) and computes utilities using future-state transitions.
  • Model-free methods: Temporal-difference learning models utilities directly and propagates rewards backward over time without learning a transition model.Its utilities converge to the same values as Adaptive Dynamic Programming, albeit more slowly.
  • Model-free methods: Q-learning updates state-action values toward the best-known future action, whereas SARSA updates using the next action selected by the agent’s policy.Q-learning is off-policy, while SARSA is on-policy.
  • Policy methods: Policy-gradient methods optimize action probabilities using discounted cumulative rewards, and actor-critic methods combine policy updates with temporal-difference value estimation.The actor learns the policy while the critic approximates the reward-related value.

4) Evolutionary Approaches

Evolutionary approaches offer derivative-free alternatives to gradient descent for training game-playing networks. Hybrid methods combine deep representations with evolutionary optimization or planning, potentially supporting learning across different timescales.

  • Evolutionary Approaches: Neuroevolution optimizes neural-network weights and topology without requiring differentiability.It can be applied to supervised, unsupervised, and reinforcement learning problems.
  • Evolutionary Approaches: Evolution Strategies use stochastic parameter exploration and can achieve results competitive with deep reinforcement learning in Atari given sufficient computation.Their high parallelizability is a central practical advantage.
  • Hybrid approaches: Hybrid game-playing systems have combined gradient-trained deep networks with networks trained through artificial evolution.These systems use condensed feature representations from the deep network as inputs to the evolved network.
  • Hybrid approaches: Hybrid approaches can also combine deep neural networks with tree search or predictive-model planning.AlphaGo is cited as an example of deep networks combined with tree search.
  • Hybrid approaches: Combining ontogenetic reinforcement learning with phylogenetic evolutionary methods may support concurrent learning on different timescales.The paper presents this as a potential impact of hybridization.

III. GAME GENRES AND RESEARCH PLATFORMS

Video-game research platforms span classic arcade, racing, FPS, and other environments, each imposing distinct perceptual, planning, observability, and action-control challenges.

  • Research platforms: Game environments are used to compare game-playing algorithms by scores or wins, extending benchmarking beyond classic control problems.Competitions use publicly available game environments to rank methods, while newer video games are generally more demanding than earlier reinforcement-learning benchmarks.
  • Arcade games: ALE provides more than 50 Atari 2600 games with score, 160×210 screen pixels, and RAM available as agent inputs.Its visual-input setting distinguishes it from classic control tasks such as Cart Pole and Mountain Car.
  • Arcade games: RLE contains seven SNES games with over 720 controller action combinations, making them more complex and realistic than Atari 2600 games.Despite this broader action space and 3D graphics, RLE has been less popular than ALE.
  • Racing games: Racing requires fine-tuned continuous steering, acceleration, and braking, with short-term planning and sometimes longer-term resource management.Other vehicles add adversarial planning under hidden information about opponents’ positions and resources.
  • First-person shooters: FPS environments combine 3D, partially observable states with fast perception, aiming, navigation, adversary prediction, and sometimes team collaboration.Visual agents must extract relevant information from pixels while responding quickly to enemies.
  • First-person shooters: ViZDoom uses Doom’s screen buffer, while DeepMind Lab supports 3D navigation and puzzle-solving tasks based on the Quake III Arena engine.These platforms provide visual reinforcement-learning environments beyond classic arcade benchmarks.

D. Open-World Games

Open-world, strategy, sports, and text-adventure environments broaden game-playing challenges through nonlinear goals, enormous planning spaces, multiple agents, language interaction, and benchmarking gaps.

  • Open-world games: Open-world games provide large environments, nonlinear gameplay, unclear goal ordering, and broad freedom of action.Agents must explore these worlds and set goals that are realistic and meaningful.
  • Strategy games: Strategy games require complex multi-unit plans with enormous branching factors, extremely long horizons, and prediction of multiple adversaries.RTS games add time prioritization because actions can occur continuously rather than in discrete turns.
  • Real-time strategy games: StarCraft is the most studied RTS platform, with BWAPI enabling state-feature extraction and action execution while the game runs.Only a few examples had applied deep learning to StarCraft in the reviewed context.
  • Sports games: Sports-game research includes realistic soccer-like environments and multi-agent settings such as RoboCup 2D Half-Field-Offense.HFO uses teams of 2–3 players assigned offensive or defensive roles on half a soccer field.
  • Text adventure games: Text adventures use descriptions and commands for querying state, interacting with characters, managing items, and navigating fictional worlds.Interfaces may be parser-based, choice-based, or hyperlink-based, with parser systems allowing open-ended inputs.
  • Text adventure games: Text adventure results are difficult to compare because the genre lacks a standard benchmark, although TextWorld was introduced as a standardized environment.Much earlier work focused on games running on Infocom’s Z-Machine engine.

A. Arcade Games

Arcade-game research established ALE as a central visual reinforcement-learning benchmark and developed value-based, actor-critic, distributed, recurrent, evolutionary, and hybrid methods around its challenges.

  • Benchmark: ALE contains more than 50 Atari games and is the main testbed for policies learned directly from raw pixels.Its benchmarked advances include many deep reinforcement-learning methods summarized across the Atari suite.
  • Value-based methods: DQN achieved human expert-level control in ALE, outperforming earlier methods and human experts on three of seven tested Atari games.The algorithm approximates action values with a neural network trained through Q-learning.
  • Value-based methods: Experience replay reuses randomly sampled past transitions, reducing update variance; a separate target network later enabled above-human scores in 29 of 49 games.Replay stores state, action, reward, and successor-state experiences for batch updates.
  • Architectures and scaling: Recurrent Q-learning supports games with partially observable states, while distributed Gorila DQN outperformed non-distributed DQN in 41 of 49 games.Recurrent layers preserve information across time, and Gorila distributes actors, replay memory, and learners.
  • Value-based improvements: Double DQN reduces action-value overestimation, prioritized replay samples high-TD-error experiences, and Dueling DQN separates state value from action advantage.These are complementary improvements to the DQN family.
  • Actor-critic methods: IMPALA scaled actor-critic learning across machines and achieved 176.9% mean human-normalized score across all 57 Atari games with one parameter set.Its median human-normalized score was 59.7%.
  • Evolutionary methods: Evolution Strategies outperformed A3C in 23 of 51 games after one hour on 720 CPUs, but used 3 to 10 times as much data.Novelty-based variants improved performance on several challenging Atari games, while doing worse on some less sparse or deceptive tasks.
  • Evolutionary methods: Deep GA achieved results comparable to DQN, A3C, and ES on 13 Atari games using up to thousands of CPUs in parallel.With similar computation, random search outperformed DQN on four of 13 games and A3C on five.

B. Montezuma’s Revenge

Montezuma’s Revenge exemplifies sparse-reward environments that remain difficult for deep reinforcement learning. The reviewed approaches improve exploration or learning through hierarchical goals, intrinsic motivation, demonstrations, language instructions, and teacher interaction.

  • Sparse rewards: DQN scores 0 and Gorila averages 4.2 points in Montezuma’s Revenge, versus 4,367 for a human expert.These results illustrate the difficulty of learning when rewards are sparse.
  • Hierarchical learning: Hierarchical-DQN reaches an average score of around 400 by separating intrinsic-goal selection from goal-directed action control.Its meta-controller selects goals, while the controller learns actions that satisfy them.
  • Intrinsic motivation: Pseudo-count methods provide intrinsic exploration bonuses when unexpected pixel configurations are observed.The bonuses can be derived from CTS or neural density models.
  • Demonstrations: Ape-X DQfD combines distributed actors with human demonstrations and outperforms previous methods in ALE, including beating level 1 in Montezuma’s Revenge.DQfD initializes replay with expert demonstration data, while Ape-X supplies the distributed architecture.
  • Language guidance: Text instructions raise performance to 3,500 points and support generalization to unseen instructions similar to those previously learned.The approach combines CNN visual input with an RNN for text-based instructions.
  • Language guidance: XWORLD agents learn to execute commands such as walking to and picking up objects through a teacher-language system linking RNN, CNN, action-selection, and recognition modules.The recognition module learns the teacher’s language through question answering.

C. Racing Games

Racing games require methods suited to continuous, high-dimensional action spaces and visual control. Reviewed systems use direct perception, policy gradients, evolution, auxiliary tasks, transfer, and intrinsic rewards, with several achieving strong human-relative performance or generalization.

  • Visual control: Direct perception trained on 12 hours of human driving in TORCS drives diverse environments and generalizes to real images.The CNN predicts meaningful affordance indicators such as car angle and distance to lane markings.
  • Action spaces: DQN cannot directly handle racing environments because its action space must be discrete and relatively low-dimensional.Policy-gradient methods such as actor-critic and DPG are presented as alternatives for continuous action spaces.
  • Racing performance: A3C reaches roughly 75% to 90% of a human tester’s score in TORCS after 12 hours of pixel-only training.Rewards were shaped as the agent’s velocity on the track, with the range covering tracks with and without opponent bots, respectively.
  • Evolutionary methods: Evolution finds a high-performing TORCS controller encoded as a recurrent network with over 1 million weights from Fourier-type coefficients.The controller relies only on high-dimensional visual input.
  • Goal-conditioned control: DFP predicts future measurements from the current state, goal, and action, then selects actions with the best predicted outcome.Training across various goals allows generalization to unseen goals at test time.
  • Auxiliary tasks: UNREAL achieves an 87% mean human-normalized score on OpenArena tasks, compared with 53% for A3C.Its auxiliary task predicts the immediate subsequent future reward from consecutive observations.
  • Exploration: Intrinsic Curiosity Module training enables navigation in complex Doom and Super Mario levels using only intrinsic rewards.The module computes an intrinsic reward from the agent’s inability to predict action outcomes.

E. Open-World Games

Open-world and strategy-like environments demand memory, curriculum selection, multi-agent coordination, partial observability, and long-horizon planning. The reviewed systems demonstrate transfer, improved evaluation, and competitive but bounded StarCraft performance.

  • Open-world transfer: H-DRLN transfers knowledge between Minecraft tasks including navigation, item collection, and placement.It uses policy distillation to retain learned knowledge in a single network.
  • Memory: Memory-based RMQN and FRMQN solve complex navigation tasks requiring memory and active perception.These architectures are inspired by Neural Turing Machines and use recurrent external-memory mechanisms.
  • Curriculum learning: TSCL enables policy-gradient learning to solve mazes that uniform subtask sampling cannot solve.A teacher prioritizes tasks where student performance is increasing or decreasing.
  • RTS challenges: RTS games require simultaneous multi-agent control on partially observable maps, while rewards are determined by winning rather than in-game scores.These properties make end-to-end learning especially complex.
  • Search and evaluation: A supervised CNN state evaluator combined with Monte Carlo Tree Search performs significantly better than previous evaluation methods in µRTS.The evaluator is trained on a generated dataset.
  • StarCraft scope: StarCraft deep-learning research focuses mainly on micromanagement and build-order planning, leaving other game aspects unaddressed.Combat rewards can be shaped from damage inflicted and damage incurred, but multi-agent credit assignment remains difficult.
  • StarCraft results: PPO-based macro-action planning outperforms the built-in StarCraft II level-10 bot but loses all games against Diamond-level human players.The authors report limited strategy diversity, and evaluation covers one matchup on one map.

G. Team Sports Games

The supplied passages describe deep-learning approaches for games involving language, mixed action spaces, prediction, and multi-agent communication. These methods combine neural representations with reinforcement learning, supervised prediction, or language models to select actions and coordinate agents.

  • Mixed action spaces: DDPG in RoboCup HFO uses separate output streams for discrete action types and continuous parameters such as power and direction.Inverting Gradients constrains gradients near action boundaries.
  • Predictive models: Object-centered predictive modeling learns billiards simulations from random interactions and uses the model for action planning.The approach learns physical intuition through an object-centered representation.
  • Predictive models: An extended ResNet-34 predicts visual outcomes in a 3D block-stacking environment where blocks usually fall.Residual shortcut connections are used in the network architecture.
  • Text-based games: LSTM-DQN completes 96% to 100% of quests on average in two text adventure games.LSTM representations convert textual world states into vectors used to estimate Q-values for possible actions.
  • Language representations: DRRN embeds state and action descriptions separately and uses their relevance as the Q-value in end-to-end deep Q-learning.Relevance can be computed with an inner product or bilinear operation.
  • Language models: Golovin uses fantasy-book-pretrained language models and five command generators without reinforcement learning.Its generators cover general, movement, battle, gather, and inventory commands.
  • Action selection: Action Elimination Networks reduce parser-based games’ large action spaces by predicting ineffective actions before Q-network evaluation.The elimination model learns while playing.
  • Multi-agent communication: DDRQN trains multiple agents to learn a communication protocol for Hats and Switch riddles using shared weights conditioned on each agent’s identity.This preserves diversity while enabling faster learning.

V. HISTORICAL OVERVIEW OF DEEP LEARNING IN GAMES

The reviewed history traces deep-learning game-playing methods through influential algorithms, extensions, genre-specific adaptations, and parallel or evolutionary alternatives.

  • DQN became a highly influential gradient-based method for pixel-based video-game playing after its original application to Atari.
  • Double DQN and Dueling DQN were early extensions, while Gorila and A3C introduced asynchronous DQN-based and actor-critic approaches.
  • UNREAL extended A3C with auxiliary learning to address sparse feedback environments, followed by further A3C extensions.
  • Methods developed for Atari were extended to Doom, Minecraft, text adventures, and other genres to handle richer data, large state spaces, language, or mapping.
  • Combining algorithmic extensions and parallelization became prominent, while evolution strategies and derivative-free methods emerged as competitive alternatives.
  • The influence diagram links algorithms to methods they used or modified, with node colors encoding benchmarks and distance from the center encoding arXiv publication date.

VI. OPEN CHALLENGES

The paper identifies open challenges spanning general game playing, sparse rewards, and scaling learning across diverse tasks and interacting agents.

  • Developing algorithms that play fundamentally different games remains difficult because current approaches require game-specific network architectures and reward functions.
  • General game playing requires learning across multiple, potentially very different games rather than only known games in the ALE framework.
  • Model-free deep reinforcement learning has overfitted to individual games and even levels, motivating continuously generated training levels in GVGAI.
  • Sparse-reward games such as Montezuma’s Revenge remain challenging despite intrinsic motivation and expert-demonstration methods.
  • Scaling multi-agent learning to the number of interacting agents found in games such as StarCraft or GTA V remains an open challenge.

4) Lifetime adaptation

The paper highlights a gap between agents trained for strong performance and agents that can adapt during play or reproduce controllable human-like behavior.

  • 4) Lifetime adaptation: Current deep-learning techniques generally require expensive retraining to adapt to unforeseen situations encountered during gameplay.
  • 4) Lifetime adaptation: Real-time behavior from a single human supplies far less data than common deep-learning methods require, linking lifetime adaptation to few-shot and transfer learning.
  • 4) Lifetime adaptation: Most approaches optimize game performance or score, whereas human-facing bots may instead need believable behavior for play, cooperation, or competition.
  • 6) Adjustable performance levels: Game testing, tutorials, and demonstrations can require agents with a particular skill level rather than agents that outperform all human players.
  • 6) Adjustable performance levels: Procedural personas encode agent preferences as utility weights, but deep-learning implementation and planning-depth control remain unresolved.
  • StarCraft’s branching factor is orders of magnitude larger than Chess’s or Go’s, making comparable deep-RL scaling an open challenge.

2) Interactive tools for game development

The paper connects game-industry adoption to controllable training tools, broader production goals, and models that can learn game rules for testing or design.

  • 2) Interactive tools for game development: Designers lack easy tools for training NPC behaviors while controlling desired and undesired outcomes.
  • 2) Interactive tools for game development: Preference learning, imitation learning, and human demonstrations provide promising routes for non-experts to shape or train game-playing behaviors.
  • 3) Creating new types of video games: Deep learning could help create new game types, but gradient-based optimization is constrained by the need for mathematical smoothness.
  • Learning a forward model could enable planning methods based on tree search or evolution and may improve generalization across game variations.
  • Real-time adaptation remains computationally infeasible, motivating network compression, pruning, and increased processing power.
Loading 1708.07902v3…