PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for General Video Game AI

Ruben Rodriguez Torrado, Philip Bontrager, Julian Togelius, Jialin Liu, Diego Perez-Liebana

arXiv:1806.02448v1cs.LGcs.AIcs.NEstat.ML

TL;DR

GVGAI has supported popular planning competitions, but its learning track had not produced many strong agents, while general game-playing evaluation calls for testing games agents were not optimized for. This paper connects GVGAI to OpenAI Gym and benchmarks deep reinforcement-learning algorithms across eight games. Results vary substantially by game: A2C scores higher than DQN and PDDQN in six of eight environments, while planning agents have a slight overall advantage with large game-to-game variation.

  • Problem

    ALE offers only a finite set of games, while assessing general game-playing requires testing on games agents were not optimized for and being able to add games.

    Source: §I · p. 1

  • Method

    The paper connects GVGAI to OpenAI Gym and tests DQN, Prioritized Dueling DQN, and A2C on eight representative games.

    Source: §I · p. 1, §V · p. 7

  • Results

    Performance varies drastically by game; A2C scores higher than DQN and PDDQN in 6 of 8 environments, while planning agents have a slight overall advantage with large variation between games.

    Source: §V · p. 7

  • Takeaways & Limitations

    The platform offers an opportunity to evaluate how algorithms learn and evolve across changing environments.

    Source: §V · p. 8

    Some or all tested learning algorithms fail on some games, including poor DQN and A2C performance on binary-score games such as Frogs.

    Source: §V · p. 7

Abstract

from arXiv · show

The General Video Game AI (GVGAI) competition and its associated software framework provides a way of benchmarking AI algorithms on a large number of games written in a domain-specific description language. While the competition has seen plenty of interest, it has so far focused on online planning, providing a forward model that allows the use of algorithms such as Monte Carlo Tree Search. In this paper, we describe how we interface GVGAI to the OpenAI Gym environment, a widely used way of connecting agents to reinforcement learning problems. Using this interface, we characterize how widely used implementations of several deep reinforcement learning algorithms fare on a number of GVGAI games. We further analyze the results to provide a first indication of the relative difficulty of these games relative to each other, and relative to those in the Arcade Learning Environment under similar conditions.

I. INTRODUCTION

ALE offers a finite set of existing games, while evaluating general game-playing requires games agents were not optimized for and the ability to add new ones. GVGAI provides a large, extensible game framework, but its learning track had not yet supported strong machine-learning agents; this paper connects it to reinforcement learning and reports benchmarks.

I. INTRODUCTION

  • ALE contains a finite set of games, whereas assessing general game-playing requires testing on games agents were not optimized for and adding new games.Creating games also allows tests targeted at particular AI capacities.Source: §I · p. 1
  • GVGAI was created as a versatile general-AI benchmark, but its learning track had few strong agents and was considered inferior to ALE for machine-learning agents.Its planning tracks were popular and drew strong tree-search and evolutionary-planning agents.Source: §I · p. 1
  • The paper connects GVGAI to OpenAI Gym and benchmarks three deep reinforcement-learning algorithms on eight dissimilar games.The interface is intended to support existing reinforcement-learning implementations and future learning-track development.Source: §I · p. 1

A. General Video Game AI

  • GVGAI is a Java benchmark for more than 160 single- and two-player 2D arcade-like games specified in VGDL.VGDL describes games and levels in separate files; game definitions cover sprites, interactions, termination conditions, and sprite-to-character mappings.Source: A. General Video Game AI · pp. 1–2
  • Earlier GVGAI learning work included evolutionary value-function learning, learned forward models, and a DQN agent trained from screen captures.The cited DQN study used seven games and reported increased winning rate and score over consecutive episodes.Source: A. General Video Game AI · p. 2
  • The first single-player learning competition received few, simple agents, mostly greedy or feature-based Q-learning and SARSA methods.The competition was held at CIG2017.Source: A. General Video Game AI · p. 2

B. Deep Reinforcement Learning

Deep reinforcement learning combines reinforcement learning with deep learning, building on successes such as DQN on Atari games. The section describes Dueling networks and actor-critic methods as developments aimed at improving learning, stability, or computation.

B. Deep Reinforcement Learning

  • Deep Q-Networks combined with reinforcement learning have been used to play Atari 2600 games from video input.The section situates deep reinforcement learning among applications in games and other fields.Source: B. Deep Reinforcement Learning · p. 2
  • Dueling networks use separate estimators for state value and state-dependent action advantage, enabling learning to generalize across actions without changing the underlying algorithm.The cited work presents this architecture as an improvement to DQN results and stability.Source: B. Deep Reinforcement Learning · p. 2
  • Asynchronous Advantage Actor-Critic (A3C) predicts policy and value functions and is inherently parallelizable.The passage reports that this approach improved the original DQN score and halved computation time even without CPU use.Source: B. Deep Reinforcement Learning · pp. 2–3

C. OpenAI Gym

OpenAI Gym provides benchmark environments for testing reinforcement-learning methods. Its examples include ALE for Atari games and MuJoCo for humanoid-like movement.

C. OpenAI Gym

  • OpenAI Gym aims to provide benchmark platforms for comparing results as researchers test reinforcement-learning ideas.The passage identifies this as the platform’s main goal.Source: C. OpenAI Gym · p. 3
  • The platform includes ALE, a collection of Atari 2600 games, among its varied benchmarks.Source: C. OpenAI Gym · p. 3
  • MuJoCo is an example of a Gym environment used to test humanoid-like movement in 2D and 3D.Source: C. OpenAI Gym · p. 3

III. METHODS

GVGAI games can be added or designed for specific problems, and the authors place them alongside existing environments to contextualize algorithm performance. They embed GVGAI as an OpenAI Gym environment, where agents receive observations and return actions under a per-tick time limit.

III. METHODS

  • The authors use GVGAI’s ability to add games and design task-specific games while comparing its current games with existing environments.This comparison is intended to show strengths and weaknesses of current reinforcement-learning algorithms and relate GVGAI results to other environments.Source: §III · p. 3

A. GVGAI-OpenAI embedding

  • The Gym embedding sends agents each game-tick observation and accepts an action or an abort request, without giving agents the forward model or access to game simulations.Observations may be JSON state objects or screenshots; games and levels can be added through VGDL.Source: A. GVGAI-OpenAI embedding · p. 3
  • Agents have a 40 ms per-tick decision limit, but the reported five-minute learning allowance excludes game advancement, serialization, and communication time.Actual learning runs can last several hours.Source: A. GVGAI-OpenAI embedding · p. 3

B. GVGAI Games

The paper samples eight GVGAI games across a range of planning difficulty, including Atari-like games and cases chosen for their reinforcement-learning challenges. The set also includes games with stochastic elements, and all experiments use each game's first level.

B. GVGAI Games

  • In Superman, villains attack the clouds beneath civilians; Superman can rescue falling civilians or jail villains for one point each, with a 1000-point bonus for catching all villains.The figure illustrates a game where the rescue and jail actions have distinct rewards from the win bonus.Source: Figure 1 · p. 3, B. GVGAI Games · p. 4
  • Sampling aimed to span games from planning-easy Aliens to planning-difficult Superman, include Atari counterparts, and cover interesting reinforcement-learning challenges.Game difficulty draws on Bontrager et al.; stochastic components occur in some games, mostly through NPC movement, and experiments used the first of five levels.Source: B. GVGAI Games · p. 3
  • The eight selected games are Aliens, Seaquest, Missile Command, Boulder Dash, Frogs, Zelda, Wait For Breakfast, and Superman.The first five were modeled after similarly named Atari games; the sample also includes games chosen for their reinforcement-learning challenges.Source: B. GVGAI Games · p. 3
  • Wait For Breakfast requires the player to sit at the sole chair after randomly timed service and eat; leaving after service causes a loss.Source: Figure 2 · p. 4

C. Benchmarks

The benchmarks use three OpenAI Baselines algorithms with a shared Atari-style convolutional network and a one-million-call budget per game. The authors chose accessible implementations to support comparable and reproducible results.

C. Benchmarks

  • The authors used open, publicly vetted baseline implementations to support comparability and reproducibility.Source: C. Benchmarks · p. 4
  • DQN, Prioritized Dueling DQN, and A2C were selected from OpenAI Baselines as documented algorithms used in environments such as ALE.The authors chose them as benchmarks because DQN and A3C, on which A2C is based, are baselines against which many reinforcement-learning developments are scored.Source: C. Benchmarks · p. 4
  • Each algorithm used the same three-convolutional-layer, two-fully-connected-layer Atari network and trained on one million frames per game.The network interprets GVGAI screenshots; the authors considered one million calls enough to indicate game difficulty under realistic computational constraints.Source: C. Benchmarks · p. 4, Table I · p. 4
  • Each baseline was tested on every game for one million calls, totaling 24 million calls; A2C was OpenAI's synchronous version of A3C.The authors describe the synchronous version as more efficient and performing as well as A3C on Atari.Source: C. Benchmarks · p. 4

IV. RESULTS AND DISCUSSION

The results section presents baseline training results for sampled GVGAI games, then compares GVGAI with Atari and reinforcement-learning agents with planning agents.

IV. RESULTS AND DISCUSSION

  • The section reports baseline performance on a sample of GVGAI games to inform comparisons with other AI techniques and environments.Source: §IV · p. 4
  • The reported results characterize only a sample of games from the GVGAI framework.Source: §IV · p. 4
  • The results are organized into learning-algorithm training outcomes, a GVGAI–Atari comparison, and a comparison with planning agents.Source: §IV · p. 4

A. Results of learning algorithms

A2C performed best on most of the eight sampled games, though the authors caution that the computational budget and sample size limit the comparison. Results varied with exploration, reward feedback, and game-specific demands.

A. Results of learning algorithms

  • Training curves report cumulative episode rewards, averaged over 20 results to smooth noise from episode restarts; rewards are not comparable across games.A2C ran 12 networks in parallel, and its plotted reward at each step averaged those networks while matching one A2C step to 12 DQN steps.Source: A. Results of learning algorithms · pp. 4–5
  • A2C's parallel Boulder Dash workers were individually chaotic, and an improvement by one worker benefited all 12 workers.The authors connect this pattern to escaping local minima and the importance of exploration.Source: A. Results of learning algorithms · p. 5
  • A2C performed best on most tested games, but the authors caution that other algorithms might catch up with more computation and that eight games are a small comparison sample.A2C also appeared to benefit from sampling more initial conditions and starting with a higher score.Source: A. Results of learning algorithms · p. 5
  • Prioritized Dueling DQN slightly outperformed vanilla DQN, while their overall performance was similar; shared initial seeds produced similar early exploration before divergence.A2C's different parallel explorations prevent a direct comparison of initial exploration and help explain its higher starting scores, especially in Aliens.Source: A. Results of learning algorithms · p. 5
  • The sampled games varied in reward feedback: none of the agents appeared to solve Frogs within the allotted calls, while Wait For Breakfast required stumbling on a fixed winning location.The authors state that the games were left unchanged, although researchers can edit VGDL files to modify reward structures for experiments.Source: A. Results of learning algorithms · p. 5
  • In Superman, agents sometimes found a successful pattern but could not reproduce it in the stochastic environment.The authors attribute difficulty to dynamic NPCs and delayed reward: points come for delivering captured villains to jail, not for capture alone.Source: A. Results of learning algorithms · p. 5
  • In Boulder Dash, A2C learned to collect diamonds and survive, while DQN and Prioritized Dueling DQN struggled.The authors suggest that collecting a diamond without moving creates negative feedback, making survival without collecting an apparent local optimum.Source: A. Results of learning algorithms · p. 5
  • Seaquest's randomness, diver-chasing requirement, and 25-tick surfacing rule challenged learning; Zelda provided feedback at each event and was a comparatively good fit.In Seaquest, the agents did not appear to learn to chase divers, and noisy results were attributed most likely to difficulty learning stochastic rules.Source: A. Results of learning algorithms · p. 5

B. Comparison with ALE

ALE has a static game set, while GVGAI can add games and offers a more fluid benchmark; the paper argues that performance on GVGAI is at least comparable to ALE, with direct score comparisons constrained by differences between similarly named games.

B. Comparison with ALE

  • GVGAI has over twice as many games as ALE, and VGDL allows researchers to design new games, including stochastic games and multiple levels.The paper presents these features as ways to create new challenges and test generalization.Source: B. Comparison with ALE · p. 6
  • GVGAI games can be hard to play and less familiar to readers than Atari games, while training runs a few times slower than Atari.The engine communicates from Java to Python through a local port; communication optimization was ongoing.Source: B. Comparison with ALE · p. 6
  • Scores on similarly named games cannot be compared directly because GVGAI versions may differ from Atari originals in their rules and scoring.The VGDL author decides how closely a game matches its inspiration and how scoring works.Source: B. Comparison with ALE · p. 6
  • The authors propose that doing well on GVGAI is at least comparable to doing well on ALE, and report that some GVGAI games remain unbeaten.They describe GVGAI as a more fluid benchmark that can continually challenge perceived RL-agent success.Source: B. Comparison with ALE · p. 6

C. Comparison with planning algorithms

The learning and planning agents' relative performance varies by game: learning agents lead on Seaquest, planning agents alone win Frogs, and results are mixed on other games. The comparison is asymmetric because learning agents train for hours while planning agents have a forward model but no training time.

C. Comparison with planning algorithms

  • The comparison is asymmetric: learning agents train for hours on each game, while planning agents have no training time but do have a forward model.The authors frame the comparison as difficult to judge a priori because each approach has a major advantage.Source: C. Comparison with planning algorithms · p. 7
  • On Aliens, all agents learn to play well; its largely deterministic enemies and projectile timing make reactive play or minimal planning sufficient.Enemy projectiles are stochastic, but take time to reach the player.Source: C. Comparison with planning algorithms · p. 7
  • Wait for Breakfast is solved perfectly by every agent except standard MCTS, which succeeds occasionally; the game rewards a fixed strategy and punishes jittery exploration.The passage also notes that planning far enough ahead can solve it.Source: C. Comparison with planning algorithms · p. 7
  • Frogs is won only by planning agents; the authors attribute learning agents' failure to the absence of intermediate rewards.Planning agents can succeed by planning far enough ahead, while Frogs offers reward only for reaching the goal.Source: C. Comparison with planning algorithms · p. 7
  • On Missile Command, two planning agents and all three learning agents perform well, with no meaningful difference between the best planning algorithm, IW, and the learning agents.The authors say it may be playable by moving near approaching missiles and attacking, but do not explain MCTS's poor performance.Source: C. Comparison with planning algorithms · p. 7
  • On Seaquest, the learning agents perform best overall, with A2C the clear winner among agents handling shooting, diver rescue, and oxygen management.The authors suggest intermediate rewards may favor learning agents, which apparently learn the game's non-trivial task sequence.Source: C. Comparison with planning algorithms · p. 7
  • On Boulder Dash, MCTS and A2C play reasonably well while the other planning and learning agents do much worse; the game combines quick reactions with long-term planning.The authors suggest GA's planning horizon is too short and IW does not handle stochastic enemies.Source: C. Comparison with planning algorithms · p. 7
  • On Zelda, all agents perform comparably; tree-search methods seem to outperform GA and learning agents, but not by a great margin.Source: C. Comparison with planning algorithms · p. 7

V. CONCLUSION

The authors connect GVGAI to OpenAI Gym and report deep RL results across eight games. Performance varies substantially by game, and learning and planning agents show different strengths.

V. CONCLUSION

  • The study creates a reinforcement-learning challenge by connecting GVGAI to OpenAI Gym and testing DQN, Prioritized Dueling DQN, and A2C on eight representative games.Source: §V · p. 7
  • A2C reaches a higher score than DQN and PDDQN for 6 of the 8 environments tested without memory replay.On GVGAI using 12 CPU cores, A2C trains five times faster than DQN trained on a Tesla Nvidia GPU.Source: §V · p. 7
  • DQN and A2C perform badly on binary-score games such as Frogs, while initial-condition dependence makes multiple runs necessary for accurate benchmarking.Default OpenAI Baselines parameters also show stabilization problems in some complex games; replay memory or learning-rate schedules may need modification to improve convergence.Source: §V · p. 7
  • Planning agents have a slight overall advantage, but results vary substantially by game; learning agents perform better on Seaquest and Missile Command.Source: §V · pp. 7–8
  • The authors expect experimentation with existing and purpose-designed games to yield insights into learning algorithms and support scientific evaluation across changing environments.Source: §V · p. 8
Loading 1806.02448v1…