Source-linked AI summary
StarCraft II: A New Challenge for Reinforcement Learning
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, Rodney Tsing
TL;DR
StarCraft II presents reinforcement learning with multi-agent interaction, partial observability, large action spaces, and long-term decision-making demands. The paper introduces SC2LE with feature-layer observations, game interfaces, rewards, mini-games, and human replay data, then evaluates supervised and baseline reinforcement learning agents. Baseline agents approach expert-level performance on some mini-games but cannot learn to win against the easiest built-in AI on the full game.
Problem
StarCraft II offers a reinforcement learning challenge involving multi-agent interaction, partial observability, large action spaces, and long-horizon play.
Method
The paper releases SC2LE, specifying feature-layer observations, actions, rewards, mini-games, full maps, and human replay data for supervised and reinforcement learning evaluation.
Results
Baseline agents perform close to expert human play on some mini-games but cannot learn to win against the easiest built-in AI on the full game, under either terminal or shaping rewards.
Takeaways & Limitations
SC2LE provides a canonical, externally defined challenge that remains intractable for off-the-shelf baseline algorithms on the full game.
Takeaways & Limitations
The release simplifies human gameplay through preprocessed observations, a reduced action space, lock-step execution, and play only against the built-in AI.
Abstract
from arXiv · showhide
This paper introduces SC2LE (StarCraft II Learning Environment), a reinforcement learning environment based on the StarCraft II game. This domain poses a new grand challenge for reinforcement learning, representing a more difficult class of problems than considered in most prior work. It is a multi-agent problem with multiple players interacting; there is imperfect information due to a partially observed map; it has a large action space involving the selection and control of hundreds of units; it has a large state space that must be observed solely from raw input feature planes; and it has delayed credit assignment requiring long-term strategies over thousands of steps. We describe the observation, action, and reward specification for the StarCraft II domain and provide an open source Python-based interface for communicating with the game engine. In addition to the main game maps, we provide a suite of mini-games focusing on different elements of StarCraft II gameplay. For the main game maps, we also provide an accompanying dataset of game replay data from human expert players. We give initial baseline results for neural networks trained from this data to predict game outcomes and player actions. Finally, we present initial baseline results for canonical deep reinforcement learning agents applied to the StarCraft II domain. On the mini-games, these agents learn to achieve a level of play that is comparable to a novice player. However, when trained on the main game, these agents are unable to make significant progress. Thus, SC2LE offers a new and challenging environment for exploring deep reinforcement learning algorithms and architectures.
1 Introduction
SC2LE introduces StarCraft II as a challenging reinforcement learning environment with complex interaction, observation, action, and planning demands. The release provides an interface, simplified mini-games, and human replay data for evaluating learning methods.
- StarCraft II combines fast micro-actions with high-level planning and execution, making victory over top human players a meaningful long-term objective.
- The domain combines multi-agent competition, hundreds of cooperating units, partial observability, fog-of-war, and an approximately 10^8-sized combinatorial action space.
- SC2LE defines low-resolution feature-grid observations and actions, engine-based rewards, and simplified mini-games alongside full game maps.
- The release includes a large and growing dataset of human game replays for evaluating reinforcement learning and related machine learning capabilities.
- Unlike prior StarCraft environments, SC2LE targets StarCraft II, uses the human user interface, and is supported by Blizzard Entertainment across major operating systems.
- Existing artificial StarCraft bots can be defeated by amateur players, motivating StarCraft II as an environment for deep reinforcement learning research.
2 Related Work
Computer games provide standardized, measurable environments for comparing learning and planning approaches. Related work includes Atari benchmarks and StarCraft-based systems, but prior efforts generally address reduced tasks or specific game components.
- Computer games offer clear success measures, rich observational data, fixed dynamics, scalable simulation, and opportunities for comparison with skilled human players.
- The Arcade Learning Environment standardized Atari tasks so scores could be compared directly across publications and algorithms.
- Prior RTS research has focused on specific components such as build orders and combat micromanagement or on techniques such as Monte Carlo Tree Search planning.
- Earlier StarCraft-related environments include BWAPI, simplified RTS systems, reduced-state Wargus, micromanagement mini-games, and replay-based learning tasks.
3 The SC2LE Environment
SC2LE exposes StarCraft II as a reinforcement learning environment through an API, PySC2, feature-layer observations, human-like actions, rewards, and mini-games. It supports full-game and focused experiments while retaining important constraints of the game, including partial observation, long-term consequences, and a large structured action space.
- Environment Components: SC2LE comprises a Linux StarCraft II binary, the StarCraft II API, and PySC2.The API supports programmatic game control, observations, actions, and replay review; PySC2 wraps it for Python reinforcement learning agents.
- Full Game and Rewards: The full game uses either terminal ternary win/tie/loss rewards or a running Blizzard score that is less sparse but not zero-sum.The Blizzard score aggregates current resources, upgrades, units, and buildings, and changes when resources are mined or units and buildings are lost.
- Observations: Feature-layer observations represent game state on configurable N × M grids through scalars and categorical layers for the minimap and screen.Examples include unit type, hit points, owner, visibility, terrain height, and fog-of-war; the minimap summarizes the world while the screen details a subsection.
- Actions: PySC2 represents actions as a function identifier with the arguments it requires, covering approximately 300 action-function identifiers and 13 argument types.For example, selecting units with a rectangle combines a selection mode with two coordinate pairs whose ranges match observation resolution.
- Mini-Games: Mini-games isolate subsets of actions or mechanics with clear reward structures, while shared map files enable identical setups and directly comparable evaluations.They provide focused steps toward the full game and can define restricted action sets, custom rewards, and time limits.
- Performance: At 64 × 64 resolution and 8 frames per action, ladder maps run at 200–700 game steps per wall-clock second, while CollectMineralShards reaches 1600–2000.Speed varies with game stage, unit count, and computer hardware.
4 Reinforcement Learning: Baseline Agents
The paper establishes A3C-based baseline agents for SC2LE, adapting policy representations and network architectures to its structured action space. These agents learn meaningful policies on mini-games, but fail to make substantial progress in the full game.
- Learning Algorithm: A3C learns the agents’ neural-network policy parameters through approximate gradient ascent on expected return.The method uses an n-step return and entropy regularisation to balance value estimation and exploration.
- Policy Representation: The action policy factors the joint action into independent distributions over the function identifier and its arguments.The number of arguments varies by function, and unavailable function identifiers are masked before sampling.
- Agent Architectures: The baseline suite adapts established architectures to SC2LE, including Atari-net, fully convolutional, and recurrent variants.These architectures are designed around the environment’s spatial observations and structured action outputs.
- Full Game: On the full game, no baseline learned to win, and agents trained with Blizzard score converged to trivial mining-preservation strategies.The strongest fully convolutional agent avoided constant losses by lifting buildings out of attack range, making it difficult for the easy AI to win within 30 minutes.
- Mini-Games: Fully convolutional agents performed best among non-human baselines across the mini-games, while Atari-net was competitive on several combat tasks.On BuildMarines, FullyConv LSTM and Random Search achieved the best results, whereas Atari-net failed to learn consistent marine production.
- Overall Findings: The results show that existing reinforcement-learning algorithms face substantially greater difficulty on the full game than on the simplified mini-games.Even the mini-games remain challenging, although agents achieved useful policies there and sometimes performed well against a novice human tester.
5 Supervised Learning from Replays
The paper uses human replays to train supervised value and policy networks for StarCraft II, evaluating outcome prediction and action imitation under partial observability. Value prediction reaches 64% accuracy, while autoregressive policy modelling improves spatial-argument prediction and replay-trained policies produce more units and play better with additional replay data.
- Replay Dataset: 800K games covering all StarCraft II matchups train networks to predict both game outcomes and player actions.The replay dataset is sampled across time, with observations including screen and minimap feature layers plus player statistics.
- Replay Dataset: The replay set averages 153 Actions Per Minute and 3789 mean Match Making Rating, with fewer than one percent of games from Masters-level players.All ranked BattleNet league games are used without filtering, producing a heavy-tailed action distribution.
- Value Predictions: Value prediction reaches 64% accuracy with FullyConv, compared with an approximately 50% random baseline, and becomes more accurate as games progress.The model predicts outcomes from a single partially observed frame without privileged information or previous observations.
- Policy Predictions: Autoregressive FullyConv outperforms FullyConv on policy prediction, especially for spatial arguments, by conditioning arguments on the selected action.Table 2 evaluates base-action, screen-argument, and minimap-argument accuracies; the table caption identifies top-1 and top-5 metrics.
- Policy Predictions: Replay-trained policies produce more units and play better as observed replay data increases, outperforming the reinforcement-learning agents on the BuildMarines mini-game.The supervised policy plays an unrestricted full 1v1 game while being compared on the restricted-action-space mini-game.
6 Conclusions & Future Work
The paper concludes that SC2LE combines an open research environment, replay-based supervised baselines, and reinforcement-learning evaluations. Mini-games can approach expert human performance, but current baseline agents cannot learn to win the full game, while the release still simplifies several human gameplay conditions.
- Conclusions: SC2LE provides a Python interface, ranked-game replay data, supervised policy and value results, and baseline reinforcement-learning results on mini-games and the full game.The release is presented as a new challenge for deep reinforcement learning research.
- Conclusions: Mini-games are treated primarily as unit tests, while the full game evaluated by final outcome is the paper’s central research problem.The authors hope the environment encourages research leading toward solving the full game.
- Results: Some mini-games approach expert human play, but baseline agents cannot learn to win against the easiest built-in AI on the full game.This remains true with either terminal win/tie/loss rewards or Blizzard’s timestep shaping reward.
- Limitations and Future Work: The release simplifies observations, actions, timing, and opponents compared with human play by preprocessing inputs, simplifying controls, using lock-step execution, and restricting full games to built-in AI.Future releases may move toward RGB observations, strict time limits, and self-play.
Contributions
The paper’s contribution includes releasing the StarCraft II API and the associated agents and experiments.
- The release includes the StarCraft II API.
- The release includes all agents and experiments reported in the paper.