Source-linked AI summary
ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, C. Lawrence Zitnick
TL;DR
Existing game platforms trade off realism, efficiency, customizability, and concurrent training support. ELF combines a flexible RL platform with an RTS engine, and its trained Mini-RTS agent beats the built-in rule-based AI more than 70% of the time in full games.
Problem
Existing game environments do not jointly provide diverse capabilities, efficient simulation, customization, and flexible support for concurrent training architectures.
Source: §1 · p. 2Method
ELF integrates concurrent game simulation with a flexible RL backend and a customizable RTS engine containing Mini-RTS, Capture the Flag, and Tower Defense.
Source: §1 · p. 2Results
More than 70% of the time, ELF's Leaky ReLU and Batch Normalization network with long-horizon training and progressive curriculum beats the rule-based built-in AI in full-game Mini-RTS.
Source: §1 · p. 2Takeaways & Limitations
ELF supports full-game bot training and flexible environment-agent mappings, while its engine also supports training at different levels of a command hierarchy.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In this paper, we propose ELF, an Extensive, Lightweight and Flexible platform for fundamental reinforcement learning research. Using ELF, we implement a highly customizable real-time strategy (RTS) engine with three game environments (Mini-RTS, Capture the Flag and Tower Defense). Mini-RTS, as a miniature version of StarCraft, captures key game dynamics and runs at 40K frame-per-second (FPS) per core on a Macbook Pro notebook. When coupled with modern reinforcement learning methods, the system can train a full-game bot against built-in AIs end-to-end in one day with 6 CPUs and 1 GPU. In addition, our platform is flexible in terms of environment-agent communication topologies, choices of RL methods, changes in game parameters, and can host existing C/C++-based game environments like Arcade Learning Environment. Using ELF, we thoroughly explore training parameters and show that a network with Leaky ReLU and Batch Normalization coupled with long-horizon training and progressive curriculum beats the rule-based built-in AI more than $70\%$ of the time in the full game of Mini-RTS. Strong performance is also achieved on the other two games. In game replays, we show our agents learn interesting strategies. ELF, along with its RL platform, is open-sourced at https://github.com/facebookresearch/ELF.
1 Introduction
ELF targets gaps between realistic but costly, inflexible games and efficient but limited research environments. It combines customizable, concurrent game simulation with an RTS engine and reports full-game Mini-RTS performance above the built-in AI.
1 Introduction
- Existing platforms trade off realism, speed, customizability, or simulation concurrency, while Python interfaces can constrain training architectures.The paper frames desirable research environments as extensive, lightweight, and flexible.Source: §1 · pp. 1–2
- ELF's RTS engine provides Mini-RTS, Capture the Flag, and Tower Defense; Mini-RTS captures StarCraft's basic dynamics and runs at 165K FPS on a 4-core laptop.The authors report full-game bot training in one day with 6 CPUs and 1 GPU.Source: §1 · p. 2
- Full access to game data and hierarchical commands supports top-level strategy training with built-in lower-level tactics, as well as supervised training on internal data.The paper contrasts this full-game scope with prior RTS research focused on tactical battles.Source: §1 · p. 2
- ELF supports one-to-one, many-to-one, and one-to-many environment-agent mappings, and can host existing C/C++ games through a simple adaptor.The paper attributes topology flexibility to its hybrid C++/Python framework and parallelizes in C++.Source: §1 · p. 2
- More than 70% of the time, the Leaky ReLU and Batch Normalization network with long-horizon training and progressive curriculum beats the rule-based built-in AI in full-game Mini-RTS.The paper also reports stronger performance on the other games.Source: §1 · p. 2
2 Architecture
ELF connects concurrent C++ game simulation to flexible Python consumers through batched state and reply exchange. Its architecture supports multiple environment-model topologies, RL methods, and game interfaces.
2 Architecture
- The producer-consumer loop runs games in C++ threads, batches ready states for Python consumers, and returns replies to resume blocked games.Consumers can run in the same process, separate processes, or on different machines.Source: §2 · pp. 2–3
- C++ threads provide environment-level parallelism, avoiding Python's Global Interpreter Lock limitation on thread-level parallelism.The paper contrasts this with Python-threaded environments and the overhead and complexity of process-level parallelism.Source: §2 · p. 3
- ELF supports one-to-one, many-to-one, and one-to-many environment-model configurations, including A3C, batched prediction, MCTS, and self-play.The configurations can be tested with minimal changes.Source: §2 · p. 3
- A unified interface hosts C/C++ games, including Atari, board games, and ELF's customized RTS engine, through a simple adaptor.RTS games can use raw pixels or lower-dimensional internal game data; web-based rendering supports debugging.Source: §2 · pp. 3–4
- The Python RL backend decouples methods from models and implements baselines including A3C, Policy Gradient, Q-learning, and Trust Region Policy Optimization.The paper says most implementations require very few lines of Python code.Source: §2 · p. 4
3 Real-time strategy Games
RTS research platforms trade realism and complex dynamics against speed, customizability, and research access. ELF provides customizable RTS environments with hierarchical commands and reports both fast simulation and full-game bots that outperform built-in rule-based AIs.
3 Real-time strategy Games
- Commercial RTS games have sophisticated dynamics but are closed-source and difficult to use for research, while research-oriented games are faster but often simpler.Some open-source RTS engines prioritize graphics and network play rather than faster-than-real-time execution or straightforward machine-learning interfaces.Source: §3 · pp. 4–5
3.1 Our approach
- The engine supports derived games through configurable commands, units, and command-triggering rules, and provides Mini-RTS, Capture the Flag, and Tower Defense.Researchers can modify these elements in C++ or through game scripts such as Lua.Source: §3.1 · p. 5
- 40K frames-per-second per core lets Mini-RTS bots be trained in a day on a single machine.The engine uses real coordinates, collisions, durative actions, and tick-driven decisions.Source: §3.1 · p. 5
- ELF's command hierarchy supports strategic, tactical, and unit-level control, with high-level commands translated into lower-level actions.Built-in rule-based AIs issue high-level commands and have access to all map information.Source: §3.1 · p. 5
- ELF enables full-game RTS bots that outperform built-in rule-based AIs, addressing work previously focused on rule-based play or tactical scenarios.The paper presents this as the first such demonstration for RTS games.Source: §3.1 · pp. 5–6
- At 1024 threads, ELF runs 3x faster than OpenAI Gym in frame-per-second-per-core tests.The comparison uses an Intel E5-2680@2.50GHz CPU without hyper-threading.Source: Figure 4 · p. 6
4 Experiments
ELF scales parallel simulation, while full-game A3C results vary with frame skip, network design, and curriculum. The experiments also compare trained agents with MCTS, whose win rates are not directly comparable because it uses full game dynamics and information.
4.1 Benchmarking ELF
- ELF's parallel ALE ran at 5.1K FPS per core on up to 16 cores, versus 1.7K FPS per core for OpenAI Gym with 16 cores and 1024 threads.The single-threaded ALE rate was around 5.8K FPS on Pong; the experiments used no hyperthreading.Source: §4.1 · p. 6
4.2 Baselines on Real-time Strategy Games
- The experiments train A3C agents on randomized initial states and evaluate 1-vs-1 full games, with trained agents observing only partial information in most settings.They report means and standard deviations across five runs; Capture the Flag used frame skip 10 for the trained agent and 50 for its opponent.Source: §4.2 · p. 6, §4.2.1 · p. 7
4.2.1 A3C baseline
- 60%: the trained Capture the Flag bot beat the built-in AI when it had a frame-skip advantage; performance was low with even frame skips.The trained bot used frame skip 10 while its opponent used 50 in the reported comparison.Source: §4.2.1 · p. 7
- Batch Normalization and Leaky ReLU each improve and stabilize A3C performance, and the two network changes are complementary.Batch Normalization stabilizes gradient flow, while Leaky ReLU preserves negative responses in sparse-input scenarios.Source: §4.2.1 · p. 7
4.2.2 Curriculum Training
- Curriculum training improves performance, and training against diverse agents is important for producing agents with low exploitability.The curriculum gradually reduces built-in-AI assistance; curriculum is also critical when fine-tuning toward a mixture of opponents.Source: §4.2.2 · p. 8
4.2.3 Monte-Carlo Tree Search
- MCTS achieves a win rate comparable to RL models, but the rates are not directly comparable because MCTS has complete game information and runs much slower.MCTS uses eight threads with 100 rollouts each and takes 2–3 seconds per move, compared with at most 1 millisecond for trained RL AI.Source: §4.2.3 · p. 9
5 Conclusion and Future Work
ELF combines concurrent game simulation with configurable environments, and its RTS engine provides three environments that run at 40K FPS per core on a laptop. The paper reports that full-game bots can be trained end-to-end in one day on a single machine and identifies further RTS research directions.
5 Conclusion and Future Work
- ELF's RTS engine provides Mini-RTS, Capture the Flag, and Tower Defense, running at 40K FPS per core on a laptop.The paper reports that a full-game bot can be trained end-to-end in one day on a single machine.Source: §5 · p. 9
- The platform supports efficient exploration of RTS reinforcement learning, including forward modeling, hierarchical RL, planning under uncertainty, and complicated action spaces.The authors describe these as future research possibilities and plan to improve the platform and build a library of maps and bots.Source: §5 · p. 9
6 Appendix: Detailed descriptions of RTS engine and games
The appendix describes ELF's three RTS environments and the engine features behind them, including tick-driven gameplay and hierarchical commands. It also reports balance checks and replay support.
6.1 Overview
- The RTS engine is tick-driven, and its overview covers gameplay visualization and the command system; the appendix lists three environments built from the engine.The environments are Mini-RTS, Capture the Flag, and Tower Defense.Source: Figure 7 · p. 12, §6.1 · p. 12, Table 8 · p. 12
6.2 Hierarchical Commands
- Commands form a hierarchy: strategic commands issue durative or immediate unit commands, and durative commands can issue further commands across ticks.Durative commands continue until a goal is achieved, while immediate commands change the situation at the current tick.Source: Figure 8 · p. 12, §6.2 · p. 12
6.3 Units and Game Dynamics
- Mini-RTS includes resource gathering, facility and troop building, defense, and invasion; its framework allows additional units to be added easily.The appendix also describes Capture the Flag's flag-carrying and respawn rules and Tower Defense's waves, tower building, and win/loss conditions.Source: §6.3 · p. 13, Table 9 · p. 13, Table 10 · p. 13
6.4 Others
- Self-play produced balanced outcomes in Mini-RTS and Capture the Flag, with player 0 win rates of 50.0(±3.0) and 49.9(±1.1), respectively.The engine also supports replay serialization and state snapshots at arbitrary ticks.Source: §6.4 · p. 13
7 Detailed explanation of the experiments
The experiment appendix specifies randomized starting conditions and discrete actions, then describes rule-based opponents for Mini-RTS and Capture the Flag. Mini-RTS uses nine strategic global commands with hard-coded execution details.
7 Detailed explanation of the experiments
- The experiments randomize starting states: Mini-RTS varies resources, bases, units, and buildings, while Capture the Flag randomizes athlete and flag positions.The flag starts at an equal distance from both bases.Source: §7 · p. 14
7.1 Rule based AIs for Mini-RTS
- Mini-RTS rule-based opponents gather resources and build attackers; SIMPLE sends five melee units to attack, while HIT N RUN uses two ranged attackers that retreat under counterattack.The trained Mini-RTS action space uses nine strategic global commands with hard-coded execution details, and building commands cancel when resources are insufficient.Source: §7.1 · p. 14, Table 11 · p. 14
7.2 Rule based AIs for Capture the Flag
- The Capture the Flag rule-based AI seeks an unoccupied flag, escorts its carrier, defends against attacks, and targets an opponent carrying the flag.The appendix identifies a separate action-space table for the trained Capture the Flag agent.Source: §7.2 · p. 14, Table 12 · p. 14