Source-linked AI summary
ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning
Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski
TL;DR
Vision-based reinforcement learning needs test beds that extend beyond Atari’s non-realistic 2D, third-person settings. The paper introduces ViZDoom, a customizable first-person 3D Doom environment, and shows that deep Q-learning trains competent bots in move-and-shoot and maze-navigation tasks, supporting the feasibility of visual reinforcement learning in such environments.
Problem
Atari 2600 benchmarks use non-realistic 2D environments and do not resemble real-world tasks, while no FPS-based environment supported agents relying exclusively on raw visual information.
Method
The paper introduces ViZDoom, a customizable Doom-based first-person 3D platform that lets bots act using only the screen buffer and supports custom scenarios.
Results
Deep Q-learning trained competent bots for move-and-shoot and maze-navigation scenarios, with the maze agent navigating surprisingly well and exhibiting human-like behavior.
Takeaways & Limitations
The results support ViZDoom’s utility as an AI research platform and indicate that visual reinforcement learning is feasible in 3D first-person environments.
Abstract
from arXiv · showhide
The recent advances in deep neural networks have led to effective vision-based reinforcement learning methods that have been employed to obtain human-level controllers in Atari 2600 games from pixel data. Atari 2600 games, however, do not resemble real-world tasks since they involve non-realistic 2D environments and the third-person perspective. Here, we propose a novel test-bed platform for reinforcement learning research from raw visual information which employs the first-person perspective in a semi-realistic 3D world. The software, called ViZDoom, is based on the classical first-person shooter video game, Doom. It allows developing bots that play the game using the screen buffer. ViZDoom is lightweight, fast, and highly customizable via a convenient mechanism of user scenarios. In the experimental part, we test the environment by trying to learn bots for two scenarios: a basic move-and-shoot task and a more complex maze-navigation problem. Using convolutional deep neural networks with Q-learning and experience replay, for both scenarios, we were able to train competent bots, which exhibit human-like behaviors. The results confirm the utility of ViZDoom as an AI research platform and imply that visual reinforcement learning in 3D realistic first-person perspective environments is feasible.
I. INTRODUCTION
Atari 2600 benchmarks are limited by their non-realistic 2D, third-person settings, motivating more challenging first-person 3D environments. ViZDoom addresses this need as a customizable, lightweight platform for learning from raw visual information, demonstrated through two deep Q-learning experiments.
- Atari 2600 games use 2D environments that hardly resemble the real world and provide a third-person perspective.
- ViZDoom provides a Doom-based 3D environment in which bots act using only the screen buffer.It also includes a relatively realistic physics model and requires agents to make tactical and strategic decisions.
- Custom scenarios can vary maps, environment elements, non-player characters, rewards, goals, and available actions.
- ViZDoom is lightweight, running at nearly 7000 frames per second on a modern computer using one CPU core.Doom’s real-time rate is 35 frames per second.
- Deep Q-learning experiments tested a basic move-and-shoot task and a 3D maze-navigation task, indicating that deep reinforcement learning can tackle first-person 3D environments.
- Raw visual input in FPS environments removes the need to provide high-level information and handcrafted features, while potentially supporting more believable agent behavior.
II. RELATED WORK
Prior FPS research commonly supplied agents with high-level game information, while raw-visual first-person reinforcement-learning environments remained scarce. The paper motivates Doom as a practical research base because it is recognizable, modifiable, lightweight, and compatible with configurable research scenarios.
- The authors report no previous FPS studies using raw visual information to develop first-person game bots, except work on Minecraft.
- Doom was selected because it met requirements including an open-source 3D FPS base, lightweight execution, speed, process control, customizable rendering, multiplayer capabilities, and scenario-creation tools.
- The candidate-engine comparison considered seven recognizable FPS games using both objective features and subjective criteria such as code complexity.
- Several alternatives were rejected because of limitations involving system requirements, screen-buffer access, modification, synchronization, scripting, complexity, or platform dependence.
- ViZDoom is based on the actively supported, open-source ZDoom engine and retains Doom’s software-rendering advantages for screen-buffer access.
B. Application Programming Interface (API)
ViZDoom’s API is designed for reinforcement and apprenticeship learning, exposing visual observations, agent actions, selected state variables, and extensive configuration through multiple language bindings.
- The API retrieves the game’s screen buffer and maps actions to keyboard buttons, button combinations, and mouse actions.
- Some game-state variables, including player health and ammunition, are available directly through the API.
- The C++ API provides control-mode and rendering options, with additional bindings for Python and Java.
- Figure 2 presents an example of using the Python API.
C. Features
ViZDoom supports varied AI experiments through multiple control modes, custom scenarios, depth-buffer access, and off-screen rendering without requiring a graphical interface.
- ViZDoom provides different control modes, custom scenarios, access to the depth buffer, and off-screen rendering.Off-screen rendering eliminates the need to use a graphical interface.
1) Control modes:
ViZDoom offers multiple control modes, including deterministic synchronous operation, human-observing spectator modes, networked multiplayer, and customizable scenarios.
- Control modes: ViZDoom implements synchronous player, synchronous spectator, asynchronous player, and asynchronous spectator control modes.
- Control modes: Synchronous modes wait for the decision maker, allowing learning systems to operate at their own pace without temporal constraints.
- Control modes: Deterministic synchronous execution supports experimental reproducibility and debugging.
- Control modes: Player modes let agents act, whereas spectator modes let humans control the game while agents observe.
- Control modes: Asynchronous multiplayer supports games with up to eight human or bot players over a network.
- Custom scenarios: Custom scenarios define maps, mechanics, terminal conditions, and rewards, enabling difficulty matching the assessed learning algorithms.
3) Depth Buffer Access:
ViZDoom exposes depth information alongside visual input and is designed to support computationally efficient rendering for learning experiments.
- Depth buffer: ViZDoom provides access to the renderer’s depth buffer to help agents interpret visual information and test object-localization learning.The depth channel can also simulate distance sensors used in mobile robots.
- Performance: Off-screen rendering reduces the burden of displaying the game and enables experiments on servers without graphical interfaces.
- Performance: Rendering performance depends mainly on actor count, resolution, and whether the depth buffer is computed.
- Performance: Nearly 7000 low-resolution frames per second can be rendered, with resolution the most important factor affecting processing speed.At low resolutions, rendering time is negligible compared with backpropagation for reasonably complex neural networks.
IV. EXPERIMENTS
The experiments evaluate visual reinforcement learning in a basic move-and-shoot task, using deep Q-learning to study action timing and learning in a 3D maze.
- Experimental aims: The experiments investigate whether reinforcement learning from visual input is feasible and how skipped frames influence learning.
- Basic scenario: The basic scenario places an agent and stationary monster in a rectangular chamber, with strafing and shooting as available actions.
- Basic scenario: The episode ends after the monster is killed or 300 frames, while rewards encourage fast elimination, preferably with one shot.The agent receives 101 points for a kill, −5 for a missed shot, and −1 for each action.
- Learning procedure: The learning procedure models the task as a Markov Decision Process and uses Q-learning with a convolutional neural network, epsilon-greedy decay, stochastic gradient descent, and experience replay.No target-network freezing was used.
3) Experimental Setup:
The experiment uses a convolutional Q-network with frame skipping to study learning speed, smoothness, final performance, robustness, and training-time trade-offs. Skipcounts of 4–10 provide the best overall balance, while larger values improve learning dynamics but reduce control precision.
- Neural Network Architecture: The network uses two convolutional layers followed by max-pooling, then a fully connected layer and eight-unit output layer for action combinations.The convolutional layers use 32 filters of widths 7 and 4 pixels; the output units correspond to combinations of left, right, and shooting actions.
- Frame Skipping: Larger skipcounts generally make learning faster and smoother, reduce irrational behaviors, and lower variance, but excessive skipping makes agents clumsy.Skipping frames means repeating the last decision during omitted frames, limiting fine-grained control at high skipcounts.
- Frame Skipping: Skipcount 4 is optimal in the scenario, while values up to 10 achieve scores close to the maximum.The results identify skipcount 4 as the best native setting, with higher values through 10 remaining competitive.
- Robustness: Agents trained with larger skipcounts are generally more robust to evaluation settings, whereas those trained below 4 are least robust.The agent trained with skipcount 4 was also the best when executed with skipcount 0.
- Training Cost: Increasing skipcount affects total learning time only slightly because higher values increase episode-restart overhead.Higher skipcounts produce more episodes, which accounts for most of the additional time.
- Practical Choice: Skipcounts from 4–10 balance learning speed and final performance, motivating schedules that begin high and decrease for fine-tuning.The proposed schedule exploits steeper early learning and later fine-grained control.
B. Medikit Collecting Experiment
The medikit-collecting experiment tests whether deep reinforcement learning can handle a more spatially demanding 3D maze task. Agents must survive on an acid surface by collecting medikits and avoiding poison, with shaping rewards used during training.
- Environment: Agents spawn randomly in an acid maze, collect medikits, and avoid poisonous blue vials appearing at random locations.The acid surface continuously reduces life, making item selection central to survival.
- Rewards: Training uses shaping rewards of 100 for collecting a medikit and −100 for collecting a vial, without counting them toward final score.These rewards help the agent learn the intended objective during training.
- Evaluation: Each episode lasts up to 2100 ticks or ends when the agent dies, while idling produces a score of 284.The maximum achievable score is 2100.
2) Experimental Setup:
The maze experiment used a convolutional deep Q-learning agent with visual and state inputs, replay-based training, and frame skipping. Learning produced competent but imperfect navigation, with performance limitations linked to hesitation and unstable dynamics.
- Learning procedure: Training used RMSProp, replay memory of 10 000 elements, mini-batches of 64, and one million learning steps.The full learning process, including testing, lasted 29 hours.
- Game and network settings: The network used three convolutional layers, a 1024-unit leaky-ReLU fully connected layer, and 16 outputs for action combinations.The convolutional filters were 7, 5, and 3 pixels wide.
- Game and network settings: The agent received 120 × 45 RGB images, health points, tick number, and four consecutive states as input.Health and ammunition were fed directly to the first fully connected layer, and skipcount was 10.
- Learning results: The agent periodically achieved a perfect score, while its average score gradually reached 1300 by the end of training.The learning curve suggested that additional training might still improve performance.
- Limitations: For some initial states, the agent still failed to survive longer than a random player, and the learning dynamics contained unexplained temporary score drops.The scenario was demanding because medikits were scarce and required sustained focus even from human players.
- Learned behavior: The learned policy sought medikits, avoided poison, backpedaled from dead ends, and avoided pushing against walls, but often hesitated between directions.The authors identify this alternating left-right behavior as the most probable direct cause of unsatisfactory performance.
V. CONCLUSIONS
ViZDoom is presented as a flexible, efficient platform for vision-based reinforcement learning in semi-realistic, first-person 3D environments. Experiments show that deep Q-learning can produce human-like navigation behavior, while frame skipping affects learning speed and performance.
- ViZDoom provides a lightweight, efficient, highly customizable platform for vision-based reinforcement learning in a semi-realistic first-person 3D world.Its API supports multiple learning paradigms and gives users control over custom scenarios.
- Deep Q-learning produced competent behavior in both move-and-shoot and hostile-maze scenarios.The maze agent learned to navigate surprisingly well and exhibited evidence of human-like behavior, although it did not find a perfect strategy.
- Skipping 4 to 10 frames provided the best learning-speed trade-off in the move-and-shoot scenario.The authors used this finding in the more involved maze-navigation experiment.
- Current limitations include deaf bots and the absence of synchronous multiplayer and automatic object labeling.These are identified as possible directions for extending ViZDoom.