Source-linked AI summary
Playing FPS Games with Deep Reinforcement Learning
Guillaume Lample, Devendra Singh Chaplot
TL;DR
The paper addresses deep reinforcement learning in challenging FPS settings where agents must act from pixels in partially observable 3D environments. It augments DRQN with jointly trained game-feature predictions and modular networks for navigation and action, achieving substantial improvements and outperforming built-in agents and humans in deathmatches.
Problem
Prior deep reinforcement learning applications mainly used 2D environments, while FPS deathmatches require pixel-based control in partially observable 3D scenes with navigation, item collection, and combat.
Method
The agent augments DRQN with game-feature information during training and modularizes control into separate networks for navigation and action phases.
Results
The architecture substantially outperforms built-in game agents and human players in deathmatch scenarios.
Takeaways & Limitations
Joint game-feature co-training and phase-specific networks provide dramatic improvements over standard DRQN on complicated deathmatch tasks and generalize to unknown maps.
Takeaways & Limitations
The game-feature information used for augmentation is available during training but not at test time, and the analogous DQN setup did not exceed 70% enemy-detection accuracy.
Abstract
from arXiv · showhide
Advances in deep reinforcement learning have allowed autonomous agents to perform well on Atari games, often outperforming humans, using only raw pixels to make their decisions. However, most of these games take place in 2D environments that are fully observable to the agent. In this paper, we present the first architecture to tackle 3D environments in first-person shooter games, that involve partially observable states. Typically, deep reinforcement learning methods only utilize visual input for training. We present a method to augment these models to exploit game feature information such as the presence of enemies or items, during the training phase. Our model is trained to simultaneously learn these features along with minimizing a Q-learning objective, which is shown to dramatically improve the training speed and performance of our agent. Our architecture is also modularized to allow different models to be independently trained for different phases of the game. We show that the proposed architecture substantially outperforms built-in AI agents of the game as well as humans in deathmatch scenarios.
1 Introduction
Deep reinforcement learning has excelled in largely 2D, fully observable tasks, but FPS games require agents to act from pixels in challenging, partially observable 3D environments. The paper introduces an agent that combines game-feature co-training with separate networks for navigation and combat, substantially outperforming built-in agents and humans in deathmatches.
- Deep reinforcement learning methods have performed well on tasks including Atari games, but commonly assume complete environmental state information.
- FPS games pose a harder setting because agents navigate partially observable 3D environments while collecting items, recognizing enemies, and fighting.
- The proposed pixel-based agent divides deathmatches into navigation and action phases and uses separate networks for each phase.
- The agent infers high-level game information, including enemy presence, to select its current phase and improve performance.The method co-trains a DQN with game features, which guides convolutional layers to detect enemies.
- The proposed architecture substantially outperforms built-in Doom agents and human players in deathmatch scenarios.The evaluation uses two tasks adapted from the ViZDoom AI Competition and also examines each architectural component.
2 Background
The background introduces DQN value estimation and Q-learning, then explains how recurrence extends these methods to partially observable environments. The proposed architecture builds on DRQN and jointly predicts game features alongside Q-learning objectives.
- Reinforcement learning learns a policy that selects actions from observations and rewards to maximize discounted future returns.
- 2.1 Deep Q-Networks: DQN uses a neural network to approximate the optimal action-value function, Q*(s, a), through Bellman-based Q-learning updates.
- 2.1 Deep Q-Networks: Experience replay stores transitions and trains Q-learning updates on randomly sampled minibatches, while epsilon-greedy exploration mixes random and highest-valued actions.
- 2.2 Deep Recurrent Q-Networks: In partially observable environments, DRQN estimates action values from the current observation and a recurrent hidden state that summarizes prior observations.An LSTM can update the hidden state recursively, and the paper's model is built on DRQN.
- 2.2 Deep Recurrent Q-Networks: The proposed architecture splits convolutional outputs into an LSTM stream and a game-feature stream, jointly training feature predictions with Q-learning objectives.
3 Model
The model augments recurrent visual control with game-feature detection and separates navigation from combat through phase-specific networks. This design addresses weak enemy detection and supports modular training for FPS deathmatches.
- 3 Model: Baseline DRQN agents performed poorly in deathmatches because they could not accurately detect enemies and fired indiscriminately.Ammo penalties either failed to stop firing or prevented firing entirely.
- 3.1 Game feature augmentation: The augmented model jointly trains the DRQN objective with game-feature prediction, sharing convolutional layers to guide enemy detection.At training time, the network receives visual frames and entity-presence indicators, although those indicators are unavailable during testing.
- 3.1 Game feature augmentation: Using only enemy presence as a game feature dramatically improved performance across every tested scenario.The model reached optimal enemy-detection accuracy of 90% after a few hours, accelerating subsequent LSTM training.
- 3.1 Game feature augmentation: A standard DQN handled stacked frames poorly for feature prediction, never exceeding 70% enemy-detection accuracy even with dropout.The difficulty arises because the feature target concerns only the last frame while convolutional layers process all stacked frames.
- 3.2 Divide and conquer: The architecture divides deathmatches into navigation and action phases, using separate networks selected according to enemy visibility and ammunition.The action network is a game-feature-augmented DRQN, while navigation uses a simple DQN.
- 3.2 Divide and conquer: Phase-specific networks make the architecture modular and allow independent or parallel training for faster development.Navigation also requires only three actions: move forward, turn left, and turn right.
4 Training
Training combines shaped rewards, frame skipping, and history-aware recurrent updates to make sparse-reward FPS learning more tractable. The procedure also evaluates training behavior through K/D curves and human-comparison scenarios.
- 4.1 Reward shaping: Reward shaping adds intermediate signals to address sparse replay data and delayed rewards in deathmatch learning.The score is based on kills minus suicides, but favorable behavior often requires many actions before receiving feedback.
- 4.1 Reward shaping: Navigation receives positive rewards for item pickups and movement, while lava contact receives a negative reward.A distance-proportional reward encourages faster exploration and discourages turning in circles.
- 4.2 Frame skip: Frame skipping repeats each selected action across skipped frames; k = 4 provided the best trade-off between training speed and control precision.Higher skip rates accelerate training but can prevent the small rotations needed for accurate aiming.
- 4.3 Sequential updates: Figure 4 tracks action-network K/D over training time for dropout, game-feature, and LSTM-update-count variations.The figure is intended to compare how these training choices affect learning curves on limited deathmatch.
- 4.3 Sequential updates: Recurrent updates ignore early sequence states with insufficient history and backpropagate through later states while using the final state to form a target.Experiments used a minimum history of 4 and updated 5 states; increasing the update count affected performance.
5 Experiments
Experiments evaluate deathmatch performance on known and unknown maps using frags, K/D ratio, kills, objects gathered, deaths, and suicides. Navigation, game-feature co-training, and dropout each improve key outcomes, while the agent outperforms human players in the reported scenarios.
- Experimental Setup: Deathmatch experiments use ViZDoom, comparing limited deathmatch on a known map with full deathmatch across unknown maps.Full deathmatch trains on 10 maps and tests on 3 maps; Table 2 averages performance over those maps.
- Evaluation Metrics: K/D ratio, kills, objects gathered, deaths, and suicides jointly measure combat effectiveness, exploration, and design effects.The final score is frags, while suicides penalize K/D because they count as deaths.
- Navigation Network Enhancement: Navigation dramatically increases object collection and improves K/D ratio in both deathmatch scenarios.With navigation, the agent collected more than three times as many objects in limited deathmatch; the K/D improvement was larger in full deathmatch.
- Comparison to Human Players: The agent outperforms human players in both single-player and multiplayer deathmatch scenarios.Human scores were averaged over 20 players; the single-player comparison used 10 bots for three minutes, and multiplayer used five minutes.
- Game Features: After 65 hours, game-feature co-training raises the best K/D score from below 2.0 to above 4.0.The comparison is between networks trained without and with game features.
- Game Features: Enemy-detection accuracy reaches 90% with dropout versus 70% without, and the authors infer that dropout is crucial for effective performance.Figure 4 supports the inference that dropout significantly improves the action network on limited deathmatch.
- Game Features: Game features improve results when used for co-training, but not when supplied directly as DQN inputs.The paper suggests this co-training strategy may also be useful in other DQN applications.
6 Related Work
Related work includes approaches that divide FPS navigation and combat, recurrent learning from raw pixels, and vanilla DQN studies in Doom.
- FPS Reinforcement Learning: Prior FPS work divided navigation and combat while using game-engine information in reinforcement-learning approaches.The cited studies are McPartland and Gallagher (2008) and Tastan and Sukthankar (2011).
- Pixel-Based Learning: Koutník et al. applied a recurrent neural network to learn TORCS from raw pixels only.TORCS is identified as a racing video game.
- Doom-Based Learning: Kempka et al. applied a vanilla DQN to simpler Doom scenarios and studied the effect of changing skipped-frame counts.This work provides related empirical analysis within Doom.
7 Conclusion
The paper presents a complete architecture for FPS deathmatch that combines game-information augmentation with modular networks for different game phases. It reports improvements over standard DRQN, built-in bots, and human players, including generalization to unknown maps.
- Architecture: The architecture augments DRQN with high-level game information and uses independent networks for different phases of deathmatch.The conclusion identifies navigation and action as separate phases handled within a modular architecture.
- Performance: These methods produce dramatic improvements over standard DRQN on complicated deathmatch tasks.The conclusion frames deathmatch as the target setting for the comparison.
- Performance: The proposed model outperforms built-in bots and human players and generalizes to unknown maps.The conclusion reports both comparative performance and map generalizability.
- Extensions: The methods are complementary to dueling architectures and prioritized replay.The paper states that these recent DQN improvements could be combined with the proposed methods.