Source-linked AI summary
Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, Thore Graepel
TL;DR
Multi-agent reinforcement learning still lacks strong evidence in complex environments where independently learning agents cooperate and compete from perceptual inputs. This work combines population-based training, learned internal rewards, and a temporally hierarchical agent for Quake III Arena Capture the Flag, where trained agents exceeded strong human players’ win rates in tournament-style evaluation. The study also examines learned game representations and behaviour, while noting important differences between human and agent interfaces.
Problem
Complex multi-agent games combining team and competitive play remain difficult for end-to-end reinforcement learning, especially when agents learn concurrently from perceptual inputs.
Method
The approach trains a population of independent RL agents concurrently, using learned internal rewards, population-based optimisation, and a temporally hierarchical representation across generated CTF environments.
Results
Trained agents exceeded strong human players’ win rates as teammates and opponents and were stronger than the evaluated state-of-the-art agents.
Takeaways & Limitations
The agents’ learned representations encoded CTF state and their behaviour included interpretable patterns such as defending, following, camping, respawning, and navigation.
Takeaways & Limitations
Human-agent comparisons remain constrained by unequal interfaces: humans receive higher-resolution, lower-latency observations and higher-fidelity, higher-resolution controls.
Abstract
from arXiv · showhide
Recent progress in artificial intelligence through reinforcement learning (RL) has shown great success on increasingly complex single-agent environments and two-player turn-based games. However, the real-world contains multiple agents, each learning and acting independently to cooperate and compete with other agents, and environments reflecting this degree of complexity remain an open challenge. In this work, we demonstrate for the first time that an agent can achieve human-level in a popular 3D multiplayer first-person video game, Quake III Arena Capture the Flag, using only pixels and game points as input. These results were achieved by a novel two-tier optimisation process in which a population of independent RL agents are trained concurrently from thousands of parallel matches with agents playing in teams together and against each other on randomly generated environments. Each agent in the population learns its own internal reward signal to complement the sparse delayed reward from winning, and selects actions using a novel temporally hierarchical representation that enables the agent to reason at multiple timescales. During game-play, these agents display human-like behaviours such as navigating, following, and defending based on a rich learned representation that is shown to encode high-level game knowledge. In an extensive tournament-style evaluation the trained agents exceeded the win-rate of strong human players both as teammates and opponents, and proved far stronger than existing state-of-the-art agents. These results demonstrate a significant jump in the capabilities of artificial agents, bringing us closer to the goal of human-level intelligence.
1 Task
Capture the Flag is a five-minute team game in which players capture the opposing flag while protecting their own. The study uses DeepMind Lab, a modified Quake III Arena environment with procedurally generated indoor and outdoor maps.
- Rules of Capture the Flag: CTF teams score by taking the opposing flag to their own base while their own flag remains safe.Players can tag opponents, causing delayed respawn; tagged carriers drop the flag, which can be returned or picked up.
- Environment: The environment is DeepMind Lab, a modified version of Quake III Arena that retains core game mechanics while reducing visual violence cues.Unlike many prior multi-agent settings, the approach uses end-to-end learning from independent agents and low-level visual inputs.
- Environment: Indoor maps are procedurally generated, flat and maze-like, whereas outdoor maps are open, hilly, and populated with varied terrain and objects.Each map type places team bases and flags in structured but different configurations.
- Environment: Randomly generating maps each episode provides a large set of environments, with some random seeds reserved for evaluation.This setup tests performance beyond the maps used during training.
2 Agent
The FTW agent combines partially observable recurrent RL with hierarchical latent representations, learned internal rewards, and population-based optimisation. Its distributed training system updates many interacting agents across parallel games.
- Reinforcement-learning formulation: In a partially observed environment, the agent maximises expected discounted return conditioned on its observation history rather than the full game state.The value function is represented as a random variable because observations do not uniquely determine the true state.
- Latent-variable policy: A KL-regularised latent-variable objective combines return maximisation with consistency between a prior policy and a variational posterior.The latent variable models dependence on past observations, while auxiliary value and pixel-control targets shape its representation.
- Temporal hierarchy: A hierarchical RNN uses fast and slow LSTM cores to support forward planning and memory at different timescales.The fast core updates every environment step, while the slow core updates every τ steps and supplies a prior for the latent state.
- Temporal hierarchy: The fast core receives CNN-encoded observations, previous actions, previous rewards, and prior-distribution parameters before producing recurrent dynamics.The slow core takes the fast core’s hidden state as input; policy, value, and pixel-control signals are derived from latent samples.
- Internal reward and optimisation: A learned transformation of game-point signals supplies more frequent internal rewards than the sparse win, draw, or loss outcome.The transformation is adapted so discounted internal rewards support maximising winning probability.
- Internal reward and optimisation: Population Based Training jointly optimises internal rewards and agent hyperparameters across a parallel population of 30 agents.Optimised hyperparameters include learning rate, the slow-LSTM timescale, KL-term weight, and entropy cost.
- Distributed training: A distributed actor-learner framework runs 1920 arena processes that continually play games using players sampled from the live training population.This architecture addresses the high simulation cost of interacting RL agents.
3 Performance Evaluation
Performance was evaluated with extended team Elo ratings across agents, bots, and human participants, including tests of human exploitative strategies and human–agent interface differences.
- Evaluation design: Team Elo ratings were estimated from evaluation games involving assessed agents, built-in bots, and human participants.The rating system was extended to teams for Capture the Flag evaluation.
- Baselines and ablations: The ablation study compared UNREAL, self-play with reward shaping, population-based training with reward shaping, and FTW without temporal hierarchy.UNREAL served as the state-of-the-art naive baseline.
- Human evaluation: 40 experienced human players competed in tournaments with randomly matched humans, FTW agents, and FTW agents without temporal hierarchy as teammates and opponents.Humans were pooled into strong and average groups for statistically valid Elo estimates.
- Human exploitation: After six hours of practice on one held-out map, two professional testers consistently beat FTW using route analysis and coordinated flag captures.The strategy exploited preferred agent routes on that fixed map.
- Human exploitation: 25% human win-rate was observed when maps were procedurally generated each episode, preventing a consistently winning strategy for the testers.The draw rate under these conditions was 6.3%.
- Human–agent differences: Humans received higher-resolution, lower-latency observations and higher-fidelity controls, whereas agents had advantages in sensorimotor precision and perception.The authors identify these differences as an obstacle to an even human–agent comparison.
- Human–agent differences: FTW agents responded to visible opponents in 258ms on average versus 559ms for humans, and successful tags took 233ms versus 627ms.The comparison concerns the interval from first seeing an opponent to attempting or successfully completing a tag.
- Human–agent differences: 80% tagging accuracy for FTW agents exceeded the 48% measured for humans, while impairing agent accuracy reduced win probability.At human-comparable accuracy, FTW agents still had greater win probability than humans.
4 Analysis
The analysis examined what FTW agents encode internally and how visual inputs influence decisions, finding representations that capture game state, behavior, and task-relevant scene information.
- Knowledge representation: Linear probes tested whether recurrent hidden states encoded 200 binary features concerning distant, recent, present, and future game states.The probes used balanced logistic regression on 4.5 million game situations with episode-wise cross-validation.
- Knowledge representation: t-SNE revealed clusters reflecting flag possession, agent location, and respawn state in the FTW representation space.These clusters represented conjunctions of known Capture the Flag state elements.
- Knowledge representation: Single-neuron responses represented whether an agent’s flag was held by an opponent or a teammate held the opponents’ flag.The result indicates that some game-state distinctions were localized in individual neurons.
- Visual analysis: Gradient-based saliency showed that value, policy, and neuron responses were sensitive to flags, other agents, and on-screen information.The exact scene objects influencing the value function were often context dependent.
- Behavior analysis: Two-second gameplay segments were compressed with a recurrent VAE and clustered into 32 behavioral prototypes using a Gaussian mixture model.The resulting cluster sequences represented high-level agent play.
5 Experiment Details
The experiment details define a team-extended Elo model, its likelihood-based fitting procedure, and the pixel and action interfaces used for agent evaluation and population-based training.
- Elo calculation: Team Elo ratings assume that a team’s rating equals the sum of its members’ skills.This extends the standard one-versus-one Elo formulation to n-versus-n Capture the Flag matches.
- Elo calculation: Each agent i receives a real-valued rating ψ_i, while match vector m records blue-team appearances minus red-team appearances.The match representation supports the team-level win-probability model.
- Elo calculation: P(blue wins against red|m, φ) = 1 / (1 + 10^(-ψ^T m/400)) gives the modeled probability of a blue-team win.Ratings are fitted by maximizing match-outcome likelihood, with Bot 4 anchored at 1000 for interpretation.
- Population-based training: For population-based training, the winning probability of π_i versus π_j assumes both team members are π_i against two copies of π_j.The corresponding match-vector entries are m_i = 2 and m_j = −2.
- Agent interface: Agents observe 84×84 RGB pixels scaled to [0, 1]^84×84×3.The observation uses three byte-valued color channels.
- Agent interface: The simplified composite action space contains 540 possible actions across yaw, pitch, strafing, movement, tagging, and jumping.Yaw exposes two values and pitch one value in the configured interface.
5.3 Procedural Environments
Procedural environments varied indoor mazes and outdoor terrain while enforcing structured flag and spawn layouts, symmetry, solvability, and map randomization.
- Indoor procedural maps: Indoor maps are flat, point-symmetric mazes with rooms, corridors, two flag bases, spawn points, contextual colors, and randomized wall artwork.The bases occupy opposite ends through the construction procedure.
- Indoor procedural maps: Indoor generation creates random rectangular rooms and fills spaces between them with backtracking-generated corridors.Room edges and corridor backtracking are restricted to even cells.
- Indoor procedural maps: Dead ends and horseshoes are removed from indoor mazes before bases and spawn points are added symmetrically.These steps shape navigable layouts and preserve paired team geometry.
- Indoor procedural maps: Indoor maps are checked for solvability, minimum base-room area, and minimum flag separation.The base room must be at least 9 units in area.
- Map randomization: Maps are randomly rotated to prevent agents from exploiting the skybox for navigation.Rotation adds another source of environmental variation.
- Outdoor procedural maps: Outdoor maps are open, hilly, naturalistic environments with obstacles and rugged terrain, placing team flags and spawns at opposite corners.Height maps use low- and high-variance diamond-square fractal generation.
5.4 Training Details
Training used long, parallelized CTF experience with population-based matchmaking, internal reward signals, and a temporally hierarchical recurrent policy.
- 2 billion steps corresponded to approximately 450K five-minute training games.Each game lasted 4500 agent steps, with observations received 15 times per second and actions repeated four times.
- A population of 30 agents tuned learning rates through population-based training, with parameters updated from batches of 32 trajectories of length 100.The initial learning rate was sampled per agent from LogUniform(10^-5, 5 · 10^-3).
- The policy used a multi-timescale recurrent neural network with external memory, generating actions from a latent variable controlled by a slower prior process.Its variational objective traded off expected reward against consistency between the two inference timescales.
- Training games randomly sampled maps, population agents, and teams, with three additional agents selected according to Elo-based skill probabilities.The process was repeated after every five-minute episode, and agents were randomly assigned to red and blue teams.
5.5 Game Events
The game outcome was too sparse to guide reinforcement learning alone, so the method used game-point events to construct more frequent internal rewards.
- 13 binary game events were assigned unique game-point values for manual reward-shaping baselines.These values came from the default Quake III Arena points system.
- The event system included penalties for being tagged and rewards for tagging opponents, with separate values for flag possession.
- Agents lacked direct access to the listed events, instead receiving internal rewards computed from available game-point signals.The internal reward function was initialized independently for each population agent and adapted through reward evolution.
5.6 Ablation
Ablations compared self-play, manual reward shaping, population-based training, evolved internal rewards, and temporal hierarchy across procedural indoor and outdoor maps.
- Two ablation series evaluated the methods separately on procedural indoor and procedural outdoor maps.
- Self-play trained one LSTM policy with four identical agents, using only terminal rewards of +1 for winning, -1 for losing, and 0 for draws.Population-based training and Elo scoring were disabled because only one policy was trained.
- Self-play + Reward Shaping added manual Quake III reward shaping, while PBT + Reward Shaping sampled four population agents and used Elo scores for population-based training.
- FTW without temporal hierarchy replaced manual shaping with an internal reward signal evolved by population-based training.
- FTW combined a temporally hierarchical recurrent core with matchmaking, population-based training, and an evolved internal reward signal.
5.7 Distinctly Selective Neurons
Distinctly selective neurons were identified by testing whether individual activations discriminated game-state features with shallow decision trees.
- A depth-1 Decision Tree identified the neuron most selective for each game-state feature using Gini impurity.The analysis used activations and labels from 100 episodes against Bot 3.
- A neuron was considered distinctly selective when the resulting stump exceeded 97% accuracy over 100 · 4500 steps.
5.8 Behavioural Analysis
The behavioural analysis represents two-second gameplay clips with agent-centric game-state features, then organizes them into interpretable behavioural clusters. This enables comparisons of behavioural patterns across agents and training conditions.
- Behavioural representation: The feature set records team and opponent context, including room occupancy, base visibility, flag status, and whether agents are respawning.Flag status distinguishes flags at base, held by teammates, held by opponents, held by the agent, or stray.
- Behavioural representation: A VAE trained on mixed gameplay data maps each 30×56 clip to a 128-dimensional latent variable.The dataset combines 1000 episodes of each analysed agent playing against pairs of Bot 3 agents.
- Behavioural clusters: 32-component Gaussian-mixture clustering converts the VAE latent representations into behavioural clusters for characterising two-second CTF clips.The GMM is fit to 400K sampled latent means with diagonal covariance.
5.9 Bot Details
The supplementary bot details describe the procedurally generated evaluation settings, agent comparisons, and analyses of generalisation and learned representations. They also specify the Quake III Arena bots used as evaluation opponents.
- Bot details: The study uses Tauri and Centuri Quake III Arena bots as evaluation opponents.These bots are the specified bot pair used for evaluation.
- Training environments: Agents were trained on two procedurally generated map styles: outdoor maps and indoor maps.The two styles were used to demonstrate robustness across distinct map-generation settings.
- Evaluation settings: Evaluation tournaments varied map sizes from 13–21, team sizes from 1–4, and indoor or outdoor styles, with teams sampled from the evaluated agent set.Indoor agents were trained on team size two and a mixture of 13 × 13 and 17 × 17 maps, making other settings generalisation tests.
- Evaluation results: In two-player fetch, FTW scored 37 flags per match, compared with 23 for FTW without temporal hierarchy and 14 for Self-play + RS.The comparison used CTF-trained agents on a fetch task without opponents.
- Representation analyses: Representation analyses compare FTW with Self-play + RS using saliency, memory-reading patterns, t-SNE, neuron activations, and linear probes.The analyses examine sensitivity to pixels, memory use, game-state clustering, selective neurons, and prediction of high-level state features.