Source-linked AI summary
Google Research Football: A Novel Reinforcement Learning Environment
Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zając, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, Sylvain Gelly
TL;DR
Reinforcement learning research needs environments that are challenging, accessible, customizable, and able to support stochastic and multi-agent research. This paper introduces Google Research Football with a physics-based football simulator, benchmarks, and academy scenarios, showing difficulty ranging from minute-scale unit tests to tasks requiring tens or hundreds of millions of steps.
Problem
Existing reinforcement learning environments can be easy to solve, computationally expensive, deterministic, single-player, or restricted in licensing and customization.
Method
The paper introduces an open-source football environment comprising a physics-based Football Engine, Football Benchmarks, Football Academy scenarios, and baseline evaluations with common reinforcement learning algorithms.
Results
The Football Benchmarks span meaningful difficulty, while Football Academy scenarios range from tasks solvable in 1M steps to harder tasks requiring 5M–50M steps or 500M training steps.
Takeaways & Limitations
Google Research Football provides customizable reference tasks and diverse scenarios for studying self-play, sample-efficient reinforcement learning, sparse rewards, and model-based reinforcement learning.
Takeaways & Limitations
The environment is motivated partly by unresolved challenges around stochasticity and multi-agent learning, which remain research questions rather than established solutions.
Abstract
from arXiv · showhide
Recent progress in the field of reinforcement learning has been accelerated by virtual learning environments such as video games, where novel algorithms and ideas can be quickly tested in a safe and reproducible manner. We introduce the Google Research Football Environment, a new reinforcement learning environment where agents are trained to play football in an advanced, physics-based 3D simulator. The resulting environment is challenging, easy to use and customize, and it is available under a permissive open-source license. In addition, it provides support for multiplayer and multi-agent experiments. We propose three full-game scenarios of varying difficulty with the Football Benchmarks and report baseline results for three commonly used reinforcement algorithms (IMPALA, PPO, and Ape-X DQN). We also provide a diverse set of simpler scenarios with the Football Academy and showcase several promising research directions.
Introduction
The paper introduces a challenging, customizable, open-source football environment for reinforcement learning, built around a physics-based 3D simulator. It targets research requiring stochasticity, multi-agent interaction, and decisions across multiple levels of abstraction.
- Motivation: Existing reinforcement learning environments can be too easy, computationally expensive, deterministic, single-player, narrowly focused, or restricted by licensing.These limitations motivate a more broadly useful research environment.
- Research Scope: The environment is intended to support research on stochasticity, self-play, multi-agent setups, model-based reinforcement learning, tactics, and strategy.It also emphasizes customizable difficulty and accessibility for research.
- Contribution: The Football Engine provides a physics-based 3D simulation of football with major rules including kickoffs, goals, fouls, cards, corners, penalties, and offsides.The simulator is designed to model a complete football game.
- Contribution: The proposed environment trains agents to control football players, pass between them, and overcome defenses to score goals.Football combines short-term control, learned passing concepts, and high-level strategy.
- Contribution: The paper contributes Football Benchmarks and Football Academy scenarios for comparing algorithms and studying progressively harder reinforcement learning tasks.The contributions include benchmark tasks of varying difficulty and diverse scenarios.
Motivation and Other Related Work
The paper positions Google Research Football against limitations in existing reinforcement learning environments, including restricted complexity, realism, accessibility, and interaction structure. It also distinguishes the environment from other football and task-specific simulators by emphasizing high-level football actions and multiple research settings.
- Existing Environment Limitations: Many established environments are solved quickly, while newer video-game simulators can require substantial computational resources.The paper identifies ease of solution and computational expense as opposing limitations in existing environments.
- Existing Environment Limitations: Deterministic environments may not adequately test whether reinforcement learning methods generalize to stochastic settings.Artificial randomness can remain structured and predictable during training.
- Existing Environment Limitations: Restrictive licenses and closed-source binaries can prevent researchers from inspecting or modifying environment implementations.The paper contrasts these constraints with the research flexibility enabled by open-source licenses.
- Existing Environment Limitations: Single-player environments omit challenges involving adaptive opponents, cooperation, communication, specialization, and robustness to agent failure.Multiplayer environments can support research on both collaborative and competitive multi-agent learning.
- Related Work: Compared with other football simulators, Google Research Football focuses on high-level actions rather than low-level robot-physics control and supports single-agent, multi-agent, single-player, and multiplayer settings.The paper also situates related environments around narrower topics such as continuous control, safety, or multi-agent play.
Football Engine
The Football Engine is a customizable, physics-based football simulator supporting standard rules, varied observations, stochasticity, configurable opponents, and multiple research interfaces.
- Simulation: The engine simulates complete 11-versus-11 football games with standard rules, editable game duration and player configurations, and player statistics that change with tiredness.Implemented rules include goals, fouls, cards, offsides, handballs, corners, goal kicks, side kicks, and penalty kicks.
- Opponent control: Researchers can configure opponent difficulty continuously from θ = 0 to 1, use built-in rule-based bots, or replace the opponent with a trained model for self-play.Suggested settings are easy θ = 0.05, medium θ = 0.6, and hard θ = 0.95.
- Observations: The environment supports state-derived observations through pixels, a Super Mini Map, and a 115-dimensional floats representation, with stacking available for some representations.Pixels are 1280 × 720 RGB images; the Super Mini Map uses four 72 × 96 binary matrices for teams, ball, and active player.
- Actions and rewards: Agents choose movement, passing, shooting, sprinting, tackling, dribbling, and player-switching actions, while researchers can use built-in SCORING or CHECKPOINT rewards or define custom rewards.CHECKPOINT adds one-time rewards for first possession in ten regions near the opponent goal, up to +1 total, and adds uncollected rewards when scoring.
- Research interface: The environment can run deterministically or stochastically, supports human inspection and replay storage, and is compatible with the OpenAI Gym API.Stochastic mode is enabled by default, while deterministic mode makes fixed-policy interactions repeat the same action and state sequence.
- Performance: The optimized C++ implementation reaches approximately 140 million steps per day on a single hexacore machine.The engine can run on commodity machines with or without GPU-based rendering.
Football Benchmarks
Football Benchmarks provide standardized full-game tasks at three opponent strengths, enabling comparisons of reinforcement learning algorithms across reward designs and computational settings.
- Benchmark design: Football Benchmarks require agents to win a full game against an opponent bot in easy, medium, or hard scenarios.The three versions differ only in opponent strength, allowing evaluation under different computational constraints.
- Algorithms and setup: The paper reports reference results for PPO, IMPALA, and Ape-X DQN using the same network architecture and stacked Super Mini Map representation.PPO runs in multiple processes on one machine, whereas IMPALA and Ape-X DQN use distributed actors.
- Evaluation protocol: The evaluation compares SCORING and CHECKPOINT rewards across three benchmarks, with hyperparameters tuned on the easy scenario and five random seeds averaged per setting.The protocol is kept consistent across algorithms to support fair comparison.
- Results: The medium benchmark can be beaten by DQN and IMPALA at 500M training steps, while the hard benchmark requires CHECKPOINT and 500M steps for a positive score.The medium result is only barely achieved with SCORING reward.
- Results: CHECKPOINT appears especially helpful for speeding training with policy-gradient methods, whereas Ape-X DQN performs similarly with both reward functions.The authors identify substantial remaining headroom, particularly on harder benchmarks and in sample efficiency.
Football Academy
The Football Academy offers 11 customizable scenarios spanning simple scoring drills, defensive challenges, and standard football situations. Experiments show that these tasks cover a wide range of difficulties and computational requirements.
- Scenario Design: The Football Academy contains 11 scenarios, including empty-goal scoring drills, defensive-line challenges, and corner or counter-attack situations.Researchers can also define their own scenarios through a simple API.
- Difficulty and Results: Empty Goal Close, Empty Goal, and Run to Score can be solved by PPO and IMPALA with either reward function using 1M steps.These scenarios function as unit tests that can produce reasonable results within minutes or hours.
- Difficulty and Results: The remaining tasks generally require 5M to 50M steps for PPO and IMPALA to make progress, with minor differences between SCORING and CHECKPOINT rewards.They support rapid experimentation on single machines before applying ideas to the Football Benchmarks.
- Difficulty and Results: CORNER appears to be the hardest scenario because agents face a full squad while the opponent can also score.Football Academy scores are capped at 1 because episodes end after scoring and can be negative through own goals or opponent scores.
Promising Research Directions
The environment supports research on self-play, multi-agent control, and representation learning. Initial experiments illustrate non-transitive opponents, scaling effects from controlling multiple players, and eventual learning from pixels.
- Self-Play: Training against a fixed opponent can exploit that opponent’s weaknesses and fail to generalize well to other adversaries.A self-play experiment trained agent A against built-in AI, then agent B against a frozen version of A; numerical results appear in Table 2.
- Benchmarking: Figures 6 and 7 report Football Academy average goal differences for IMPALA and PPO using the SCORING reward.These figures provide experimental results for the Football Academy scenarios.
- Multi-Agent Learning: Controlling more players initially slows training, but policies eventually learn more complex behaviors and achieve higher scores.The multi-agent experiment controls 1 to 3 players in the 3 versus 1 with Keeper scenario using IMPALA.
- Representation Learning: Pixel-based observations take significantly longer to train, but learning eventually occurs and outperforms the handpicked extensive Floats representation.The comparison uses IMPALA with CHECKPOINT reward on the easy 11-versus-11 benchmark.
Conclusions
The paper presents Google Research Football as an open-source, customizable environment for challenging football reinforcement learning experiments. Its simulator, benchmarks, and academy scenarios support research across several current RL problems.
- Conclusions: The environment combines a highly optimized C++ football simulator, Football Benchmarks, and progressively harder Football Academy scenarios.The paper presents these components as resources for comparing algorithms and investigating research challenges.
- Conclusions: The authors expect the environment to support research on self-play, sample-efficient reinforcement learning, sparse rewards, and model-based reinforcement learning.These directions are stated as intended uses of the environment’s components.
Hyperparameters & Architectures
The experiments use IMPALA, PPO, and Ape-X DQN with a shared representation strategy and large-scale randomized training runs. Their architectures differ mainly in policy or value outputs and in how acting and learning are distributed.
- Architectures: The experiments use IMPALA, PPO, and Ape-X DQN, with the model architecture inspired by the Large architecture from Espeholt et al. (2018).Figure 8 depicts the architecture used for IMPALA and PPO; Ape-X DQN uses a similar network with Q-value outputs.
- Representations: The stacked Super Mini Map was selected as the default representation for all Football Benchmarks and Football Academy experiments.This choice was based on the representation experiments, while three other representations were also available.
- Training Protocol: Each of six benchmark settings uses five random seeds for 500 million steps, while all 11 Football Academy scenarios use five seeds for 50 million steps.The benchmark settings combine three tasks with two reward functions.
- Hyperparameter Search: Hyperparameter searches evaluate 100 parameter sets with three random seeds separately for SCORING and CHECKPOINT rewards.The best set is selected by average performance, after 500M steps for IMPALA and Ape-X DQN or 50M steps for PPO.
- Algorithms: IMPALA decouples acting from learning and uses V-trace to handle off-policy data, whereas PPO is an online clipped policy-gradient method.PPO runs over 16 parallel workers in these experiments.
- Algorithms: Ape-X DQN decouples acting from learning through a distributed replay buffer and uses a scalable DQN variant with dueling and double Q-learning.Several hyperparameters, including unroll length, n-step return, actor count, and discount factor, were aligned with IMPALA.
Benchmarks
The benchmark materials report five-run means and standard deviations for IMPALA, PPO, and DQN under SCORING and CHECKPOINT rewards, alongside hyperparameter settings and Football Academy scenario details. Additional figures show Academy average goal differences for IMPALA and PPO with CHECKPOINT reward, although the PPO plot contains only v1.x results.
- Benchmark results: Five-run means and standard deviations are reported for all algorithms in the Football Benchmarks under SCORING and CHECKPOINT rewards.These results correspond to Tables 5 and 6 and were presented in the main paper’s Figure 4.
- Experimental configuration: Tables 7–9 document the hyperparameter-search ranges and final experimental values for IMPALA, PPO, and DQN.The settings are reported separately for scoring and checkpoint rewards.
- Football Academy figures: Figure 9 plots IMPALA’s Football Academy average goal difference with CHECKPOINT reward.The passage identifies the plotted metric and algorithm but does not state a numerical outcome.
- Football Academy figures: Figure 10 plots PPO’s Football Academy average goal difference with CHECKPOINT reward using v1.x results because the v2.x experiment did not finish.The figure note directs readers to arXiv for the full v2.x results.
- Football Academy: Football Academy scenarios generally end after 400 frames or when possession is lost, a team scores, or play stops, at medium difficulty level 0.6.The stopping conditions include the ball leaving the pitch and awarded free kicks.