Source-linked AI summary
Game Plan: What AI can do for Football, and What Football can do for AI
Karl Tuyls, Shayegan Omidshafiei, Paul Muller, Zhe Wang, Jerome Connor, Daniel Hennes, Ian Graham, William Spearman, Tim Waskett, Dafydd Steele, Pauline Luc, Adria Recasens, Alexandre Galashov, Gregory Thornton, Romuald Elie, Pablo Sprechmann, Pol Moreno, Kris Cao, Marta Garnelo, Praneet Dutta, Michal Valko, Nicolas Heess, Alex Bridgland, Julien Perolat, Bart De Vylder, Ali Eslami, Mark Rowland, Andrew Jaegle, Remi Munos, Trevor Back, Razia Ahamed, Simon Bouton, Nathalie Beauguerlange, Jackson Broshear, Thore Graepel, Demis Hassabis
TL;DR
Football analytics poses difficult prediction and strategic-analysis problems, while its data and methods offer opportunities for AI research. This paper surveys intersections of statistical learning, game theory, and computer vision and illustrates them with football analyses. It envisions reciprocal benefits for football and AI, while noting that live-play game-theoretic analysis remains challenging.
Problem
Football’s few scoring events and complex game states make it difficult to assign credit to players and assess actions, while live-play game-theoretic analysis remains challenging.
Source: §2.1 · p. 5, §2.2 · p. 6Method
The paper surveys football analytics across statistical learning, game theory, and computer vision, and illustrates their combination with penalty-kick and counterfactual-prediction examples.
Source: §6 · p. 28Results
The paper presents football analytics as a domain with reciprocal potential benefits for AI research and football analytics.
Source: §6 · p. 30Takeaways & Limitations
The envisioned Automated Video Assistant Coach would process broadcast video and advise coaching staff before, during, and after matches.
Source: §6 · p. 28Live-play game-theoretic analysis is constrained by the large player count, exponential strategy space, variable trajectories, and long sequences.
Source: §2.2 · p. 6
Abstract
from arXiv · showhide
The rapid progress in artificial intelligence (AI) and machine learning has opened unprecedented analytics possibilities in various team and individual sports, including baseball, basketball, and tennis. More recently, AI techniques have been applied to football, due to a huge increase in data collection by professional teams, increased computational power, and advances in machine learning, with the goal of better addressing new scientific challenges involved in the analysis of both individual players' and coordinated teams' behaviors. The research challenges associated with predictive and prescriptive football analytics require new developments and progress at the intersection of statistical learning, game theory, and computer vision. In this paper, we provide an overarching perspective highlighting how the combination of these fields, in particular, forms a unique microcosm for AI research, while offering mutual benefits for professional teams, spectators, and broadcasters in the years to come. We illustrate that this duality makes football analytics a game changer of tremendous value, in terms of not only changing the game of football itself, but also in terms of what this domain can mean for the field of AI. We review the state-of-the-art and exemplify the types of analysis enabled by combining the aforementioned fields, including illustrative examples of counterfactual analysis using predictive models, and the combination of game-theoretic analysis of penalty kicks with statistical learning of player attributes. We conclude by highlighting envisioned downstream impacts, including possibilities for extensions to other sports (real and virtual).
1. Introduction
Football analytics has adopted data-driven methods later than other sports, but growing data and AI capabilities are opening new opportunities. The paper maps research frontiers combining statistical learning, computer vision, and game theory, and illustrates their potential through football analyses and an envisioned assistant coach.
1. Introduction
- Football analytics remains comparatively young, following late data collection, difficult playing conditions, and a longstanding reliance on human expertise.Recent availability of high-resolution video, event streams, and tracking data has coincided with advances in deep learning.Source: §1 · p. 3
- The paper identifies the underexplored intersections of statistical learning, computer vision, and game theory as promising frontiers for football analytics.It distinguishes GT&SL, SL&CV, and GT&CV, and presents an integrated approach spanning all three as the ultimate frontier.Source: Figure 1 · p. 2, §1 · p. 4
- An envisioned Automated Video Assistant Coach would integrate the frontiers to support understanding and improvement of play and benefit players, coaches, and spectators.The system is presented as a future human-centric AI application, not as an evaluated tool.Source: §1 · p. 4
- The paper illustrates the GT&SL frontier by combining game-theoretic penalty-kick analysis with statistical learning, alongside counterfactual trajectory prediction.It revisits penalty-kick analysis building on Palacios-Huerta (2003, 2016), and describes a what-if analysis of predicted trajectories.Source: §1 · p. 4
2. Literature Overview: AI for Football Analytics
The literature overview surveys statistical learning, game theory, and computer vision in football, emphasizing their current uses, constraints, and opportunities for integration.
2.1 Statistical Learning
- Football statistical learning seeks player and team representations and models for styles, impact, and temporal or counterfactual predictions.Sparse goals and hand-crafted, largely on-ball state features complicate credit assignment; temporal prediction remains an open challenge likely requiring novel methods and diverse data.Source: §2.1 · p. 5
2.2 Game Theory
- Football game theory commonly analyzes set pieces such as penalties, while live play poses larger strategic and action-mapping challenges.Live play involves 22 active players, an exponentially growing strategy space, variable trajectories, and longer sequences that may call for extensive-form analysis.Source: §2.2 · p. 6
2. Literature Overview: AI for Football Analytics
- The overview frames combining new data types with statistical learning and computer vision as a way to advance descriptive and prescriptive game-theoretic analysis.The paper presents these as opportunities to address existing challenges, not as already-established solutions.Source: §2 · p. 4, §2.2 · p. 6
2.3 Computer Vision
- Computer vision can turn rich, inexpensive football video into spatially detailed representations for downstream analytics.Tracking, pose estimation, and event detection can inform player-skill learning, prediction, ranking, and strategic analysis; broadcast footage also brings occlusion and camera-view challenges.Source: §2.3 · p. 7
3. Game Plan: Long-term Research Vision
The section maps a long-term football analytics vision across the intersections of statistical learning, game theory, and computer vision. It presents these intersections as opportunities for both football and AI, from strategic decision-making and video modeling to football as a more controlled, multimodal AI testbed.
3.1 Frontier 1: Interactive Decision-making (GT & SL)
- Football analytics has yet to fully exploit game theory or its combination with statistical learning, including for penalty-kick analysis enriched with player-specific statistics.Counterfactual player-behavior predictions could also support opponent-specific tactical optimization, iterated best responses, and multi-agent reinforcement learning.Source: §3.1 · p. 8
3.2 Frontier 2: Predictive Modeling from Videos (SL & CV)
- Vision systems could provide more numerous and accurately timed event labels, while video captures subtle clues that event streams or tracking data may miss.Richer video may improve predictions of player movement and action value by capturing more of the partially observable game state; dynamics models may help resolve tracking ambiguities caused by occlusion or players leaving the camera view.Source: §3.2 · p. 9
3.3 Frontier 3: Generative Game-Theoretic Video Analysis Models (GT & CV)
- Computer vision can supply game-theoretic models with spatial representations, while game-theoretic context can constrain video generation through empirical payoff tables and player decisions.The proposed generation pipeline can use intermediate features such as pose or trajectory, and generated outputs could support prescriptive analysis or inform future footage acquisition.Source: Figure 2 · p. 9, §3.3 · pp. 10–11
3.4 Football as an AI Testbed: Microcosm
- Football offers abundant, correlated video, audio, and text, alongside historical statistics, annotated tracking and event data, and crowd-sourced predictions and odds.The authors describe football as a relatively well-controlled setting for evaluating multimodal models on vision, audio, and text with shared game context.Source: §3.4 · pp. 11–12, Figure 4 · p. 12
- An Automated Video Assistant Coach is proposed to reduce reliance on costly hand labeling and assist players, coaches, broadcasters, and spectators.Envisioned uses include analyzing player weaknesses, tailoring tactics to opponents, evaluating rosters or scouting counterfactuals, and surfacing or generating relevant highlights.Source: §3.4 · pp. 12–13
4. Football for AI Research
Football offers AI research a hierarchy of challenges, from learning representations across its data modalities to predictive, prescriptive, and human-factor analysis. The paper identifies representation learning, strategic prediction and decision-making, and human factors as key research layers.
4. Football for AI Research
- Football research is organized from representations of synchronized video, event streams, and tracking data toward prescriptive and predictive analysis and human-factor modeling.Source: §4 · p. 13, Figure 5 · p. 14
4.1 Representation Learning
- Football representations must capture multimodal long-term playing styles and contextual player and team dynamics, which historical results and player aggregates may miss.The paper targets representations of both long-term team styles and in-game dynamics.Source: §4.1 · pp. 13–15
4.2 Predictive Modeling and Decision-Making
- Game-theoretic models could add counterfactual player responses and learned meta-strategies to trajectory prediction, supporting forecasts of team tactics and applications such as scouting and highlight selection.The authors note that learning meta-strategies and predictive models is challenging because the strategy space grows exponentially with the number of on-pitch players.Source: §4.2 · pp. 15–16
4.3 Human Factors
- Human-focused football analysis includes injury and fatigue, player relationships, psychological factors, and cooperation; injury models remain limited in the outcomes they represent and in data availability.Current injury-prediction methods do not explicitly capture uncertainty, injury type or severity, or projected recovery time; injuries are relatively infrequent because clubs take preventive measures.Source: §4.3 · p. 16
5. Illustrative Examples: Frontier 1 (GT&SL)
This section illustrates how game theory and statistical learning can generate football insights through penalty-kick strategic analysis, including scoring-location patterns across kicker clusters and player-style-informed recommendations. It also presents generative trajectory models for counterfactual analysis, predicting defender responses to changed ball trajectories.
5.1 Game Theory: Elementary Concepts
- Empirical game theory models penalty kicks and set pieces from interaction data, supporting prediction and prescription, while normal-form games, bimatrix payoffs, and Nash equilibria organize strategic analysis.The framework can also extend to dynamic trajectory analysis and higher-level set-piece strategies through empirical game-theoretic analysis of meta-games.Source: §5.1 · p. 17
5.2 Game Theory for Penalty Kick Analysis
- Using 12,399 penalty kicks, the reproduction’s results agree with Palacios-Huerta’s: Jensen-Shannon divergence is 0.84% for Nash distributions and 1.2% for empirical frequencies; observed play is ϵ-Nash with ϵ=0.4%.Source: §5.2 · p. 20
- For Natural/Non-Natural actions, left- and right-footed players’ games cannot be shown to differ, and the apparent difference in the natural-shot/natural-goalkeeper cell diminishes with experience.The authors suggest lower experience may reduce kick control and goalkeeper proficiency, while experience removes the observed footedness differences.Source: Table 3 · p. 21, Table 4 · p. 21, §5.2 · p. 21
- In the Left/Center/Right game, outcomes differ across footedness when the goalkeeper moves toward the ball, and empirical-to-Nash Jensen-Shannon divergence is 0.75% versus 0.087% for Natural/Non-Natural actions.The authors conclude this action scheme is not consistent across footedness or with the game players follow.Source: §5.2 · pp. 21–22, Table 5 · p. 22
5.3 Augmenting Game-theoretic Analysis of Penalty Kicks with Embeddings
- To address sparse player-specific data, the authors cluster 635 experienced players’ 18-dimensional Player Vectors into six groups; four cluster pairs differ significantly at p<0.10.The analysis excludes the goalkeeper cluster and a one-player outlier from game-theoretic comparisons because their shot samples are insufficient.Source: §5.3 · pp. 23–25, Table 7 · p. 24
- Clustered kickers show statistically similar empirical behavior but different Nash recommendations; Cluster 1’s Nash divergence from Cluster 4 is 0.57%, versus 0.09% from Cluster 5 and 0.03% from Cluster 2.The authors caution that low-sample payoff tables may make some Nash distributions inaccurate; the analysis is restricted to Premier League players from 2016–2019.Source: Table 8 · p. 24, Table 7 · p. 24, Table 9 · p. 26
- Successful-goal locations differ across kicker clusters: Cluster 2 scores mostly bottom-left, while other clusters are more balanced, potentially reflecting smaller samples.Source: §5.3 · p. 27
5.4 Generative Trajectory Prediction Models for Counterfactual Analysis
- Ghosting prescribes trajectories players should have taken rather than actual ones, with generative prediction extending from set-piece setups to short plays and long-term team strategies.Team-level predictions would benefit from game-theoretic and multi-agent considerations and are perceived to play a key role in an established AVAC system.Source: §5.4 · p. 27
- A trajectory model conditioned on observed ball, defender, and attacker states predicts future movement; perturbing the ball illustrates defender reactions, including a goalkeeper chasing it into the penalty area.Source: §5.4 · p. 28
6. Discussion
Football analytics is both a challenging real-world setting for AI and a potential source of advances across statistical learning, computer vision, and game theory. The discussion connects these fields to an envisioned assistant coach, hierarchical reinforcement learning, and possible applications beyond football.
6. Discussion
- Football offers a challenging but useful AI setting, combining human factors and infrequent scoring with increasingly available multimodal sensory data.The authors identify the intersection of statistical learning, computer vision, and game theory as especially relevant to developing and evaluating AI algorithms.Source: §6 · p. 28
- An Automated Video Assistant Coach (AVAC) is envisioned to process broadcast footage and advise coaches before, during, and after matches.The paper illustrates combinations of game theory and statistical learning with a penalty-kick case study using more than 15,000 kicks and Player Vectors to discern kicking styles.Source: §6 · p. 28
- Football decisions can be framed as reinforcement-learning problems at three linked timescales: scoring goals, winning matches, and winning championships.The levels share context top-down and bottom-up; the discussion connects player credit assignment and multi-agent interactions to reinforcement learning and game theory.Source: §6 · p. 29, Figure 14 · p. 29
- As football analytics shifts from understanding play toward optimizing player and team decisions at broader levels, the authors foresee reinforcement learning becoming increasingly important.The discussion also draws a connection between real-world football and RoboCup, where reinforcement learning can optimize policies for players and robots.Source: §6 · pp. 29–30
- Football analytics methods could extend to esports such as Dota 2 and StarCraft, where team outcomes involve cooperative and competitive behavior.The paper suggests automatic game analysis could identify team strengths, weaknesses, tactics, and directions for improvement.Source: §6 · p. 30
- The authors foresee mutual benefits: football analytics may contribute to AI research, while AI methods may affect football and related domains.They describe this as a feedback loop between the football microcosm and the foundational fields highlighted in the paper.Source: §6 · p. 30
A. Additional Works Related to Statistical Learning in Football
This section surveys statistical-learning work on valuing actions and predicting trajectories in football. These methods estimate contributions in context and support ghosting and counterfactual analysis.
A. Additional Works Related to Statistical Learning in Football
- Valuing Actions by Estimating Probabilities (VAEP) values on-ball actions through their effect on game state and the probabilities that a team will score.Such scores can be used to assess player contributions or the mutual chemistry of player pairs.Source: A. Additional Works Related to… · pp. 30–31
- VAEP addresses action valuation by estimating a game state that can include match events, player traits, time, and player and ball positions.The passage also notes that a tournament-wide state may include prior and upcoming matches and players’ accumulated yellow cards.Source: A. Additional Works Related to… · p. 30
- Trajectory-prediction models can support ghosting by predicting what another team or player would have done in a given play.Accurate predictions can also support counterfactual reasoning about alternative scenarios that did not occur; cited work has predicted ball and player trajectories and identified roles from predicted trajectories.Source: A. Additional Works Related to… · p. 31
B. Pose Estimation
The section distinguishes bottom-up and top-down approaches to multi-person pose estimation and describes their accuracy and computation trade-offs. It also reports the two-stage G-RMI approach used for pose predictions.
B. Pose Estimation
- Bottom-up pose estimation detects joints before grouping them into people, whereas top-down methods detect people first and estimate each person’s pose.The section reports that top-down computation grows linearly with the number of people, while bottom-up computation stays constant; top-down methods are often more accurate when people overlap significantly.Source: B. Pose Estimation · p. 31
- The experiments use G-RMI, a top-down approach that first detects person instances and then estimates keypoints within each resulting crop.The first stage uses Faster-RNN for instance detection.Source: B. Pose Estimation · p. 31
- G-RMI combines dense keypoint classification and offset regression, then aggregates predictions with Hough voting and applies confidence scoring and non-maximum suppression.The authors plan to build on this approach for the previously described challenges.Source: B. Pose Estimation · p. 31
C. Player Vectors
The section defines playing style through players’ preferred field areas and the actions they perform there. It constructs 18-dimensional Player Vectors from spatial action counts, compressed by non-negative matrix factorization.
C. Player Vectors
- Playing style is defined as a player’s preferred field areas and the actions they tend to perform in each location.The section follows the definition in Decroos and Davis (2019).Source: C. Player Vectors · p. 31
- Player Vectors are built from Liverpool Premier League event data from 2017 to 2019, counting passes, dribbles, shots, and crosses across a 60 × 40 pitch grid.For each player and action type, the grid counts summarize spatial preferences.Source: C. Player Vectors · p. 32
- Non-negative matrix factorization compresses the spatial action-count matrices, and the four action-type vectors are concatenated into an 18-dimensional vector per player.Each compressed dimension quantifies a player’s preference for performing an action type in an area.Source: C. Player Vectors · p. 32