PaperlayerView exact revision on arXiv

Source-linked AI summary

From Motor Control to Team Play in Simulated Humanoid Football

Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojciech M. Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasenclever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan D. Tracey, Karl Tuyls, Thore Graepel, Nicolas Heess

arXiv:2105.12196v1cs.AIcs.MAcs.NEcs.RO

TL;DR

The paper asks how learning systems can integrate low-level movement, longer-term decisions, and multi-agent coordination in embodied settings. It trains simulated humanoid football teams with staged imitation and reinforcement learning, finding coordinated 2v2 play with naturalistic movement, football skills, and team-level strategies. The study demonstrates integrated multi-scale control, while remaining substantially simpler than real football and not addressing direct sim-to-real transfer.

  • Problem

    Existing research largely studies motor control, goal-directed behaviour, and cooperation separately, while principles for integrating multiple control levels remain poorly understood.

  • Method

    A three-stage framework combines imitation learning for movement, reinforcement-learning drills for reusable football skills, and multi-agent reinforcement learning for team gameplay.

  • Results

    The agents learn coordinated 2v2 football with naturalistic movement, football skills, team-level tactics, and learned representations of relevant game features and future behaviour.

  • Takeaways & Limitations

    End-to-end learning can combine skill transfer and self-play to produce complex movement coordination, long-horizon goals, and cooperation among embodied agents.

  • Takeaways & Limitations

    The study does not tackle full-scale football, and the developed method is not suitable for direct learning on robotics hardware or direct sim-to-real transfer.

Abstract

from arXiv · show

Intelligent behaviour in the physical world exhibits structure at multiple spatial and temporal scales. Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals defined on much longer timescales, and in terms of relations that extend far beyond the body itself, ultimately involving coordination with other agents. Recent research in artificial intelligence has shown the promise of learning-based approaches to the respective problems of complex movement, longer-term planning and multi-agent coordination. However, there is limited research aimed at their integration. We study this problem by training teams of physically simulated humanoid avatars to play football in a realistic virtual environment. We develop a method that combines imitation learning, single- and multi-agent reinforcement learning and population-based training, and makes use of transferable representations of behaviour for decision making at different levels of abstraction. In a sequence of stages, players first learn to control a fully articulated body to perform realistic, human-like movements such as running and turning; they then acquire mid-level football skills such as dribbling and shooting; finally, they develop awareness of others and play as a team, bridging the gap between low-level motor control at a timescale of milliseconds, and coordinated goal-directed behaviour as a team at the timescale of tens of seconds. We investigate the emergence of behaviours at different levels of abstraction, as well as the representations that underlie these behaviours using several analysis techniques, including statistics from real-world sports analytics. Our work constitutes a complete demonstration of integrated decision-making at multiple scales in a physically embodied multi-agent setting. See project video at https://youtu.be/KHMwq9pv7mg.

1. Introduction

The paper addresses how embodied intelligent behaviour can integrate low-level motor control, longer-term goal-directed decisions, and coordination with other agents. It studies this integration by training simulated humanoid football teams through progressively richer learning stages.

  • The integration challenge: Integrated control across motor, cognitive, and social levels remains poorly understood, with few successful embodied-system demonstrations.Unsuitable decompositions into hierarchical modules can significantly impair performance.
  • Football as a testbed: Football combines fast body control, mid-level skills such as kicking and dribbling, and long-horizon team behaviour such as scoring.The setting couples low-level movement with high-level coordination in a dynamic context involving the ball, goals, and other players.
  • Approach: The training framework progresses from imitation learning for movement, through reinforcement-learning drills for football skills, to multi-agent reinforcement learning for full gameplay.It combines prior knowledge from imitation with self-play and an emerging population-based curriculum.
  • Findings: The trained players develop human-like movement, ball handling, cooperative play, and team tactics including moving into space, defensive positioning, and passing.Game performance correlates with movement robustness, coordination, team tactics, and predictions of teammates’ and opponents’ behaviour.
  • Analysis: The study analyses evolving movement and team strategies alongside learned internal representations using AI methods and real-world sports analytics.The paper examines both behavioural performance and the representations underlying multi-scale play.

2. Environment

The environment places fully articulated humanoids in a physics-based 2v2 football task designed to couple realistic motor control with context-dependent coordination. It simplifies some football rules and observations to support continuous end-to-end learning.

  • Physical setting: The environment uses fully articulated 56-degree-of-freedom humanoids in a high-fidelity MuJoCo simulation with realistic body dynamics and joint force limits.Agents must locomote through torques applied at the joints, with foot contact and friction forces.
  • Task scope: The task focuses on movement coordination among small groups of highly articulated players rather than the full football problem.Its high-fidelity physics and high-dimensional bodies support behaviours that would be difficult to handcraft.
  • Simplifications: The environment simplifies football rules by omitting fouls and interruptions and preventing the ball from leaving the pitch, facilitating continuous gameplay and end-to-end learning.The agents also receive state features for partial observation rather than performing state estimation.
  • Observation limits: Egocentric vision is available but unused, making the observation model less realistic and likely discouraging behaviours such as running backwards or hanging back.The limitation concerns the observation model rather than the simulated rigid-body dynamics.
  • Gameplay: Episodes use 2v2 teams, with reward delivered when the ball enters the opponents’ goal before players and ball reset at random locations.Both teammates receive the goal reward, and the environment permits an arbitrary number of players even though this study uses two-player teams.

3. Learning Framework

The framework trains simulated humanoid football agents through transferred low-level motor control, mid-level drill skills, and multi-agent game learning. Population-based training shapes exploration and opponent curricula while drill priors regularize full-game policies.

  • Three-stage framework: Three stages progress from imitation-learned low-level movement, through reinforcement-learned football drills, to multi-agent reinforcement learning for full-game play.The stages transfer a general-purpose motor module and reusable drill priors into coordinated football learning.
  • Task formulation: Football training models four-player games as multi-agent reinforcement learning, while drills use single-agent tasks with shared action and partially consistent observation and dynamics spaces.Players independently select continuous actions, and football rewards are assigned according to terminal match outcomes.
  • Behaviour shaping: Sparse, high-variance football rewards motivate shaping and drill-prior regularization, which biases policies toward reusable mid-level behaviours during exploration.The regularizer penalizes latent-space KL divergence between football policies and a mixture of four pretrained drill priors.
  • Population-based training: Population-based training optimizes reward-shaping and learning hyperparameters while increasing the proportion of high-performing agents to create an automatic curriculum over coplayer strength.Agents are selected, evaluated, and updated through an outer loop around reinforcement-learning optimization.
  • Three-stage framework: Agents act through continuous latent motor intentions that are passed to a fixed low-level controller, transferring movement skills across drills and football.The latent action representation replaces raw per-joint actions during reinforcement learning.

4. Experiments

The experiments evaluate a staged football-training framework across independent player populations, held-out evaluation agents, learned behaviours, and distributed infrastructure. Training produces increasingly capable movement, ball-handling, and cooperative team strategies, while multi-agent evaluation remains intrinsically difficult because performance depends on the other agents.

  • Experimental setup: Three independent experiments trained populations of 16 football players with distinct weight and hyper-parameter initializations, using the same NPMP across experiments.The setup was designed to assess the reliability of the full framework.
  • Training procedure: Players were trained for 8 × 10^10 environment steps, corresponding to six weeks of wall-clock time.After about two weeks, the best agent decisively beat all evaluation agents, although performance continued improving without saturating.
  • Learned behaviours: Training produced context-dependent movement and ball-handling skills, including recovery, fast locomotion, rapid turning, dribbling around opponents, and accurate shooting.Locomotion also became robust to external pushes, and players engaged in close-quarter duels.
  • Learned behaviours: Team behaviour progressed from individualistic ball-chasing to coordinated strategies involving division of labour, passing, and near-term tactics directed toward long-term team goals.Several of these behavioural motifs repeated reliably across games.
  • Multi-agent evaluation: Performance was measured through 64 matches against 13 held-out evaluation agents at regular training intervals, with Elo scores reported for the population’s top three players.The evaluation agents differed in skill levels and behavioural traits and were not used as training opponents.
  • Training infrastructure: The distributed system used a central orchestrator, 4,096 CPU actor workers, 128 inference servers with batched requests, and learners that continuously updated population networks from replay data.The infrastructure separates simulation, inference, matchmaking, and learning to support many heterogeneous models and variable-throughput inference.

5. How Football Agents Play

The analysis tracks movement, football skills, teamwork, and policy sensitivity throughout training. Players first develop locomotion and ball handling, then shift from individual ball chasing toward coordinated passing and positioning.

  • Analysis methods: The analysis measures locomotion, football skills, and teamwork during games against evaluation agents, supplemented by counterfactual policy divergence and a controlled probe task.The probe task tests whether a passer directs the ball toward a teammate and values potential passes.
  • Phase 1: During the first 1.5 × 10^9 environment steps, speed, possession, and recovery from falls improve rapidly, while teammates often crowd around the ball.After six hours, agents recover from 80% of falls; early play remains individualistic as division of labour decreases.
  • Phase 1: Early in training, the ball induces far more policy divergence than teammates or opponents, while probe performance falls below 50% and pass-value correlation is negative.After about five hours, ball-induced divergence is 40 times teammate-induced divergence and 700 times opponent-induced divergence.
  • Phase 2: After 8 × 10^10 environment steps, division of labour exceeds 0.85 as one player seeks possession while the teammate moves upfield to anticipate a pass.Passing also increases: 6% of touches are passes and approximately 40% of passes travel more than 10m.
  • Phase 2: In the probe task, the passer eventually kicks toward the receiver’s wing 60% of the time, indicating sensitivity to teammate possession value.Probe performance improves as agents learn coordinated strategies.

6. How Football Agents Work

The agents organize game-state information and learned football skills into internal representations that support context-dependent decisions. Analyses link these representations to behavior, future-state prediction, and adaptive use of drill priors.

  • Internal game-state representations: The agents’ LSTM states cluster by conjunctions of high-level game features, including distinct shooting and dribbling states.Comparable clusters are not present in raw observations, suggesting the internal representation organizes game situations more distinctly.
  • Internal game-state representations: Most analyzed game features are emphasized in the internal representation, including which agent is closest to the ball, teammate proximity, and shooting.Some features are preserved from raw observations, while teammate-fallen information is de-emphasized.
  • Internal game-state representations: Several individual recurrent units discriminate game situations, allowing some features to be decoded by simple activation thresholds.This indicates that information about some game features is localized within the internal state.
  • Prediction and performance: The agents’ ability to predict future game features is correlated with good performance in both intra-population and inter-population analyses.The analysis uses internal states from agents selected at 40 × 10^9 environment steps and evaluates different prediction horizons.
  • Use of drill priors: Football policies adaptively align with different drill priors during play, including diverging from follow and aligning with shoot around kicks.KL-divergence analysis compares the trained policy with follow, dribble, shoot, and kick-to-target experts; scoring-channel contribution increases before successful shots.

7. Ablation Study

The ablations show that the standard combination of drill priors and shaping rewards produces the strongest and most consistent players. Removing priors causes degenerate movement, while additional coordination rewards can reduce performance.

  • Reward ablation: Players trained with drill priors but sparse rewards learned to score, yet often moved slowly and failed to recover after falling.The authors conclude that sparse reward learning is possible with mid-level priors, while movement skills still benefit from shaping rewards.
  • Drill-prior ablation: Training without drill priors produced poor, high-variance performance and degenerate movement, including rolling instead of running or hitting the ball with the hands.These results indicate that the low-level skill module alone did not adequately shape player movement.
  • Overall comparison: The standard training scheme produced the best-performing players, with similarly high performance across all three independent experiments.The standard scheme combines low-level skills, mid-level drill priors, and simple dense shaping rewards.
  • Additional coordination rewards: Additional shaping rewards for coordinated behavior performed well but worse overall than the standard scheme.Movement and ball-handling were similar, but average ball velocity toward the goal was approximately 30% lower.
  • Interpretation: Designing reliable coordination rewards is difficult because conflicting incentives can reduce performance and enlarge the population-based-training hyperparameter search.The authors suggest cooperative behavior priors as a possible alternative and note the population size was 16.

8. Related Work

The paper combines multi-scale control, humanoid motor learning, and independent multi-agent reinforcement learning in a physically simulated football setting. Its key distinction is transferring learned skills as priors rather than as a fixed architectural hierarchy.

  • Scope: The work addresses multi-scale behavior, physical humanoid control, teamwork, and robustness to adversarial opponents within one challenge.These components are each described as fundamental open problems in AI research.
  • Emergent coordination: Independent-agent coordination remains difficult because joint exploration and optimization are non-stationary and non-Markovian for each learner, especially with high-dimensional control and sparse distal rewards.The cited literature identifies this as an unresolved problem.
  • Emergent coordination: Prior emergent coordination results covered discrete-control games or simpler physical embodiments, whereas this work studies coordinated behavior in articulated humanoids with continuous control.The agents communicate only through physical actions and observations rather than explicit communication channels.
  • Competition and self-play: Self-play and competition can produce complex strategies in continuous-control environments, motivating physically grounded multi-agent learning with richer interactions.The paper extends this direction by combining complex embodiments, high-dimensional control, and realistic physics.
  • Multi-scale control: The agent combines a low-level motor-control skill module with a non-hierarchical mid-level behavior prior and end-to-end training.Its behaviors originate from demonstrations, pre-training tasks, and end-to-end learning across locomotion, football skills, and team coordination.
  • Relation to hierarchical learning: Unlike Layered Learning, the system transfers pre-learned skills as priors over motor intentions rather than as parametrized components in a fixed hierarchy.This leaves individual skills and multi-agent behaviors more tightly coupled.

9. Discussion

The study demonstrates end-to-end 2v2 football gameplay in simulated humanoids, integrating natural movement, football skills, and team coordination across timescales. The approach combines staged learning and transferable behaviours, while remaining limited to simplified simulated settings.

  • Discussion: Players learned natural human-like movements and coordinated as a team during end-to-end 2v2 football gameplay.The setting couples movement skills with longer-horizon goal-directed behaviour.
  • Discussion: Learned behaviour improved on a human football coordination metric and used representations emphasizing relevant high-level game features.Players also learned future predictions similar to observations reported in human soccer players.
  • Discussion: The three-stage framework combines low-level motor imitation, football training drills, and multi-agent self-play for the full task.Increasingly complex skills are acquired gradually and reused to address exploration and credit-assignment challenges.
  • Discussion: The study addresses naturalistic humanoid behaviour, multi-scale hierarchical behaviour, and coordination in challenging multi-agent scenarios.The authors conclude that agents can coordinate complex movements to interact with objects and achieve long-horizon goals cooperatively.
  • Limitations: The study focuses on competent 2v2 gameplay with simplified rules rather than full-scale football.The authors identify larger teams, penalties, throw-ins, and a dedicated goalkeeper role as directions for future work.
  • Limitations: The experiments use realistic-physics simulation but do not target direct sim-to-real transfer or learning directly on robotic hardware.The authors cite data-efficiency and safety considerations among the barriers to hardware learning.

B.1. Task Abstraction Using Stochastic Games

The task framework models each learning problem as a stochastic game with shared action structure and task-specific observations. Policy learning uses recurrent, multi-channel value functions and constrained policy updates from replay data.

  • Task formulation: Each task is represented as a stochastic game, extending Markov decision processes to multiple decision makers.The formulation includes task-specific state, observation, action, transition, and initial-state components.
  • Task formulation: Humanoids use a shared 56-dimensional continuous action space across tasks and players, while observations are only partially shared across tasks.Proprioceptive observations are present for all players and tasks; other observations are task-specific.
  • Learning: Agents maintain separate action-value functions for multiple reward channels and update them using trajectories from replay buffers.The method applies off-policy correction through retrace target Q-values.
  • Multi-agent learning: Because agents do not observe opponent identities, recurrent networks use interaction history to infer coplayer dependence.The framework uses independent learning and independent execution for each agent.
  • Learning: Policy improvement first solves a constrained update and then projects the result onto a parametric policy under a trust-region constraint.The projection minimizes KL divergence to the non-parametric policy while improving learning stability.

B.3. Motion Capture and NPMP Training Details

Motion-capture football data is converted into humanoid joint trajectories and used to train low-level tracking and distilled motor-prior models. The dataset includes short snippets and some contact-rich behaviours, but architecture choices were not systematically explored here.

  • Data preparation: The motion-capture dataset contained 560 player clips totaling roughly 1 hour and 45 minutes of player-time.The trajectories were cut into 1245 snippets lasting 4–8 seconds.
  • Data preparation: A meaningful subset of 100/1245 snippets contained behaviours such as diving, slide tackles, and recovery after falls.These behaviours were included to support richer low-level motor control.
  • Motion processing: Motion-capture point clouds were converted to joint angles, adapted to subject-specific body proportions, and retargeted to the standard humanoid model.Tracking used a 30 ms/timestep control resolution.
  • NPMP architecture: The distilled NPMP uses a two-hidden-layer 1024-unit encoder with 60 latent variables and a three-hidden-layer 1024-unit decoder.The latent prior is modeled as a simple autoregressive process.
  • Limitations: The network architecture was informed by prior exploration but was not systematically explored in this study.This bounds conclusions about the chosen encoder and decoder sizes.

B.4. Agent Architecture

The agent architecture encodes proprioception, task context, and player relationships before recurrent policies act in a latent motor-intention space. Football relationships are represented as unordered sets with attention, while training combines shaping rewards and population-based evaluation.

  • State representation: Each agent encodes proprioceptive features and task-context features, with football observations additionally including teammates and opponents.Feature encoders are implemented as two-layer MLPs.
  • Relationship representation: Teammates and opponents are treated as unordered sets, making the number of learned parameters independent of the number of players.Pairwise encodings are processed by a multi-headed attention module.
  • Relationship representation: The attention mechanism uses queries based on proprioceptive and context encodings and produces six weighted pairwise relationship embeddings.Pairwise peer encodings can include self-peer relationships through sampling with replacement.
  • State representation: The final football state representation concatenates proprioceptive, context, and player-player interaction encodings.The environment-state architecture is summarized in Figure 11.
  • Policy architecture: LSTM modules process interaction history, while policies sample latent motor actions that a fixed low-level controller converts into environment actions.Reinforcement learning therefore operates over latent motor intentions.
  • Training objectives: Training uses task-specific shaping rewards with independently adjustable discount factors and coefficients.The shaping rewards are described in Table 5.
  • Population training: Population-based training evaluates agents through pairwise payoffs and improves agents relative to the population’s average player.The population contains 16 continuously learning agents, with draws counted toward both beta-distribution parameters.

C.1. Behavior Statistics

The study defines behavioral statistics for locomotion, football skills, teamwork, and OBSO, then evaluates them across training snapshots and episodes against evaluation agents.

  • Basic locomotion skills: Locomotion is measured by speed and the proportion of falls from which an agent recovers.An agent’s ability to get up is defined as its recovery proportion after falling.
  • Football skills: Football skill metrics measure team ball control, pass frequency, and pass range.Ball control is the proportion of timesteps when the closest player to the ball is on the team; pass frequency counts touches leading to passes of at least 5m, while pass range measures passes of at least 10m.
  • Teamwork statistics: Division of labour increases as one player approaches the ball while the teammate adopts an alternative behavior instead of also seeking possession.The metric is near 0 when teammates crowd the ball and near 1 when only one teammate tries to possess it.
  • OBSO: OBSO is tracked as an externally defined sports-analytics measure that the agents are not explicitly trained to optimize.The analysis asks whether agents improve a useful external metric without directly encouraging its optimization.
  • Evaluation procedure: Behavioral statistics are averaged over 100 episodes per snapshot, with each episode lasting 3000 environment steps or 90 seconds.Population statistics average the 300 episodes played by the top three agents, and training progression uses 16 snapshots through 80 × 10^9 environment steps.

C.3. Probe Tasks

The probe tasks operationalize passing competence and OBSO using interpretable tests of receiver-directed kicks, value alignment, and scoring opportunities derived from multiple football models.

  • Passing probes: The passing-direction probe tests whether the ball’s lateral velocity points toward the receiver’s position.The score is based on the y component of ball velocity when the passer first moves the ball forward.
  • Passing probes: A probe score of 1 means the passer always kicks toward the receiver, whereas 0 means it never does.Intermediate outcomes reflect the proportion of probe episodes in which the pass direction agrees with the receiver direction.
  • Pass-value correlation: Pass-value correlation tests whether value functions assign higher value when the ball travels toward rather than away from the receiver.The statistic correlates the scoring value estimate with an indicator of receiver-directed ball movement.
  • OBSO: OBSO estimates an off-ball player’s immediate scoring probability by combining pass control, ball transition, and scoring models.The overall probability is computed as the product of these component probabilities at receiving positions on the pitch.
  • OBSO: The OBSO components model pass control using PPCF, ball movement using transitions, and scoring as a function of distance to goal.Parameters are fitted independently for agents and evaluation agents at each training point, using observed player and ball speeds and interaction data.

C.4.2. Additional OBSO Results

Additional analyses examine how OBSO and internal representations develop during training, showing increasingly valuable positioning and structured, efficiently decodable game-state information.

  • OBSO results: High-OBSO passes and average OBSO consistently increase with training time.Because agents are unaware of OBSO during training, the result indicates progressively improved positioning that raises scoring opportunities.
  • Knowledge representation: The agent’s internal representation contains cluster structure reflecting conjunctions of game features such as falling, ball proximity, and home-field position.This structure is found by comparing t-SNE projections of raw observations and recurrent hidden states.
  • Coordination and performance: Coordination statistics are positively correlated with agent performance across training regimes.The analysis compares passing range and passing frequency with agent Elo at 40e9 environment steps.
  • Representational efficiency: Linear and nonlinear decoding perform similarly for several game features, suggesting efficient access to represented information by the policy head.For agent on the ground, agent closest to ball, teammate closest to ball, and shooting, two-sided t-tests report p-values above 0.05.
Loading 2105.12196v1…