Source-linked AI summary
Emergent Coordination Through Competition
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, Thore Graepel
TL;DR
How cooperative behavior emerges and how to assess it remain challenges in competitive multi-agent reinforcement learning. The paper studies these questions in physics-based soccer using population-based training that evolves shaping rewards and Nash-averaging evaluation. It reports coordinated behavior, including repeated passing motifs.
Problem
Assessing cooperation in competitive multi-agent games is difficult, while reward shaping in continuous control can perturb objectives and is challenging in cooperative settings.
Source: §3.3 · p. 4, §5.2 · p. 8Method
The authors train independent teams in competitive soccer with population-based training that optimizes shaping-reward weights and discounts, and evaluate agents using Nash averaging.
Source: §1 · pp. 1–2, §3.3 · p. 4Results
Population-based training evolved agents from myopic shaping rewards toward long-horizon game rewards, with coordinated behavior that included repeated passing motifs.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
Nash averaging offers a way to evaluate competitive agents when handcrafted bots or established human baselines are unavailable.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.
1 INTRODUCTION
The paper introduces a MuJoCo soccer environment to study continuous multi-agent reinforcement learning and shows that decentralized population-based training can evolve cooperative behavior from simple shaping rewards. It also evaluates trained teams with Nash averaging to address evaluator diversity and exploitability.
1 INTRODUCTION
- The authors introduce a MuJoCo soccer environment with simulated physics and simple bodies, using 3-dimensional actions to study coordination, shaping rewards, exploitability, and evaluation.The environment can scale to more agents and more complex bodies.Source: §1 · p. 1
- The framework trains independent off-policy RL learners with recurrent memory and decomposed shaping-reward channels using decentralized population-based training.Reward channels have individual weights and discount factors that are optimized online.Source: §1 · pp. 1–2
- Decentralized population-based training evolves agents from random behavior through ball chasing to more cooperative, strategic play, while shifting rewards toward long-horizon team objectives.The method automatically optimizes shaping-reward weights and discount factors across separate reward channels using competitive match results.Source: §1 · p. 2
- Trained teams show non-transitive pairwise rankings, so the authors evaluate them with Nash averaging against pre-trained agents in its support set.This approach addresses the difficulty of relying on handcrafted bots or established human baselines.Source: §1 · p. 2
2 PRELIMINARIES
The soccer domain is framed as mixed cooperative-competitive multi-agent reinforcement learning, modeled as a Markov game in which agents learn to maximize individual cumulative rewards. Its states and actions are continuous, and agents may use history-dependent policies.
2 PRELIMINARIES
- Soccer is treated as mixed cooperative-competitive multi-agent reinforcement learning, typically modeled as a Markov game, with the mix depending on agents’ reward alignment.Source: §2 · p. 2
- The soccer domain has continuous state and action spaces, with transition distributions treated as densities.Source: §2 · p. 2
- Agents maximize expected cumulative reward, discounted by γ < 1 for infinite horizons, using policies that may depend on observation-action history; the single-agent case is a POMDP.All players share the same action and observation spaces.Source: §2 · p. 2
3 METHODS
The method combines continuous-control SVG0 with population-based training to address exploitability and optimize hyperparameters and shaping rewards online. It also uses separate discount factors to distinguish long-horizon environment rewards from dense shaping rewards.
3.1 POPULATION BASED TRAINING
- PBT trains a population through co-play, with weaker agents inheriting stronger agents’ parameters and mutated hyperparameters, helping optimize hyperparameters online and reduce exploitability.Agents optimize expected return across games against other policies drawn from the population.Source: §3.1 · pp. 2–3
3.2 RETRACE-SVG0
- SVG0 estimates policy gradients across games, while an entropy-regularized objective is optimized with Adam; a recurrent critic conditions value estimates on other players’ observed behavior.The critic uses experience replay, k-step TD errors, and off-policy Retrace corrections; recurrence helps account for unidentified opponents and population diversity.Source: §3.2 · pp. 3–4
3.3 DECOMPOSED DISCOUNTS AND ACTION-VALUE ESTIMATION FOR REWARD SHAPING
- PBT optimizes shaping-reward weights using environment-reward fitness, while separate discount factors let agents pursue sparse environment rewards far ahead and dense shaping rewards myopically.The authors motivate shaping as challenging in sparse-reward continuous control and cooperative settings, where poorly designed or scaled rewards can produce degenerate behavior.Source: §3.3 · p. 4
4 EXPERIMENTAL SETUP
The section specifies a 2v2 soccer environment and population-based training, then bootstraps evaluation with Nash-averaged teams to address pitfalls of Elo-style rankings.
4.1 MUJOCO SOCCER ENVIRONMENT
- The MuJoCo 2v2 soccer environment gives each player three-dimensional control and 93-dimensional observations; matches last up to 45 seconds or until a team scores.Training fields ranged from 20m × 15m to 28m × 21m, while testing used a fixed 24m × 18m field; players acted every 0.05 seconds and player-player contact was disabled.Source: §4.1 · pp. 4–5
4.2 PBT SETTINGS
- Population-based training used 32 agents, evolving agents below a 0.47 expected win rate and uniformly selecting four agents per match for diverse pairings.A grace period and burn-in followed evolution to limit its frequency and maintain population diversity.Source: §4.2 · p. 5
4.3 EVALUATION
- Nash-averaging ranks agents through pairwise win-rate meta-games and is invariant to redundant strategies, addressing misleading Elo or survival rankings in non-transitive play.Evaluation was bootstrapped from 10 teams trained for 25B frames each and 1M tournament matches; variants were scored by support-weighted expected score difference against three teams.Source: §4.3 · pp. 5–6
- Nash support exposed non-transitivity: A beat/drew B in 59.7%, B beat/drew C in 71.1%, and C beat/drew A in 65.3%; Elo instead ranked B first, A fifth.Elo scores were 1084.27 for B, 1068.85 for C, and 1016.48 for A; Elo alone would imply B should beat A at 62%.Source: §4.3 · pp. 5–6, Figure 2 · p. 6
5 RESULTS
The results trace a progression from random behavior through ball chasing to coordinated play, while showing how population evolution, reward shaping, and other algorithmic components affect learning. Behavior statistics, policy analyses, and probes provide evidence of emerging cooperation.
5.1 ABLATION STUDY
- Evolution begins improving population performance at 2B steps, while critic learning rate and entropy cost decrease and discount factor increases over training.The increasing discount factor shifts focus toward long-term return as exploration gives way to exploitation.Source: §5.1 · p. 6, Figure 3 · p. 7, Figure 4 · p. 7
- Dense shaping rewards ease early learning, but evolution reduces vel-to-ball’s relative importance and shifts from scoring focus toward defending later in training.The conceding penalty initially evolves toward zero relative to scoring, then agents later pay more attention to conceding goals.Source: §5.1 · p. 7, Figure 3 · p. 7, Figure 5 · p. 8
- The feedforward policy with recurrent action-value function reached weighted expected goal difference 0 at 22B vs 35B steps for the recurrent policy with recurrent action-value function.A recurrent policy may underperform its feedforward counterpart with a recurrent action-value function because it overfits to its population’s behaviors.Source: §5.1 · p. 7, Figure 3 · p. 7
- Reward-specific discount factors extend horizons for sparse rewards and vel-ball-to-goal but keep vel-to-ball short-term; the decomposed agent reached 0.2 at 15B vs 35B steps and attained highest tournament Elo.Despite asymptotically underperforming lstm + evo + rwd shp, it learned faster; this suggests the training population is less diverse than the Nash-averaging evaluation set.Source: §5.1 · pp. 7–8, Figure 6 · p. 8
5.2 EMERGENT MULTI-AGENT BEHAVIORS
- Agents develop recurring cross-passing motifs, and in probes later-training players pass in response to teammate position rather than dribbling regardless of it.One probe agent performed two consecutive passes, from blue0 to blue1 and back.Source: §5.2 · p. 8, §5.2.3 · p. 10
5.2.1 BEHAVIOR STATISTICS
- Long-range passes become increasingly common, reaching the frequency of long-range interceptions, while teammates spread out after initially converging on the ball.Passes over 10m increase, whereas interceptions over 10m remain flat; both ordinary pass and interception rates remain flat during training.Source: §5.2.1 · pp. 8–9, Figure 7 · p. 9
5.2.2 COUNTERFACTUAL POLICY DIVERGENCE
- Policy divergence first responds strongly to ball position, then increasingly to teammate position from 5B steps, providing evidence of emergent cooperation.Teammate position has more impact than any single opponent position, before opponents become increasingly relevant to game dynamics.Source: §5.2.2 · p. 9, Figure 8 · p. 9
5.2.3 MULTI-AGENT BEHAVIOR PROBING
- In probes, later-training blue0 actively seeks passes conditioned on blue1’s configuration, unlike the individualistic agent that dribbles regardless of teammate position.In the “8e10 left” trace, blue0 and blue1 complete two consecutive passes.Source: §5.2.3 · pp. 9–10, Figure 9 · p. 10
6 RELATED WORK
The work adapts population-based training from capture-the-flag to continuous simulated physics, emphasizing open-ended multi-agent interactions. It contrasts independent asynchronous learning with prior centralized coordination approaches and positions the environment as a research platform for studying cooperation alongside competition.
6 RELATED WORK
- Population-based training was introduced for capture-the-flag by Jaderberg et al. (2018); PSRO and NFSP instead play previous best responses, while this approach trains against simultaneous learners.The paper describes simultaneous learning as more data efficient and parallelizable than playing previous best-response policies.Source: §6 · p. 10
- Prior deepRL coordination work requires some centralization, whereas this framework uses fully independent asynchronous learning and demonstrates evidence of complex coordinated behaviors.Examples of centralized approaches include Lowe et al. (2017), Foerster et al. (2018; 2016), Sukhbaatar et al. (2016), and Mordatch & Abbeel (2018).Source: §6 · p. 10
- Unlike RoboCup teams using reinforcement learning elements but not end-to-end RL, this environment is intended as an extendable research platform for complex multi-agent interactions.Proposed extensions include complex bodies, more agents, and multi-task, transfer, and continual learning; passing in soccer is an example of physically observable and interpretable interaction.Source: §6 · p. 10
- Bansal et al. (2017) and Al-Shedivat et al. (2017) observed emergent complexity from competition in 1v1 MuJoCo Sumo, while this work studies cooperation within teams as well as competition.Other efforts to optimize rewards for multi-agent teams include Liu et al. (2012).Source: §6 · p. 10
7 CONCLUSIONS AND FUTURE WORK · A OFF-POLICY SVG0 ALGORITHM · A.1 POLICY UPDATES
The paper introduces a continuous-physics 2v2 soccer environment and shows that distributed population-based training with optimized shaping rewards can produce coordinated behavior, including repeated passing. It also presents methods for analyzing and evaluating agents, while identifying robustness and scalability as future research directions.
7 CONCLUSIONS AND FUTURE WORK
- Distributed population-based training with automatically optimized shaping rewards learned coordinated behavior, including repeated passing, in a new continuous-physics 2v2 soccer environment.The framework trains teams of independent reinforcement-learning agents end to end.Source: §7 · p. 10
- Automatically optimizing separate discount factors for shaping rewards is intended to shift agents from myopic shaping-reward optimization toward sparse long-horizon team rewards and cooperative behavior.Source: §7 · p. 10
- The paper introduces counterfactual policy divergence for behavior analysis and finds non-transitive pairwise match results, motivating robustness as future work.Source: §7 · p. 10
- The soccer environment provides a platform for continuous-physics multi-agent research and can scale to more agents and more complex bodies, which the paper leaves for future work.Source: §7 · p. 10
A.1 POLICY UPDATES
- SVG0 is a special case of the policy-gradient family in Heess et al. (2015b), is closely related to DPG, and has DPG as a special case.The appendix provides a specific SVG0 derivation.Source: §A.1 · p. 14
- Reparameterization expresses a stochastic policy as a deterministic policy conditioned on state and random variable η, with actions sampled using η ∼ ρ.Source: §A.1 · p. 14
- In the policy-gradient weighting ζ, γ is typically replaced by 1 to avoid excessively discounting future-state terms; the text points to Algorithm 3 in Heess et al. (2015b).For recurrent policies, it refers readers to Heess et al. (2015a).Source: §A.1 · p. 14
A OFF-POLICY SVG0 ALGORITHM
- Algorithm 2 describes off-policy SVG0 updates using a replay buffer, current-policy action sampling, and off-policy critic training before a policy update.The supplied algorithm passage shows the policy update beginning but does not provide its complete expression.Source: §A.1 · p. 14
A.2 Q-VALUE UPDATES
The Q-value critic conditions on an agent’s observation history to handle hidden game state and other players’ behavior. It is trained from replay with off-policy retrace correction, target-network bootstrapping, and recurrent updates.
A.2 Q-VALUE UPDATES
- The critic treats other players as environment dynamics and estimates a player-independent action value, conditioning on observation history because game state and other-agent identities are hidden.An LSTM provides a low-dimensional summary of that history, allowing the Q-function to condition implicitly on observed behavior and generalize across players and replayed behaviors.Source: §A.2 · pp. 14–15
- The critic is trained on replayed trajectory snippets by minimizing k-step TD error with off-policy retrace correction and a separate target network for bootstrapping.The target network and policy are periodically synced with their online counterparts; in the experiments, synchronization follows every 100 gradient steps.Source: §A.2 · p. 15
- Soccer experiments use k = 40 and a replay-buffer threshold to keep experience relatively recent despite off-policy corrections.The target network and policy are synced every 100 gradient steps.Source: §A.2 · p. 15
- For LSTM replay, the stored internal state at each snippet’s start primes the recurrent model, which then updates its state during a 40-step truncated backpropagation-through-time unroll.The initial memory state is stored alongside trajectory data in the replay buffer.Source: §A.2 · p. 15
B POPULATION-BASED TRAINING PROCEDURE … C.1 POLICY PARAMETRIZATION AND OPTIMIZATION
Population-based training uses Elo-based win-rate estimates to guide agent evolution, with eligibility limits, selection, inheritance, and mutation shaping the process. Policies use neural-network actor and critic models optimized with SVG0 and Adam.
B.1 FITNESS
- Elo ratings estimate pairwise win rates and serve as fitness, steering hyperparameter evolution toward winning against other agents rather than hand-designed shaping rewards.For teams, fitness uses the average Elo score; ratings are updated from match results.Source: §B.1 · pp. 15–16
B.2 EVOLUTION ELIGIBILITY
- An agent becomes evolution-eligible after 2 × 109 learning frames overall and 4 × 108 since its last eligibility; a potential parent needs 4 × 108 frames since evolving.The eligibility rule follows Jaderberg et al. (2017), and the parent’s waiting period is described as a “burn-in” period.Source: §B.2 · p. 16
B.3 SELECTION
- For an eligible agent, selection samples another agent uniformly at random and chooses it for evolution only when the estimated win rate is below Tselect.The candidate must have completed its burn-in period.Source: §B.3 · p. 16
B.4 INHERITANCE
- An evolving agent inherits each hyperparameter from its selected parent independently with probability 0.5.Source: §B.4 · p. 16
B.5 MUTATION
- Each evolution action mutates hyperparameters with probability 0.1 at a multiplicative perturbation scale of 0.2, while bounded parameters are kept within valid ranges.Source: §B.5 · p. 16
C.1 POLICY PARAMETRIZATION AND OPTIMIZATION
- Policies and critics use neural networks, with shared player-feature preprocessing and separate actor and critic weights.A shared two-layer network embeds each player’s features into 16 dimensions; max, min, and mean pooling makes the architecture invariant to teammate and opponent ordering. The actor and critic then use 512- and 256-unit feed-forward layers followed by a 256-unit feed-forward or LSTM layer; Gaussian policies are learned with SVG0, and Adam applies gradient updates.Source: §C.1 · p. 17
D HEAD-TO-HEAD TOURNAMENT OF TRAINED AGENTS · E HYPERPARAMETER EVOLUTION
A 50,000-match tournament favored the fully recurrent agent with separate reward channels, while mixed performance against Nash evaluators underscored opponent non-transitivity. Across three hyperparameter-evolution runs, critic learning rate and entropy regularizer changed consistently, suggesting performance was more sensitive to critic than actor learning rate.
D HEAD-TO-HEAD TOURNAMENT OF TRAINED AGENTS
- The tournament showed advantages from shaping rewards, a recurrent critic, and separate reward and discount channels, with a further marginal contribution from a recurrent actor.The round robin compared the best Elo-selected team from each of five populations.Source: D HEAD-TO-HEAD TOURNAMENT OF TRAINED… · p. 17
- The tournament comprised 50,000 matches between the five populations’ best teams, all trained for 5e10 agent steps.Each learner processed at least 5e10 replay-buffer frames, while raw environment steps were much lower.Source: D HEAD-TO-HEAD TOURNAMENT OF TRAINED… · p. 17
E HYPERPARAMETER EVOLUTION
- Across three replicated feed-forward runs, critic learning rate and entropy regularizer evolved consistently.Figure 11 displays the hyperparameter evolution in separate rows for the three seeds.Source: E HYPERPARAMETER EVOLUTION · p. 17, Figure 11 · p. 18
- Critic learning rate tended to decrease over training across the replicated runs.The experiment used a feed-forward policy and critic network.Source: E HYPERPARAMETER EVOLUTION · p. 17, Figure 11 · p. 18
- Consistent evolution across seeds suggested greater performance sensitivity to critic learning rate than to actor learning rate.The authors reasoned that an unimportant hyperparameter would evolve less consistently because selection would instead be driven by other hyperparameters.Source: E HYPERPARAMETER EVOLUTION · p. 17
D HEAD-TO-HEAD TOURNAMENT OF TRAINED AGENTS
- The fully recurrent agent with separate reward and discount channels achieved the highest Elo in a 50,000-match tournament, though its performance against Nash evaluators was more mixed.The tournament compared the best teams from five populations, each trained for 5e10 agent steps; the mixed results highlight possible non-transitivities and the need for opponent robustness.Source: D HEAD-TO-HEAD TOURNAMENT OF TRAINED… · p. 17, Figure 10 · p. 18
F BEHAVIOR VISUALIZATIONS
The section visualizes agent-behavior traces around a repeated “cross pass” motif, supplementing videos available on the website3.
F BEHAVIOR VISUALIZATIONS
- The visualizations show traces of agent behavior in a repeated “cross pass” motif.The paper also provides videos at the website3.Source: F BEHAVIOR VISUALIZATIONS · p. 18
- In Figure 12’s left panel, red agent 0 passes to agent 1, who appears to move into position to receive.Source: Figure 12 · p. 19
- In Figure 12’s right panel, blue agent 1 passes to agent 0.Source: Figure 12 · p. 19