PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Coordination Through Competition

Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, Thore Graepel

arXiv:1902.07151v2cs.AI

TL;DR

The paper asks how cooperative behavior can emerge in competitive multi-agent reinforcement learning without predefined evaluation tasks or human baselines. It introduces a continuous-physics soccer environment and trains decentralized agents with population-based co-play and automatically optimized shaping rewards. Agents progress from random behavior to ball chasing and then evidence of cooperation, while Nash-averaging evaluation addresses opponent diversity and exploitability.

  • Problem

    Cooperative behavior is difficult to study in continuous competitive multi-agent reinforcement learning because shaping rewards can misalign with team objectives and standard evaluators may be unavailable.

  • Method

    The paper trains independent agents with decentralized population-based training and co-play, automatically optimizing shaping-reward weights and separate discount factors in a continuous-physics soccer environment.

  • Results

    Agents progress from random behavior through ball chasing to cooperative and strategic behavior, including repeated passing motifs, while shaping rewards shift toward long-horizon team rewards.

  • Takeaways & Limitations

    Automatically optimizing simple shaping rewards can produce long-horizon cooperative team behavior in the studied soccer environment.

Abstract

from arXiv · show

We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.

1 INTRODUCTION

The paper introduces a continuous-physics competitive soccer environment for studying cooperative behavior in multi-agent reinforcement learning. Decentralized population-based training with co-play evolves agents from individual shaping-reward optimization toward cooperative team behavior, while game-theoretic evaluation addresses difficult baseline and exploitability issues.

  • Environment: The authors introduce a scalable multi-agent soccer environment with simulated physics and simple bodies to study coordination, shaping rewards, exploitability, and evaluation.The environment uses MuJoCo and a 3-dimensional action space, while remaining extensible to more agents and complex bodies.
  • Training framework: Decentralized population-based training with co-play can produce emergent cooperation without centralized learning in the soccer domain.The framework uses independent learners with recurrent memory and decomposed shaping-reward channels.
  • Behavioral progression: Agents evolve from random behavior through ball chasing to cooperative and strategic behavior with awareness of other agents.The progression is associated with shaping rewards becoming more aligned with long-horizon team objectives over time.
  • Coordination evidence: The paper evaluates coordination using game statistics, value-function analysis, and counterfactual policy divergence.These methods provide quantitative evidence alongside visual demonstrations of behavior.
  • Evaluation: Nash averaging is used to evaluate agents against a support set of pre-trained agents when handcrafted bots or human baselines are unavailable.The paper highlights evaluator diversity and exploitability by observing non-transitive pairwise rankings in tournaments.

2 PRELIMINARIES

The paper formulates soccer as a multi-agent reinforcement learning problem in which agents interact through a Markov game and optimize individual cumulative rewards. The formulation accommodates continuous states and actions, partial observations, stochastic transitions, and history-dependent policies.

  • MARL formulation: Multi-agent reinforcement learning models agents interacting with an environment while learning to optimize individual cumulative rewards.The setting may be cooperative, competitive, or mixed depending on reward alignment.
  • Markov game: A Markov game comprises a state space, agent-specific observation and action sets, reward functions, observation functions, transition dynamics, and an initial-state distribution.Each agent receives observations through its observation function and acts within its action set.
  • Dynamics: In the soccer application, state and action spaces are continuous, and transition distributions are treated as densities.Agents sequentially choose actions from observations and generate trajectories over a finite or potentially unbounded horizon.
  • Objective and policy: Agents maximize expected cumulative reward, with discounting by γ < 1 when an infinite horizon requires convergence.Policies may depend on an agent’s history of prior observations and actions rather than only the current state.
  • Special case: A one-agent Markov game becomes a partially observed Markov decision process, and all players in this work share the same action and observation spaces.The POMDP correspondence follows from the single-agent special case.

3 METHODS

The method combines population-based training, co-play, continuous-control reinforcement learning, recurrent critics, and decomposed shaping rewards with separately optimized discount factors. This design lets training adapt hyperparameters and reward channels online while improving robustness to opponent variation and separating short-horizon shaping from long-horizon environmental objectives.

  • Framework: The framework combines single-agent continuous-control reinforcement learning with population-based training to optimize agents and hyperparameters online.It is designed to address exploitability from overfitting to one opponent policy and the difficulty of tuning shaping rewards.
  • Population-based training: Population-based training maintains independently learning agents whose parameters and hyperparameters can be inherited from stronger agents and mutated during training.The population evolves hyperparameters rather than committing to one fixed value.
  • Population-based training: The PBT-MARL procedure initializes a population, trains agents through matches, updates ratings, and periodically selects agents for an evolution procedure.Eligible agents can inherit network parameters and hyperparameters from selected population members.
  • Co-play: Co-play trains agents against sampled members of the learning population, making other agents part of each learner’s environment.Optimizing returns averaged over population opponents provides robustness to exploitability.
  • Continuous-control learning: SVG0 estimates continuous-control policy gradients, while entropy regularization is added to the objective and optimized with Adam.The method averages gradients across games to optimize the population-based objective.
  • Value estimation: The recurrent critic uses experience replay, k-step TD errors, off-policy Retrace corrections, and a target network to estimate values under changing opponents.Recurrence helps the critic condition implicitly on observed behavior when opponent identities are not revealed.
  • Reward shaping: Shaping rewards are weighted individual signals optimized by population-based training, while separate discount factors are learned for each reward channel.The design permits high discounting for sparse environmental rewards and myopic optimization for dense shaping rewards.

4 EXPERIMENTAL SETUP

The experimental setup uses a 2v2 MuJoCo soccer environment and evaluates trained agents with Nash-averaged evaluators derived from tournament results. The evaluation compares pairwise expected goal differences while accounting for diverse, non-transitive agent performance.

  • 4.2 TRAINING: Population-based training uses 32 agents, with evolution triggered when an agent’s expected win rate against another selected agent falls below 0.47.After evolution, agents undergo replay-buffer refill and burn-in periods before learning or evolving again.
  • 4.3 EVALUATION: Nash averaging selects evaluator teams from tournament results using a meta-game over pairwise win rates and mixed strategies.The procedure is designed to avoid rankings being biased by redundant agents.
  • 4.3 EVALUATION: Three evaluator teams receive nonzero Nash-averaging weights because they exhibit diverse policies and non-transitive performance.The evaluators were selected from 10 teams after 1M tournament matches, with 25B learning frames per team.
  • 4.3 EVALUATION: Figure 2 presents support weights, pairwise expected goal differences, Elo ratings, and pairwise differences across evaluator and full-agent sets.The four panels separate the Nash support set, evaluator comparisons, tournament Elo ratings, and all-agent comparisons.

5 RESULTS

Population-based training with co-play and evolving reward mechanisms produces a progression from ball chasing toward coordinated soccer behavior. Evaluation and behavioral analyses indicate that long-horizon planning, passing, and teammate-sensitive policies emerge during training.

  • Population-based training with co-play induces progression from random behavior to ball chasing and finally coordinated behavior.
  • 5.1 ABLATION STUDY: Evolution improves population-level performance while shifting learning rates, entropy costs, and discount factors toward lower exploration and longer-term returns.Evolution begins at 2B steps and quickly improves agent performance; critic learning rate and entropy costs decrease while discount factors increase.
  • 5.1 ABLATION STUDY: Dense shaping rewards accelerate early learning, but evolution later shifts emphasis from scoring toward defending and reduces reliance on velocity-to-ball rewards.Agents first optimize scoring, then incorporate defending as the relative weight of conceding changes over training.
  • 5.1 ABLATION STUDY: Recurrent action-value functions reach weighted expected goal difference 0 at 35B steps versus 22B for the feedforward counterpart.The recurrent policy may overfit to behaviors from its own population when evaluated against out-of-sample agents.
  • 5.1 ABLATION STUDY: Different reward components evolve different planning horizons: sparse rewards and ball-to-goal velocity become longer-horizon, while velocity-to-ball becomes short-horizon.
  • 5.2 EMERGENT MULTI-AGENT BEHAVIORS: Behavioral evidence includes increasingly frequent long-range passes, teammate-sensitive policies, and recurring passing motifs such as crosses and consecutive passes.Long-range passes eventually reach equal frequency with long-range interceptions, while counterfactual analysis shows teammate position increasingly influences policy.

6 RELATED WORK

The paper situates its simulated-physics soccer environment within continuous-control and game-theoretic multi-agent research, emphasizing interpretability, extensibility, and population-based competition.

  • 6 RELATED WORK: The simulated-physics soccer domain targets open-ended continuous-control research and makes complex multi-agent interactions physically observable and interpretable.Passing is given as an example of an interpretable interaction.
  • 6 RELATED WORK: Population-based training plays simultaneous learning agents rather than previous best responses, offering a more data-efficient and parallelizable alternative to PSRO and NFSP.PSRO and NFSP are motivated by fictitious play and double-oracle methods for matrix games.
  • 6 RELATED WORK: The environment is intended as an extensible research platform that can support multiple directions beyond the presented setting.The passage states that the environment is easily extendable along several lines, though the excerpt does not enumerate them.

7 CONCLUSIONS AND FUTURE WORK

The paper concludes that decentralized population-based training with optimized shaping rewards can produce coordinated behavior in a simulated-physics soccer domain, while evaluation reveals non-transitive matchups and a need for robustness.

  • 7 CONCLUSIONS AND FUTURE WORK: A new 2v2 simulated-physics soccer domain supports continuous multi-agent reinforcement learning research with independent agents and repeated passing motifs.The environment is presented as a platform for multi-agent research in continuous physical worlds.
  • 7 CONCLUSIONS AND FUTURE WORK: Distributed population-based training and automatic optimization of shaping-reward channels learn the environment end-to-end.The method also automatically optimizes separate discount factors for shaping rewards to support transition toward sparse long-horizon team rewards.
  • 7 CONCLUSIONS AND FUTURE WORK: Counterfactual policy divergence is introduced to analyze agent behavior.
  • 7 CONCLUSIONS AND FUTURE WORK: Non-transitive pairwise match results highlight a practical need for robust evaluation, which the paper leaves as future work.
  • 7 CONCLUSIONS AND FUTURE WORK: The environment can be scaled to more agents and more complex bodies, but those extensions are left for future work.

A.2 Q-VALUE UPDATES

The Q-value update method addresses partial observability and changing opponent policies by estimating values from observation histories with recurrent critics and recent off-policy experience.

  • A.2 Q-VALUE UPDATES: Because other agents’ identities and the true game state are unavailable, the critic models an agent’s observation history to estimate action values.The method also addresses changing effective dynamics caused by evolving policies in off-policy replay.
  • A.2 Q-VALUE UPDATES: The replayed experience is represented as a k-step trajectory snippet sampled uniformly from the replay buffer and corrected with an off-policy Retrace target.
  • A.2 Q-VALUE UPDATES: Target networks and policies are periodically synchronized for stability, with synchronization after every 100 gradient steps in the experiments.
  • A.2 Q-VALUE UPDATES: The soccer experiments use k = 40 and restrict replay data to remain relatively recent despite applying off-policy corrections.
  • A.2 Q-VALUE UPDATES: When the critic uses an LSTM, replay restores its stored initial memory state and trains through truncated backpropagation-through-time unrolls of length 40.

B.1 FITNESS

Fitness and evolution use Elo-based competitive performance to select agents and transfer or mutate hyperparameters, while actor-critic networks estimate policies and values from processed player observations.

  • B.1 FITNESS: Elo ratings estimate expected pairwise win rates and determine agent eligibility for evolution, using average Elo for multi-player teams.
  • B.1 FITNESS: Using Elo as fitness automatically optimizes hyperparameters toward win rate against other agents instead of requiring individually hand-crafted shaping rewards.
  • B.1 FITNESS: The Elo update applies a K-scaled difference between the observed score and expected win rate.
  • B.1 FITNESS: Agents become eligible for evolution after processing 2 × 10^9 frames overall and 4 × 10^8 frames since their previous eligibility.
  • B.1 FITNESS: Evolution compares an eligible agent with a burn-in-completed agent and selects a replacement when the expected win rate falls below a threshold.The selection procedure samples the comparison agent uniformly and returns it when the expected win rate is below Tselect.
  • B.1 FITNESS: Selected agents inherit hyperparameters independently with probability 0.5 and mutate them with probability 0.1 at a perturbation scale of 0.2.Bounded ranges keep selected hyperparameters valid during training.
  • B.1 FITNESS: The networks embed each player’s features with a shared two-layer network, aggregate dimension-wise extrema and means, and then apply actor and critic feed-forward layers.The actor and critic use separate weights, and their final layers can be recurrent through an LSTM.
  • B.1 FITNESS: Gaussian policies are trained with SVG0, while Adam applies gradient updates to the policy and critic.

D HEAD-TO-HEAD TOURNAMENT OF TRAINED AGENTS

A 50,000-match round-robin tournament compared the strongest teams from five populations, while repeated-seed analysis examined which hyperparameters evolved consistently. The resulting Elo ranking favored recurrent agents with shaping rewards and separate reward and discount channels.

  • 50,000 matches were played between the best teams selected by Elo from five populations.All teams had been trained for 5e10 agent steps, although raw environment steps were lower.
  • The tournament was used to compare shaping rewards, recurrent critics, and separate reward and discount channels.The reported comparison attributes team differences to these architectural and reward-design choices.
  • Critic learning rate and entropy regularizer evolved consistently across three training seeds.The critic learning rate generally decreased over time, which the study treats as evidence that these hyperparameters mattered to performance.
  • 1071 Elo ranked the LSTM agent with evolution, shaping rewards, and separate channels highest.The ranking placed the corresponding LSTM configurations at 1069 and 1006 Elo, followed by feed-forward teams at 956 and 898.

F BEHAVIOR VISUALIZATIONS

Behavior visualizations document a repeated cross-pass motif in which agents pass to teammates who move into receiving positions. The traces therefore illustrate coordinated passing behavior in the trained teams.

  • Repeated cross-pass traces visualize agents passing to teammates who move into receiving positions.The figure shows both red and blue teams performing this passing pattern.
Loading 1902.07151v2…