Emergent Complexity via Multi-Agent Competition

Trapit Bansal; Jakub Pachocki; Szymon Sidor; Ilya Sutskever; Igor Mordatch

Source-linked AI summary

Emergent Complexity via Multi-Agent Competition

Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch

arXiv:1710.03748v3Oct 10, 2017cs.AI

TL;DR

The paper asks whether complex agent behavior requires a complex training environment. It introduces simple competitive 3D physics environments trained with self-play and finds that agents develop diverse, complex skills, while competition supplies a natural curriculum.

Problem
Highly capable agents are generally thought to require complex environments, which can be difficult to create.
Method
The authors train agents with self-play in competitive 3D simulated-physics environments using distributed PPO and an exploration curriculum.
Results
Agents develop complex skills including running, blocking, ducking, tackling, fooling opponents, kicking, and defending with their arms and legs.
Takeaways & Limitations
Simple competitive environments can produce highly complex skills, with the competition generating complexity and a naturally calibrated challenge as agents improve.

Abstract

from arXiv · show

Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX

1 INTRODUCTION

The paper argues that self-play in competitive multi-agent environments can generate behavior much more complex than the environment itself. It applies this idea to continuous control, where agents learn diverse motor skills in simple 3D physics environments.

Highly complex tasks generally require difficult-to-create complex environments, but competition can generate complexity through the agents themselves.
Self-play creates a natural curriculum because opponents of comparable strength provide an appropriate challenge at each skill level.As agents improve, the competing agents make the effective environment more complex and help prevent learning from getting stuck.
The paper investigates whether competitive multi-agent environments can produce complex skills in continuous control domains involving balance, dexterity, and manipulation.
The authors introduce competing 3D physics tasks and train agents with distributed Proximal Policy Optimization plus a simple exploration curriculum.
Agents learn running, blocking, ducking, tackling, fooling opponents, kicking, and defending with both arms and legs, including behaviors without directly engineered rewards.

2 PRELIMINARIES

The preliminaries formulate multi-agent reinforcement learning as partially observable Markov games and review policy-gradient methods. They emphasize PPO’s clipped surrogate objective and established cooperative and competitive multi-agent RL research.

A multi-agent Markov game specifies states, observations, actions, transitions, and agent-specific rewards, with each agent acting through a stochastic policy.
Policy-gradient methods estimate parameter gradients to maximize expected return without requiring an explicit model of the world.
Baselines and generalized advantage estimation reduce gradient-estimate variance, trading some variance reduction for bias.
PPO uses a surrogate objective that penalizes large policy changes and supports multiple optimization epochs on sampled trajectories.
Prior multi-agent reinforcement learning work includes cooperative settings, evolutionary competition, and competitive or collaborative deep Q-learning.

3 COMPETITIVE ENVIRONMENTS

The paper introduces four 1-vs-1 competitive environments in a 3D simulated-physics world, using ant and humanoid bodies. Each task assigns opposing goals and competition-based rewards.

The four environments are Run to Goal, You Shall Not Pass, Sumo, and Kick and Defend.
Experiments use ant bodies with 12 degrees of freedom and 8 actuated joints, and humanoids with 23 degrees of freedom and 17 actuated joints.
Run to Goal: In Run to Goal, each agent races toward a goal on the opponent’s side, with the first to arrive winning.The winner receives +1000 and the opponent -1000; if neither reaches a goal, both receive -1000.
You Shall Not Pass: In You Shall Not Pass, one agent blocks the opponent from reaching its goal while remaining standing, whereas the opponent tries to cross.
Sumo: In Sumo, agents try to knock the opponent down or push it outside a round arena.The winner receives +1000 and the loser -1000; a draw gives both agents -1000.
Kick and Defend: In Kick and Defend, one agent shoots a ball through a 6-unit-wide goal while the other defends within a goalkeeping area 3 units from the goal.Additional defender rewards for ball contact and remaining standing were intended to produce more realistic defending behavior.

4 TRAINING COMPETITIVE AGENTS

The framework uses distributed PPO for competitive agents, addressing sparse-reward exploration and opponent-selection instability through an annealed exploration reward and sampling older opponents.

Training framework: Distributed PPO enables decentralized, large-batch training for multiple competitive agents.The large batch sizes help ameliorate variance and aid exploration.
Exploration challenge: Sparse rewards make discovering successful behavior unlikely, especially for complex 3D agents that first need basic motor skills.For example, a kicker must move toward and displace the ball past a defender and goal boundaries.
Exploration curriculum: An exploration reward teaches basic motor skills early, then is gradually annealed to zero so training favors the sparse competition reward.The dense reward supports behaviors such as walking forward or standing before competition optimization dominates.
Exploration curriculum: Dense-reward training typically lasts only about 10-15% of epochs, while complex emergent behaviors receive no dedicated dense-reward term.The dense terms include distance to goal, x-direction velocity, control cost, impact cost, and standing reward.
Opponent sampling: Training against only the most recent opponent creates an early skill imbalance that can prevent the weaker agent from recovering.The framework therefore examines opponent sampling as a training-stability issue.

5 EXPERIMENTS

Experiments train competitive agents in simulated-physics environments and examine learned behaviors, exploration curricula, opponent sampling, randomization, and policy ensembles. The results show complex, varied behaviors, transfer beyond competition, curriculum benefits, and robustness differences across bodies and training strategies.

5 EXPERIMENTS: Agents are trained on four competitive tasks using the previously described methods to test whether multi-agent training provides a natural curriculum for complex behaviors.The environments use MLP or LSTM policies and value functions, with LSTM used for sumo and kick-and-defend.
5.2 LEARNED BEHAVIORS: The learned behaviors include blocking, standing robustly, toppling opponents, running, ducking, stable fighting, charging, fooling opponents, kicking, and defending with hands and legs.Different random seeds often produce somewhat different behaviors across the tasks.
5.2 LEARNED BEHAVIORS: Skills learned in Sumo can transfer to standing upright against wind forces even when the agent receives no opponent information and never observes the windy environment.The experiment reports that the agent remained upright despite not seeing the windy environment or observing wind forces.
5.4 EFFECT OF OPPONENT SAMPLING: For Humanoid, δ = 0.5 gives the highest win-rate and lowest loss, whereas δ = 0 is best for Ant.The table reports expected loss and expected win-rate for agents trained with each opponent-sampling threshold.
5.3 EFFECT OF EXPLORATION CURRICULUM: Agents trained with an annealed exploration reward beat agents trained with a continuously dense exploration reward, indicating that competition supplies much of the learned behavior's curriculum.The exploration reward is described as crucial for reaching the sparse competition reward, while complex behaviors often receive no direct reward.
5.5 GENERALIZATION AND ROBUSTNESS: Environment randomization improves generalization but can hinder early learning, with highly randomized kick-and-defend agents unable to learn kicking.Randomized factors include arena radius, ball position, and agent starting positions.
5.5.2 COMPETING AGAINST ENSEMBLE OF POLICIES: Training policy ensembles significantly outperforms single-policy training for Humanoid, while Ant performance is similar between the two approaches.The authors associate ensemble training with greater robustness to local minima and qualitatively more robust Humanoid behavior.

6 CONCLUSION

The paper presents competitive multi-agent 3D environments in which agents develop highly complex skills despite simple environments and rewards.

Competitive multi-agent 3D environments produce highly complex skills from simple environments and simple rewards.

A EXPLORATION REWARDS

The exploration rewards combine locomotion incentives with task-specific terms for humanoid and quadruped agents. They reward forward movement, standing or survival, and progress toward task goals or against opponents.

Locomotion rewards: Humanoid locomotion rewards combine forward velocity, action and impact costs, and a standing bonus.The reward is written as rt(s, a) = vfwd + ct(s, a) + Calive.
Locomotion rewards: Quadruped locomotion uses forward velocity, action and impact costs, and a height-dependent survival term.The quadruped reward is defined separately from the humanoid reward, with different coefficients and survival thresholds.
Task rewards: Run-to-goal rewards combine locomotion reward with the agent’s l1 distance from the goal along the x-axis.The same structure is used for humanoid and ant agents.
Task rewards: In You Shall Not Pass, the blocking agent is rewarded for locomotion, standing, and keeping the opponent far from its goal.The reaching agent receives the run-to-goal reward, while the blocker uses the opponent’s distance to the goal.
Task rewards: In Sumo, agents receive locomotion and standing rewards while being penalized for distance from the ring’s center.The humanoid and ant use their respective locomotion rewards in the Sumo objective.
Task rewards: Kick and Defend rewards combine task-specific ball or goal terms with survival incentives and locomotion-related components.The defender’s standing bonus is positive only when positioned in front of the goal area.

B.1 TRANSFER RESULTS

The study tests whether a Sumo agent can maintain balance under wind perturbations despite receiving no opponent information, comparing it with a walking humanoid agent trained in a single-agent environment. Performance is measured by how long each agent remains standing within the arena.

The Sumo agent is evaluated on standing while exposed to wind forces, with opponent-related observations replaced by zeros.The evaluation averages survival steps over 200 episodes, ending when the agent falls or leaves the arena, with a 500-step maximum.
Performance is measured by the number of steps before falling or being pushed out of the arena.An episode ends when zbody ≤0.5 or the agent exits the arena.
The comparison uses a humanoid agent trained for walking in OpenAI Gym Humanoid-v1 with PPO and the same LSTM policy architecture.For this agent, force is applied either along or opposite to its walking direction in half of the episodes.