Source-linked AI summary
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, Rob Fergus
TL;DR
Model-free reinforcement learning is sample inefficient, particularly with sparse rewards, motivating methods that let agents explore without external supervision. The paper trains two versions of an agent through asymmetric self-play, with Alice proposing tasks and Bob completing them, and reports faster learning alongside strong performance across challenging benchmarks. The method is scoped to reversible or resettable environments and its theoretical guarantees rely on restrictive tabular assumptions.
Problem
Model-free reinforcement learning often requires many episodes because agents must learn environmental dynamics and reward-seeking behavior together, with sparse rewards providing limited policy updates.
Method
Asymmetric self-play has Alice alter the environment to propose tasks and Bob undo or repeat them using internal rewards, automatically generating exploration curricula in reversible or resettable environments.
Results
On challenging benchmarks, asymmetric self-play is at least as good as state-of-the-art exploration-based reinforcement-learning methods, while enabling significantly earlier reward gains in the reported Reinforce and TRPO experiments.
Takeaways & Limitations
The approach can encourage exploration and automatically generate curricula across discrete and continuous input settings, and in simple environments optimal agents can transition efficiently between reachable states.
Takeaways & Limitations
The approach is restricted to nearly reversible or resettable environments, while the tabular universal-Bob argument assumes finite states, deterministic Markovian transitions, and randomized starts covering the state space.
Abstract
from arXiv · showhide
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against one another. Alice proposes a task for Bob to complete; and then Bob attempts to complete the task. In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset. Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively. Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent. When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.
1 INTRODUCTION
Model-free reinforcement learning is sample inefficient because agents must learn environment dynamics and reward-seeking behavior simultaneously, especially when rewards are sparse. The paper introduces unsupervised training intended to improve exploration and accelerate learning new tasks.
- Model-free reinforcement learning typically requires many episodes because agents learn environmental rules and reward-producing trajectories simultaneously.Sparse rewards further limit the experience that directly updates the policy.
- Sparse rewards make learning inefficient because only a small fraction of experience directly updates the policy.
- The paper introduces unsupervised training that enables exploration and environmental learning without external reward.
- The proposed training is intended to let agents learn new tasks within the environment quickly.
2 APPROACH
Asymmetric self-play gives Alice the role of proposing tasks and Bob the role of completing them, using internal rewards in reversible or resettable environments. Their reward interaction automatically generates tasks near Bob’s capabilities, while theoretical analysis characterizes fast state transitions under specific tabular assumptions.
- Self-play mechanism: Alice proposes tasks by changing the environment, while Bob attempts to undo or repeat those changes depending on whether the environment is reversible or resettable.The method is restricted to environments that are nearly reversible or can be reset.
- Self-play mechanism: Self-play uses internal reward only, allowing Alice and Bob to explore and learn the environment without supervisory signals.Bob can simultaneously train on target-task episodes with external reward.
- Reward design: Alice’s reward increases with Bob’s completion time but penalizes Alice’s own delay, discouraging impossible or unnecessarily long tasks.If Bob fails before the episode limit, tB is set to tMax −tA.
- Reward design: The Alice–Bob feedback loop selects tasks just beyond Bob’s current abilities, automatically constructing an increasingly challenging exploration curriculum.
- Policy parameterization: Alice and Bob use policy functions conditioned on current and target-state observations, with Bob targeting s0 in reversible environments and Alice’s STOP state in resettable ones.
- Policy parameterization: The approach is implemented with tabular policies and neural networks for discrete or continuous inputs, using separate parameters for Alice and Bob.
- Universal Bob: Under finite, deterministic, Markovian tabular assumptions with state-covering randomized starts, equilibrium Bob is a fast policy.
- Universal Bob: The equilibrium argument shows that any challenge Bob handles slowly must either be proposed or become profitable for Alice to propose, contradicting equilibrium otherwise.
3 RELATED WORK
The paper situates asymmetric self-play within self-play reinforcement learning, intrinsic motivation, curriculum learning, goal-conditioned value functions, and related task-generation methods. Its distinguishing feature is using internal self-play to explore environments and automatically construct curricula without explicit novelty or manually specified goals.
- Self-play and intrinsic motivation: Unlike competitive self-play in games, this scheme uses purely internal reward to help an agent learn about its environment and address sparse external rewards.
- Self-play and intrinsic motivation: Intrinsic-motivation methods often reward novel states, whereas asymmetric self-play targets states Bob struggles to reach even after repeated exposure.
- Curriculum learning: Unlike typical curriculum learning, Alice and Bob devise the curriculum automatically without manual specification or training labels.
- Goal-conditioned methods: The task-proposal paradigm resembles goal-conditioned value functions, but the paper’s novelty lies in how Alice defines Bob’s goal.
- Task-generation methods: Compared with prior task-proposal work, this approach differs in how tasks are proposed and communicated, avoiding the stated need for explicitly partitionable and sampleable goal spaces.
- Task-generation methods: Related concurrent methods generate curricula from internal targets, distant start states, adversarial perturbations, or random-Alice-like strategies.
4 EXPERIMENTS
Experiments evaluate asymmetric self-play across toy, MazeBase, and RLLab tasks, mixing self-play with target-task training through policy gradients. The approach generates increasingly difficult tasks and generally accelerates learning, while direct target-task training can fail in some settings.
- Self-play and target-task episodes are mixed and optimized with policy gradients across continuous and discrete environments.The experiments evaluate both reverse and repeat self-play on MazeBase, RLLab, and StarCraft3 tasks.
- 4.1 LONG HALLWAY: The toy chain curriculum efficiently trains Bob for the target task while increasingly forcing exploration of the state space.The target task is essentially the same as the self-play task in this toy setting, so this experiment is not unsupervised learning.
- The paper notes a stability criticism of the approach, but argues that it relies on a sudden Bob-policy jump relative to Alice that is unlikely in practice.
- 4.1 LONG HALLWAY: Alice initially gives Bob easy problems, then constructs harder problems, allowing self-play to match the performance boost of count-based exploration.Random Alice eventually harms learning because Bob does not encounter long treks, whereas policy gradient sees few early successes.
- 4.2 MAZEBASE: In MazeBase, repeat self-play enables significantly faster success than target-task-only training or self-play with random Alice.Alice and Bob automatically build a curriculum through self-play in the MazeBase environment.
- 4.3 RLLAB: MOUNTAIN CAR: For Mountain Car, reset self-play performs comparably to VIME and SimHash, while direct policy-gradient training on target-task samples cannot solve the problem.Reset self-play is used because coasting downhill is easier than climbing uphill in this asymmetric environment.
5 DISCUSSION
Asymmetric self-play is presented as an intrinsically motivated method for exploration and automatic curriculum generation. Across the evaluated settings, it is at least as good as state-of-the-art exploration-incentivized RL methods, while StarCraft results show faster learning and higher convergence than a count-based baseline.
- Asymmetric self-play encourages exploration and automatically generates curricula in discrete and continuous input settings with function approximation.
- In StarCraft, self-play greatly speeds learning the marine-training sub-task and surpasses the count-based approach at convergence.
- The approach is at least as good as state-of-the-art RL methods that incorporate an incentive for exploration on the challenging benchmarks considered.
- In simple environments, optimal agents using the specified self-play rewards can transition between any pair of reachable states as efficiently as possible.
A PSEUDO CODE
The pseudo code alternates Alice’s task-proposing episode with Bob’s task-completion episode, then trains Bob separately on target-task episodes using environmental rewards.
- Algorithm 1 alternates Alice’s and Bob’s turns until Alice stops, Bob reaches the target state, or the episode reaches the maximum length.
- Alice observes the environment and selects actions conditioned on the initial state, while Bob selects actions conditioned on the current and target states.
- The self-play rewards assign Alice γ max(0, tB − tA) and Bob −γtB, favoring task proposals that Bob completes efficiently.
- Algorithm 2 runs Bob on target-task episodes, accumulates environmental reward, and updates Bob’s policy with the resulting return.
B HYPERPARAMETERS USED IN THE EXPERIMENTS
The experiments use specified neural-network, optimization, and environment-encoding settings, while the maze task distinguishes self-play assumptions from the fixed target-task setup.
- Neural-network parameters are initialized from N(0, 0.2), and RMSProp uses hyperparameters 0.97 and 1e−6.
- Some experiments use different hyperparameter values for self-play and target-task episodes, with additional entropy regularization maximizing softmax-layer entropy.
- When the maze light is off, the agent sees only the light switch; in reverse self-play, Bob need not match states invisible under that observation.
- In the maze target task, the agent and goal begin on opposite sides of a wall, with the light off and door closed, requiring light, key, door, and goal interactions.
- Alice and Bob use fully connected networks with hidden layers of 100 and 50 tanh units, bag-of-words object-location inputs, and softmax action outputs.
C.1 BIASING FOR OR AGAINST SELF-PLAY
Figure 6 shows that self-play benefits depend on how closely the self-play task matches the target task. At p(Light off)=0.3, reverse self-play works well while repeat self-play performs poorly, and significant speedups occur when self-play is not biased against the target.
- Figure 6: At p(Light off)=0.3, reverse self-play works well, whereas repeat self-play performs poorly.
- Figure 6: The reverse-self-play result at p(Light off)=0.3 reflects Bob typically returning the light to the on position, matching the target task’s required function.
- Figure 6: Figure 6’s right panel plots speedup against p(Light off), with unsuccessful Mazebase runs assigned a unity speed-up factor.
- Figure 6: Significant speedups occur when self-play is similar to the target task, but the benefit diminishes when self-play is biased against it.
- Figure 6: For reverse self-play, low p(Light off) makes self-play and target tasks similar; for repeat self-play, the opposite relationship applies.
D SWIMMERGATHER EXPERIMENT
A SwimmerGather run tracks target-task and self-play rewards alongside Alice’s evolving task proposals. Alice’s handover locations vary across training stages, and her behavior correlates with self-play and initially with target-task reward.
- Alice’s changing action count and travel distance correlate with Alice–Bob reward and initially with reward on the test target.
- Figure 7 tracks target-task reward, reversible-self-play reward, Alice’s action count, and Alice’s travel distance before handing control to Bob.
- Figure 8 shows how the distribution of Alice’s handover locations varies across training stages.
E STARCRAFT EXPERIMENT
The StarCraft agent controls multiple unit types through shared action-space structure and local/global observations. With longer episodes, self-play continues to outperform baseline methods.
- Agent and action representation: The agent controls multiple active units in parallel, with unit type encoded in observations while actions share a common space.
- Observations: Each unit receives a local observation, while the global observation includes unit counts and accumulated minerals.
- Self-play: In self-play, Bob perceives only the global observation of his target state.
- Exploration reward: Count-based exploration adds an extra reward scaled by α, with α = 0.1 reported as performing best.
- Extended episodes: With episode length increased from 200 to 300, self-play still outperforms baseline methods.
F.1 META-EXPLORATION FOR ALICE
The standard objective can make Alice seek one hardest challenge rather than broad task coverage. In realistic optimization settings, this can produce sub-optimal exploration, including a preferred direction in Swimmer.
- Meta-exploration objective: Alice’s standard expected-reward objective favors finding the single hardest task rather than covering the space of tasks that are difficult for Bob.
- Optimization limitations: With realistic optimization methods or function approximation, Alice and Bob can become stuck in sub-optimal minima.
- Coverage limitation: Local policy updates guarantee fast behavior only on challenges Alice assigns with non-zero probability, without guaranteeing complete challenge coverage.
- Reversibility limitation: In non-fully-reversible environments, equilibria may concentrate Bob’s fast policy on only a subset of challenges.
- Swimmer example: In Swimmer, Alice’s handover distribution shows a preferred direction instead of teaching efficient transitions between arbitrary states.
- Possible remedy: Multiple Alices with diversity regularization are proposed as one way to encourage broader behavior coverage.
F.2 COMMUNICATING VIA ACTIONS
The paper limits Alice to proposing tasks by physically performing them. This is practical in resettable or nearly reversible environments because it addresses task sampling, representation, communication, and difficulty.
- Alice proposes tasks by executing them, a practical approach restricted to environments that allow resetting or are nearly reversible.
- This task representation addresses parameterizing task sampling, communicating tasks, and ensuring appropriate task difficulty.
- The resulting tasks incentivize efficient transitions, while other reward functions could target state-transition statistics such as causality, temporal ordering, or cluster structure.