Source-linked AI summary
Automatic Goal Generation for Reinforcement Learning Agents
Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel
TL;DR
Single-task reward functions do not scale to environments requiring diverse goals. The paper uses adversarial goal generation to adaptively construct an appropriately difficult curriculum, enabling efficient learning across feasible goals under sparse rewards without prior environmental or task knowledge.
Problem
RL commonly trains agents for a single reward-defined task, while many environments require success across diverse goals.
Method
Goal GAN adversarially generates goal states at an appropriate difficulty for the current policy, dynamically adapting the curriculum without prior environment or task knowledge.
Results
The framework enables agents to quickly learn policies reaching all feasible goals and considerably improves sample efficiency in multi-goal learning.
Takeaways & Limitations
The method supports multi-task settings such as navigation and manipulation and naturally handles sparse rewards without manually modifying each task’s reward function.
Takeaways & Limitations
The approach assumes continuous goal-space generalization and that every reachable goal has a policy achieving it reliably.
Abstract
from arXiv · showhide
Reinforcement learning is a powerful technique to train an agent to perform a task. However, an agent that is trained using reinforcement learning is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing. We use a generator network to propose tasks for the agent to try to achieve, specified as goal states. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent. Our method thus automatically produces a curriculum of tasks for the agent to learn. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment. Our method can also learn to achieve tasks with sparse rewards, which traditionally pose significant challenges.
1. Introduction
Standard RL trains an agent for a single reward-defined task, but many environments require success across diverse goals. The paper proposes Goal GAN to generate appropriately difficult goals and automatically curriculum-train a policy across feasible tasks.
- RL agents are typically trained to optimize one reward function for one task.
- Real-world settings may require diverse tasks, including navigation to varying positions or moving objects to varying locations.
- Goal GAN uses a discriminator to assess goal difficulty and a generator to propose goals appropriate for the current policy.
- The framework automatically creates a curriculum by generating goals slightly more difficult than those the agent already achieves.
- The method improves sample efficiency for reaching all feasible goals and handles sparse rewards without manually modifying each task’s reward function.
2. Related Work
Related work addresses multi-task learning through contextual policy search, intrinsic motivation, skill learning, and curricula. This paper differs by generating a continuous-space curriculum for directly learning multiple goals, including under sparse rewards.
- Contextual Policy Search: Contextual policy-search methods treat tasks as contexts, whereas this work uses a curriculum for efficient multi-task learning in sparse-reward settings.
- Intrinsic Motivation: Intrinsic-motivation methods explore parameterized task spaces, but the paper reports that SAGG-RIAC explores less efficiently than its approach.
- Skill-learning: Skill learning reuses skills for collections of tasks, but prior work often targets discrete environments or continuous tasks with demonstrations.
- Skill-learning: Previous skill-learning approaches commonly pre-train primitives and require downstream training to compose them purposefully.
- Curriculum Learning: Unlike hand-designed curricula, this method generates a training curriculum directly in continuous task space and trains policies to achieve multiple goals.
3. Problem Definition
The paper formulates learning as training a goal-conditioned policy to maximize average success across a goal distribution. Goals define state sets with sparse binary rewards, while the approach assumes local generalization and reliably solvable reachable goals.
- Traditional RL learns a policy maximizing expected cumulative reward for a single reward function over a finite episode.
- The framework replaces one reward function with a family indexed by goals, each corresponding to a target state set.
- A goal-conditioned policy receives the goal as input and acts optimally with respect to that goal’s reward.
- The sparse reward indicates whether the agent reaches the target set, and episodes terminate when the target is reached.
- The overall coverage objective is average success probability across goals sampled from a test distribution, assumed uniform for simplicity.
- Assumptions: The method assumes policies interpolate among nearby goals, provide useful initialization for close goals, and can reliably achieve every reachable goal.
4. Method
The method labels goals by their difficulty for the current policy, trains a Goal GAN to generate intermediate-difficulty goals, and repeatedly uses them to improve policy coverage. This creates an automatic curriculum without prior knowledge of the environment or tasks.
- Goal labeling and curriculum: The algorithm labels goals according to whether the current policy achieves an intermediate level of return, then trains a generator on those labels.The process iterates through goal labeling, generator training, policy training, and policy convergence.
- Goal labeling and curriculum: Goals of Intermediate Difficulty constrain training to goals with enough reward signal while excluding goals the policy already performs above Rmax.Rmin supplies a minimum expected return for learning, while Rmax prevents repeated focus on mastered goals.
- Adversarial goal generation: Goal GAN uses adversarial training to approximate uniform sampling from the set of Goals of Intermediate Difficulty.The modification incorporates positive examples from the desired distribution and negative examples outside its support, improving accuracy with few positive samples.
- Adversarial goal generation: A goal generator maps noise z to goals, while a goal discriminator distinguishes intermediate-difficulty goals from other goals.The networks are optimized similarly to Least-Squares GANs, with binary labels enabling training from negative examples.
- Iterative policy optimization: At each iteration, generated and replayed goals train the policy, whose returns label goals for the next GAN update.The replay buffer is used to prevent catastrophic forgetting.
- Iterative policy optimization: The resulting curriculum emerges as a by-product of optimization and requires no prior knowledge of the environment or tasks.The generator tracks goals receiving intermediate returns until the policy learns to reach all feasible goals.
5. Experimental Results
Experiments evaluate whether Goal GAN improves curriculum efficiency, tracks appropriately difficult and multimodal goals, and scales to high-dimensional spaces with lower-dimensional feasible regions. Across these settings, the method improves training efficiency, maintains appropriate goal difficulty, tracks multiple modes, and discovers feasible goals as dimensionality increases.
- Experimental setup: The experiments test faster coverage maximization, appropriate-difficulty goal sampling, multimodal distribution tracking, and scaling to high-dimensional goal spaces.Evaluation uses Free Ant, Maze Ant, a multipath point-mass maze, and an N-dimensional point-mass environment.
- Ant locomotion: Goal GAN trains Free Ant and Maze Ant more quickly than the compared baselines.The y-axis reports average return over feasible goals, while the x-axis counts newly sampled goals; curves average 10 random seeds.
- Ant locomotion: Uniform sampling wastes attempts on infeasible or currently unreachable goals, whereas the other baselines perform better but do not surpass Goal GAN.Asymmetric Self-play also incurs an additional goal-generator training burden not represented in the plots.
- Ant locomotion: Goal GAN shifts its sampled-goal distribution toward regions where the current policy receives learning signal but needs improvement, keeping about 20% of generated goals appropriately difficult.Samples combine newly generated goals with replayed goals to reduce forgetting; analogous Free Ant behavior forms a growing ring around the origin.
- Ant locomotion: Goal GAN performs close to rejection sampling while requiring far fewer samples, and the rejection-sampling oracle serves as an upper bound.The oracle samples uniformly from goals meeting the appropriate-difficulty criterion but is orders of magnitude more expensive in labeling.
- Multipath point-mass maze: In the multipath maze, Goal GAN tracks multimodal appropriately difficult goals, needs no prior knowledge of the number of modes, and consistently reaches full coverage.Replay produces trailing high-reward spread, while Goal GAN produces more concentrated goal nodes.
- N-dimensional point mass: As dimensionality increases, uniform sampling worsens because feasible states occupy a decreasing fraction of the full space, while Goal GAN’s performance decays less and discovers the feasible subset.The oracle and L2-reward baseline achieve perfect performance in this task, where the optimal policy moves straight toward the goal.
6. Conclusions and Future Work
The paper proposes training a single policy to achieve diverse goals under sparse rewards through an automatically generated curriculum. Goal GAN generates goals at an appropriate difficulty without prior environmental or task knowledge.
- The method trains a single policy to succeed on a variety of goals under sparse rewards.
- Generative adversarial training automatically produces goals that are neither too hard nor too easy for the current policy.
- The curriculum dynamically adapts to agent performance without prior knowledge of the environment or tasks.
A. Implementation details
The implementation bootstraps Goal GAN from states visited by the initial policy and maintains replay of earlier goals. This addresses sparse-reward initialization and prevents forgetting previously learned goals.
- A regularized replay buffer reuses goals from previous iterations so the policy does not forget earlier capabilities.
- Random or uniform generator initialization can yield no-reward goals, causing slow training under sparse rewards.
- The generator is initialized from states visited by the initial policy, whose goals are likely already achievable.
- Goal GAN is trained to match the initial policy's state-visitation distribution over achievable goals.
B. Experimental details
The experiments evaluate goal-conditioned control in a high-dimensional quadruped and maze settings. Goals are defined by center-of-mass positions, with bounded environments and episode horizons.
- The Ant is a dynamically complex quadruped with 8 actuated joints, making navigation require motor coordination.
- In Free Ant, the agent must reach any commanded point in the square [−5m, 5m]2 within 500 timesteps.
- The maze environment constrains movement to a 6m x 6m maze centered within a 10m x 10m state-space.
B.3. Point-mass specifications
The point-mass experiments vary goal-space dimensionality and use fixed rollout and optimization settings. The agent observes position and velocity, while Goal GAN goals receive Gaussian perturbations.
- Each point-mass rollout provides 400 timesteps, with 0.02 seconds per timestep and acceleration capped at 5 m/s2 per dimension.
- The point-mass observation contains position and velocity, totaling 2N dimensions.
- Generated goals are perturbed independently in each dimension with zero-mean, unit-variance normal noise.
- The policy uses a goal-conditioned neural network and is optimized with TRPO across repeated policy-update iterations.
C. Study of GoalGAN goals
The study examines how GoalGAN labels and selects goals for curriculum learning. It avoids extra labeling rollouts by reusing recent policy-update data and compares this approach with learning-progress selection.
- Goal labels indicate whether a goal’s expected return lies within the current policy’s appropriate difficulty range.The range is defined by Rmin ≤ ¯Rg(πi) ≤ Rmax.
- Reusing rollouts from the previous policy update avoids executing additional rollouts solely for labeling.This introduces a small labeling delay because the rollouts were collected under πi−1 rather than πi.
- The delayed-label approximation does not significantly affect learning compared with labels estimated from three new rollouts.The true-label curves are only slightly better than the method’s curves in Figures 8a–8b.
- The label estimates are nearly invariant for Rmin ∈ (0, 0.25) and Rmax ∈ (0.75, 1).Most goal success rates average at most four attempts, leaving these threshold inequalities unchanged across the stated ranges.
- The study also compares GoalGAN’s goal selection with learning-progress criteria used in prior work.For continuous goal spaces, learning-progress estimation requires measuring performance before and after each policy update.
D. Goal Generation for Free Ant
In the Free Ant experiment, GoalGAN generates progressively farther goals as the policy expands its reachable region. In a multi-path maze, the method reliably achieves full coverage across runs.
- GoalGAN produces a growing circle around the origin as the Free Ant policy learns to reach nearby points.The generator responds by sampling goals at increasingly distant positions.
- The Free Ant experiment evaluates learning to move a robotic quadruped to all points in free space.The generated goals are visualized alongside policy performance over the state space.
- Figure 8 compares training efficiency for the method and different variants using averages over 10 random seeds.
- All runs of the algorithm reliably reach full coverage in the Multi-path Point-mass Maze.The result demonstrates that the method handles environments with several ways of reaching each part of the maze.
F. Comparisons with other methods
The paper compares GoalGAN with related goal-generation approaches and documents how SAGG-RIAC is implemented for the experiments. These comparisons use goal difficulty, sampled regions, and learning curves as evaluation views.
- Goal visualization: GoalGAN samples goals classified by reward level: high-reward, appropriately difficult, or low-reward goals.The categories are defined using Rmin and Rmax relative to the current policy’s expected return.
- Goal visualization: Figures 10 and 14 visualize policy performance and generated regions across the feasible state or goal space.Figure 10 colors grid cells by expected return, while Figure 14 shows SAGG-RIAC-generated regions.
- SAGG-RIAC implementation: The SAGG-RIAC implementation samples Nnew goals, collects goal-directed rollouts, updates regions, and trains with TRPO in each iteration.
- SAGG-RIAC implementation: The implementation uses recommended mode proportions p1 = 0.7, p2 = 0.2, and p3 = 0.1.
- GoalGAN comparison: Figure 11 isolates GoalGAN samples to examine whether the generator fits multimodal goal distributions.Replay-buffer samples are excluded from this visualization.
- GoalGAN comparison: In the multi-path maze, the full learning curves show reliable full coverage across all algorithm runs.