Source-linked AI summary
Reverse Curriculum Generation for Reinforcement Learning
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, Pieter Abbeel
TL;DR
Goal-oriented RL is difficult when rewards are sparse and reward shaping or demonstrations are unavailable. The paper learns from a single achieved goal state by adaptively training from nearby starts and expanding outward, and reports success on difficult simulated navigation and manipulation tasks that state-of-the-art RL methods cannot solve.
Problem
Goal-oriented tasks often provide only sparse rewards, making exploration difficult and motivating reward shaping or expert demonstrations.
Method
The method automatically adapts start-state distributions, training first near a provided goal and then from increasingly distant states generated through short random walks.
Results
The approach solves difficult navigation and fine-grained manipulation tasks reported as unsolvable by state-of-the-art learning methods.
Takeaways & Limitations
Reverse curriculum generation provides an efficient way to optimize performance on the original goal-oriented problem without reward engineering or demonstrations.
Takeaways & Limitations
The current curriculum grows uniformly outward from a single goal until covering the original start distribution, rather than biasing starts toward a desired distribution.
Abstract
from arXiv · showhide
Many relevant tasks require an agent to reach a certain state, or to manipulate objects into a desired configuration. For example, we might want a robot to align and assemble a gear onto an axle or insert and turn a key in a lock. These goal-oriented tasks present a considerable challenge for reinforcement learning, since their natural reward function is sparse and prohibitive amounts of exploration are required to reach the goal and receive some learning signal. Past approaches tackle these problems by exploiting expert demonstrations or by manually designing a task-specific reward shaping function to guide the learning agent. Instead, we propose a method to learn these tasks without requiring any prior knowledge other than obtaining a single state in which the task is achieved. The robot is trained in reverse, gradually learning to reach the goal from a set of start states increasingly far from the goal. Our method automatically generates a curriculum of start states that adapts to the agent's performance, leading to efficient training on goal-oriented tasks. We demonstrate our approach on difficult simulated navigation and fine-grained manipulation problems, not solvable by state-of-the-art reinforcement learning methods.
1 Introduction
The paper addresses sparse-reward goal-oriented RL by eliminating reward engineering and demonstrations, instead learning outward from a known goal state. Its adaptive reverse curriculum generates increasingly distant starts and solves difficult navigation and manipulation tasks.
- Motivation: Sparse binary rewards make goal-oriented tasks difficult, while dense shaping can produce undesired behavior and requires task-specific expert effort.Demonstrations provide another source of prior knowledge but require expert intervention.
- Approach: The method begins near the goal, where reaching it is easier, then expands toward harder start states using short random walks from partially successful states.This reverse-learning strategy is inspired by value iteration, which solves harder sub-problems using easier ones.
- Approach: The framework automatically adapts the start-state curriculum by observing agent performance during training.It requires no task knowledge beyond one state that achieves the goal.
- Contributions: The paper formalizes selecting start-state distributions to maximize learning speed and proposes a varying distribution that forms an automatic curriculum.These are presented as central contributions of the framework.
- Evaluation: Experiments show the approach solves difficult navigation and fine-grained manipulation tasks that state-of-the-art learning methods cannot solve.
2 Related Work
Prior curriculum methods in RL commonly use fixed task sequences or require baseline information, while related adaptive approaches may focus exploration narrowly or rely on stronger assumptions. This paper instead targets efficient learning across continuously parameterized start states.
- Curriculum learning: Most practical RL curriculum approaches rely on pre-specified task sequences, unlike adaptive curricula developed mainly for supervised learning.
- Curriculum learning: A related framework ranks finite tasks by baseline performance, but requires each task to be independently learnable.
- Adaptive task generation: The paper distinguishes its method from asymmetric self-play, which may concentrate exploration on only a subset of appropriately difficult tasks.The comparison is especially relevant to continuous state-action spaces typical in robotics.
- Backward solution construction: Related backward-controller methods grow stabilized trajectories from the goal, but require a dynamic model or baseline state distributions.The proposed approach does not require a dynamic model.
- Start-state distributions: Earlier restart-distribution work proved that a changed start distribution can improve training, but did not provide a practical procedure for choosing it.
3 Problem Definition
The paper formulates learning a policy that reaches a goal space from a start-state distribution, then modifies that distribution during training to improve learning while evaluating performance on the original distribution. The setup assumes reset access, one known goal state, and random-action connectivity.
- Problem formulation: The problem is to learn a policy that reaches a specified goal space from start states sampled from a given distribution.
- Preliminaries: An MDP is defined by state and action sets, transition probabilities, bounded rewards, a start-state distribution, and a finite horizon; the policy maximizes expected return.
- Modified training distribution: The method uses a different start-state distribution ρi at each iteration to maximize learning rate, while evaluating progress under the original distribution ρ0.Convergence of ρi to ρ0 is desirable but not required when supports coincide.
- Goal-oriented tasks: Goal-oriented tasks use a binary reward for reaching Sg, terminate upon reaching the goal, and interpret return from each start as the goal-reaching probability.
- Start and goal spaces: Training from many feasible starts around the goal is intended to improve robustness to deviations from an intended trajectory.The goal space for fine-grained manipulation is a small neighborhood around the desired configuration.
- Assumptions: The approach relies on arbitrary state resets, one provided goal state, and a communicating class under uniformly sampled random actions containing the starts and goal.These assumptions support initializing and expanding the curriculum from the goal.
4 Methodology
The method trains policies from start states that progressively expand outward from a provided goal state, using performance-adaptive sampling to form a curriculum. It generates feasible nearby states through short action-space random walks and retains partially successful starts for further training.
- Policy optimization: “Good starts” are states from which the current policy reaches the goal sometimes but not always, providing a strong learning signal.They are selected using return bounds Rmin and Rmax, which correspond to bounds on success probability.
- Sampling nearby states: Nearby feasible states are generated by resetting to a seed and executing short Brownian-motion rollouts with Gaussian action noise.Action-space perturbations are used because direct state-space noise can produce infeasible robotic configurations.
- Reverse curriculum: The algorithm begins with the goal state and progressively trains the policy from increasingly distant start states.Each iteration samples new states near selected starts and reuses previous good starts through a replay buffer.
- Algorithm properties: The replay buffer preserves earlier good starts to reduce catastrophic forgetting while the sampled distribution expands through the state space.Under the communicating-class assumption, the expansion can eventually reach all start states in S0.
- Policy optimization: The curriculum adapts to performance by retaining starts whose estimated returns fall between Rmin and Rmax for the next iteration.States that are mastered are removed, while states still requiring training guide further nearby-state sampling.
5 Experimental Results
Experiments evaluate adaptive start-state curricula on simulated navigation and fine-grained manipulation tasks. Modifying the start-state distribution improves learning and final performance, while focusing sampling on good starts accelerates learning further.
- Task settings: The evaluation covers point-mass and ant maze navigation plus ring-on-peg and key-insertion manipulation tasks in MuJoCo.The target start-state distributions are uniform over feasible positions or joint configurations for each task.
- Start-state distribution: Adaptive start-state methods consistently outperform uniform sampling across the evaluated goal-oriented tasks.The comparison uses TRPO and evaluates learning curves by mean and variance over five random seeds.
- Start-state distribution: Uniform sampling reaches only about 10% success on the ring task and 2% on the key task, typically from starts already near the final position.The proposed methods reach the goal from a wider range of far-away start states under the same evaluation metric.
- Good starts: Training on good starts improves the learning rate beyond sampling nearby states from all previously used starts.The all-starts ablation still outperforms unmodified sampling but learns more slowly.
- Good-start sampling: Oracle rejection sampling further improves point-mass-maze learning, showing that the practical nearby-state approximation trades performance for greatly reduced computation time.The oracle is orders of magnitude more expensive in sample complexity and is practical only for the easier point-mass task.
- Alternative comparison: Asymmetric self-play performs poorly in the point-mass maze because Alice often becomes trapped in a local optimum and proposes poor starts for Bob.The comparison highlights a failure mode of an alternative start-state generation strategy in this setting.
6 Conclusions and Future Directions
The paper concludes that automatically adapting start-state distributions can solve difficult goal-oriented tasks that standard RL methods cannot, while identifying limits and future extensions.
- The method automatically adapts start-state distributions to efficiently optimize performance on the original problem.
- The current curriculum expands uniformly outward from a single goal until covering the original start-state distribution.
- The authors propose combining automatic curriculum generation with biased start distributions, goal generation, and domain randomization in future work.
A.1 Hyperparameters
The experiments use fixed policy-optimization settings and construct each training distribution from newly generated and previously sampled start states.
- Each iteration retains M = 10000 seed and generated start states before subsampling Nnew = 200 new states and adding Nold = 100 old states.
- Policies use a (64, 64) Gaussian MLP with TRPO, a (32, 32) baseline MLP, and batches of 50,000 timesteps.
- Training uses horizon T = 500 for most tasks and T = 2000 for Ant maze experiments, ending episodes when the goal is reached.
- The optimization uses discount factor γ = 0.998, with Rmin = 0.1 and Rmax = 0.9 for the start-state definition.
A.2 Performance metric
Performance is evaluated by reaching a specified goal region from feasible start states sampled uniformly from the original start-state set.
- The task objective is reaching goal region Sg from feasible states s0 ∈ S0 within a specified distance of the goal.
- Progress on ηρ0(πi) is measured using trajectories initialized from states uniformly sampled from S0.
- For quasi-static manipulation tasks, the evaluation generates only initial joint positions and sets all initial velocities to zero.
B.1 Distance reward shaping
Distance-based reward shaping is not reliably beneficial: it leaves the ring task largely unchanged but substantially harms the key task by guiding updates toward a suboptimal local optimum.
- Distance reward shaping does not improve training because distance to the reference state is not useful for guiding policies from farther starts.
- The added distance reward guides policy updates toward a suboptimal local optimum in the key task.
- Direct TRPO on the original maze MDP produces high variance because early batches may contain few trajectories reaching the goal.
B.3 Failure cases of Asymmetric Self-play
The asymmetric self-play baseline can become trapped because Alice’s sparse reward and unimodal policy prevent her from proposing new start states as Bob improves. In the point-mass maze, this failure leaves Bob unable to reach the goal from more than 40% of start states.
- More than 40% of point-mass maze start states remain unreachable by Bob when asymmetric self-play gets stuck in a local optimum.
- Alice’s unimodal Gaussian policy tends to converge toward a single direction or small subset of states rather than exploring all similarly rewarding states.
- Alice receives reward rA = max(0, tB − tA), favoring states that are difficult for Bob but quick for Alice to reach.Here, tA is Alice’s time from the goal and tB is Bob’s time back to the goal.
- When Bob improves faster than Alice, many proposed states satisfy tB < tA and produce zero reward, making Alice’s learning signal sparse.
- In the illustrative environment, increasing Bob’s reachable radius makes Alice’s previously optimal proposals unrewarded, so her policy remains stuck and cannot generate new proposals.The red region marks states Bob can already solve from, while blue points mark Alice’s proposed start states.
- Increasing Alice’s variance or resetting her policy would not suffice in real tasks because Alice would need to track Bob’s increasingly complex policy.