Source-linked AI summary
Automatic Curriculum Learning through Value Disagreement
Yunzhi Zhang, Pieter Abbeel, Lerrel Pinto
TL;DR
Multi-goal RL depends on selecting informative training goals, while uniform or previously reached goals can be too hard or too easy to help. The paper proposes a value-disagreement goal sampler that prioritizes goals near the policy’s solvability frontier. Across 18 sparse-reward tasks, the authors report substantial performance improvements over HER and other standard RL algorithms.
Problem
In multi-goal RL, choosing training goals affects sample efficiency, while sampling previously reached goals can repeatedly select goals too easy to provide useful training signal.
Source: §1 · p. 2Method
VDS uses epistemic uncertainty in an ensemble of value functions to prioritize goals at the policy’s learning frontier.
Source: §1 · p. 2Results
Across 18 sparse-reward tasks, the authors report substantial performance improvements over HER and other standard RL algorithms.
Source: §6 · p. 9Takeaways & Limitations
The authors report that VDS is robust to different hyperparameters and can sample goals at the frontier of the learning process.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
1 Introduction
The section frames sample-efficient multi-goal learning as a goal-selection problem: uniform or previously reached goals can be uninformative, while frontier goals offer a stronger learning signal. It proposes sampling goals using value-function uncertainty and evaluates the approach across sparse-reward tasks.
1 Introduction
- The paper situates goal curricula within a broader sample-efficiency problem, noting that learning multiple skills compounds the sample demands of model-free RL.It contrasts this with humans learning many skills from orders of magnitude fewer samples.Source: §1 · p. 1
- The introduction describes curriculum learning as an organized ordering of tasks, drawing on examples of progressively more difficult biological learning tasks.It cites grasping strategies and animal motor learning as examples.Source: §1 · p. 1
- Previously reached goals become more diverse as learning progresses, but can be too easy to provide a useful training signal.This motivates selecting goals for their informativeness rather than relying only on past successes.Source: §1 · p. 2
- Adversarial goal samplers target goals near the horizon of solvability, but the cited approaches face instability, extra sampling costs, and scalability concerns.The authors also question the scalability of setting up an explicit two-player game across problem settings.Source: §1 · p. 2
- VDS samples goals with high epistemic uncertainty in the value function, which the authors associate with goals at the policy’s solvability frontier.The method is intended to avoid goals the policy confidently rates as either easy or hard.Source: §1 · p. 2
- The authors report learning curves on 18 challenging sparse-reward tasks and improved sample efficiency over standard RL algorithms.The evaluated tasks include maze navigation, robotic manipulation, and dexterous in-hand manipulation.Source: §1 · p. 2
2 Background and Preliminaries
The background introduces goal-conditioned RL and Hindsight Experience Replay (HER), which relabels experienced transitions with goals reached later in the same episode. These ideas set up learning universal policies under sparse rewards.
2 Background and Preliminaries
- Before describing their framework, the authors discuss relevant background on goal-conditioned RL and refer readers to Sutton et al. and Kaelbling et al. for an in-depth survey.Source: §2 · p. 2
2.1 Multi-Goal RL
- Multi-goal RL trains a policy over state-goal pairs to maximize expected discounted trajectory reward, with goals sampled from a goal parameter space.The formulation can be treated as standard RL with an expanded state space S × G.Source: §2.1 · p. 3
- The paper uses sparse rewards that pay 0 when the state is within ε of the goal and −1 otherwise, and employs HER to address the resulting optimization difficulty.The authors note that sparse rewards avoid hand-engineering a reward function but are not very informative.Source: §2.1 · p. 3
2.2 Hindsight Experience Replay (HER)
- HER stores experienced transitions under additional goals corresponding to states achieved later in the episode, enabling replay with alternative goals in off-policy optimization.The approach relies on the goal not influencing environment dynamics.Source: §2.2 · p. 3
3 Method
The method samples goals from a policy-dependent proposal distribution, replacing uniform sampling with a curriculum. Value disagreement estimates which goals lie near the policy’s learning frontier, and the resulting sampler supplies goals to the policy-learning loop.
3.1 Goal Proposal Module
- The goal proposal module draws a goal at the start of each episode for the policy-conditioned trajectory.The goal distribution is denoted Cπ to make its dependence on the current policy explicit.Source: §3.1 · p. 3
- Uniform goal sampling can select mostly hard goals early and mostly easy goals later, so the proposal module is intended to adapt goal difficulty during learning.The method targets an automatic curriculum rather than a fixed ordering of goals.Source: §3.1 · p. 3, §3 · p. 3
3.2 Value disagreement
- The value-disagreement sampler assigns greater sampling weight to goals with high epistemic uncertainty, which are treated as likely frontier goals of appropriate difficulty.The method estimates disagreement across an ensemble of independently trained Q-functions using their standard deviation.Source: §3.2 · pp. 3–4
- The paper defines a learnable goal-conditioned Q-function and states that it is optimized with a Bellman update.The Q-function is intended to track policy performance.Source: §3.2 · pp. 3–4
- The algorithm samples candidate goals, forms the proposal distribution, collects a goal-conditioned trajectory, stores its data, updates the value functions, and trains the policy.The value ensemble receives transition data while the base RL algorithm is treated as a black box.Source: §3.2 · p. 4
- The experiments use DDPG as the base RL algorithm and the identity function for f; the goal sampler is described as independent of the base optimizer.The paper says ablations examine alternative choices of f.Source: §3.2 · p. 4
4 Experiments
Experiments evaluate VDS across robotic manipulation and navigation tasks, compare it with goal-sampling baselines, and examine its sampling behavior and design choices. VDS improves sample efficiency on most tested environments, samples near-frontier goals, and complements HER.
4.1 Experimental setup
- The evaluation covers 13 robotic tasks, three maze tasks, and two Ant environments; Figure 3 plots success rate over training with confidence across five seeds.Figure 2 illustrates eight of the 13 robotic environments; the maze tasks are used to investigate VDS.Source: §4.1 · p. 4, Figure 2 · p. 5, Figure 3 · p. 5
4.4 Improvements using VDS
- VDS+HER improves sample efficiency over baselines on most of 16 environments, while both VDS+HER and RandomDDPG outperform GoalGAN on two Ant environments.The Ant comparison uses fixed-length trajectories and omits time spent labeling goal difficulty, differences the authors say favor GoalGAN.Source: Figure 3 · p. 5, §4.4 · p. 6, Figure 4 · p. 6
4.2 Training details
- The training setup separates the value ensemble from policy optimization: the ensemble selects training goals, while DDPG updates the policy using on-policy data and replay buffers.The policy uses an ϵ-greedy strategy; evaluation goals are random, and success means reaching the goal at the episode’s last timestep.Source: §4.2 · p. 5
4.3 Baseline Methods
- Comparisons include HER’s uniformly sampled goals, RIAC’s learning-progress regions, Covar-GMM’s reward-time covariance, ALP-GMM’s reward-difference score, and GoalGAN’s intermediate-difficulty goals.HER uses DDPG; GoalGAN generates goals labeled as intermediate difficulty from replay-buffer rewards.Source: §4.3 · p. 6
4.5 Does VDS sample meaningful goals?
- VDS samples goals near the learning frontier, where ensemble disagreement is higher, and the sampled goals shift toward harder goals over training.The authors describe frontier goals as neither mastered nor as hard as goals far from the policy’s mastered region.Source: §4.5 · p. 7
4.6 Ablations on VDS
- VDS is insensitive to the sampling function and ensemble size; an ensemble of 10 performs the same as one of 3, and VDS improves vanilla DDPG without HER.The authors conclude that VDS complements HER and performs best when combined with it.Source: §4.6 · p. 8
5 Related Work
Related work spans curricula over training examples, tasks, and goals, as well as self-play and neural-network uncertainty estimation. VDS applies ensemble disagreement to goal-conditioned value functions for curriculum generation.
5.1 Curriculum Learning
- Prior curriculum learning spans supervised examples and control tasks, while this work applies the curriculum idea to goals that an RL algorithm must solve.The cited work includes gradually increasing training-example complexity, self-paced learning, and curricula over robotic control space.Source: §5.1 · p. 8
5.2 Self-Play based curriculums
- Reverse curricula require resetting near arbitrary states, HER can sample goals that are too easy, and self-play can be hard to formulate for arbitrary control problems.The authors report VDS gains over HER and other automatic curricula, and better performance than GoalGAN on Ant-navigation tasks.Source: §5.1 · p. 8, §5.2 · p. 8
5.3 Uncertainty estimation through neural networks
- For epistemic uncertainty, VDS uses an ensemble of neural networks inspired by Lakshminarayanan et al., adapting uncertainty estimation to its goal-conditioned setting.The cited uncertainty methods have also been applied to exploration, but the authors distinguish their problem setting.Source: §5.3 · p. 8
6 Conclusion
The conclusion reports improved performance across sparse-reward tasks and robustness to hyperparameters, while the background sections describe the off-policy RL and DDPG foundations used in the experiments.
6 Conclusion
- Across 18 sparse-reward tasks, the method substantially improves performance over HER and other standard RL algorithms, remains robust to hyperparameters, and samples goals at the learning frontier.The conclusion also suggests possible extension to real-world robotics and visual reinforcement learning.Source: §6 · p. 9
A Reinforcement learning
- Off-policy RL reuses replay-buffer experiences, improving sample efficiency in practice relative to collecting fresh on-policy data each iteration.The experiments use DDPG as the base optimizer, while the framework is compatible with standard off-policy algorithms.Source: A Reinforcement learning · p. 12
B Deep Deterministic Policy Gradients (DDPG)
- DDPG is an actor-critic algorithm with a deterministic continuous-action policy and a Q-function approximator, trained from replay-buffer episodes.The Q-function minimizes one-step Bellman error, while the policy uses the deterministic policy gradient.Source: B Deep Deterministic Policy Gradients… · p. 12
C Environments •
The evaluated environments cover Fetch-arm manipulation, dexterous-hand tasks, and blocked-maze navigation, using sparse rewards tied to reaching target states.
C Environments •
- Fetch tasks use a 7-DoF arm and 3-D goals, rewarding target proximity; evaluated tasks include reaching, pick-and-place, pushing, and sliding.The Fetch reaching observation has 10 dimensions; object tasks use 25-dimensional observations.Source: C Environments • · p. 12
- Hand environments simulate a 24-DoF robotic hand with a 20-dimensional action space; reach tasks have 15-dimensional goals and 63-dimensional observations, while manipulation tasks have 7-dimensional goals and 61-dimensional observations.Large action space, coupled complex dynamics, and sparse rewards make these tasks challenging.Source: C Environments • · p. 12
- Maze tasks use 2-D blocked arenas, target-position rewards, fixed starts, and 50-step episodes; variants include random blocks and a central region enclosed by walls.An agent receives 0 for reaching sufficiently close to the target and -1 otherwise.Source: C Environments • · p. 13
D Implementation Details and Hyperparameters
Implementation settings specify separate DDPG-based and SAC-based configurations, with HER used by default unless stated otherwise.
D Implementation Details and Hyperparameters
- DDPG-based methods use one CPU and two parallel environments; ensemble Q-functions use target networks, a 1e-3 learning rate, 0.95 Polyak coefficient, 1e6 buffer, and batch size 1000.DDPG and HER inherit official HER hyperparameters, including batch size 256 and exploration settings.Source: D Implementation Details and… · p. 13
- SAC-based methods use one CPU and one environment, with 1e-3 learning rates, a 2,000,000 replay buffer, and batch size 1000.Hyperparameters inherit from the official GMM implementation.Source: D Implementation Details and… · p. 13
- Unless stated otherwise, implementations employ HER with an expected ratio of four transitions with swapped goals to each regular transition in a batch.Source: D Implementation Details and… · p. 13
E Ablation: How important is the choice of disagreement function
The ablation reports that performance is insensitive to the choice of disagreement function, while describing how the function shapes goal sampling.
E Ablation: How important is the choice of disagreement function
- The disagreement function maps value-ensemble variance to the disagreement used by the goal proposal module, and monotonic functions of ensemble standard deviation are candidates.The experiments use ensemble variance for simplicity.Source: E Ablation: How important is the… · p. 13
- The function choice can smooth the goal distribution or concentrate more weight on a small region; a constant function reduces VDS to uniform goal sampling.Source: E Ablation: How important is the… · p. 13
- VDS is not sensitive to the tested choice of disagreement function f.The tested examples are exponential, tanh, and square functions.Source: E Ablation: How important is the… · p. 13
F Ablation: How important is the choice of ensemble size?
On maze navigation, changing the value-ensemble size causes no significant performance change; the method is also reported as insensitive to the choice of disagreement function.
F Ablation: How important is the choice of ensemble size?
- Maze performance shows no significant impact from changing ensemble size, and a few value networks perform similarly to larger ensembles.The experiments use an ensemble size of 3 for all tasks; the authors say extremely accurate epistemic-uncertainty estimates are not required to see benefits.Source: F Ablation: How important is the… · pp. 13–14, Figure 7 · p. 14
- The method is not sensitive to the choice of disagreement function f on Maze environments.The tested choices include exponential, tanh, and square functions.Source: Figure 6 · p. 14
- Applying VDS without HER performs better than vanilla DDPG.Figure 8 compares different combinations of VDS and HER on vanilla DDPG.Source: Figure 8 · p. 14
G Ablation: How does VDS perform in combination with HER?
VDS and HER can be combined, and the paper reports the best results when using them together. Their approaches to supplying learning signal are described as orthogonal and collaborative.
G Ablation: How does VDS perform in combination with HER?
- VDS combined with HER produces the best result in a majority of environments.HER alone is already competitive in the two environments shown in Figure 8, while VDS improves performance on vanilla DDPG there.Source: G Ablation: How does VDS perform in… · p. 14
- VDS and HER are described as orthogonal: HER relabels experience, while VDS uses the value ensemble to guide real-environment interaction efficiently.The paper says the two methods collaboratively strengthen the learning signal processed by the policy throughout training.Source: G Ablation: How does VDS perform in… · p. 14