Exploration via Hindsight Goal Generation

1 Introduction

• Sparse or episodic rewards slow learning, and engineering dense rewards for real-world tasks is non-trivial.With reward only at the end of a trajectory of length L, Q-function updates require at least L steps to propagate the return.Source: §1 · pp. 1–2
• HER replays previously visited states as achieved hindsight goals, creating an implicit curriculum from easier goals toward the final goal.Unlike explicit curriculum learning, HER does not require an explicit distribution of initial environment states.Source: §1 · p. 2
• HGG generates goals that are easy to achieve in the short term and likely to lead to the actual goal in the long term.It lower-bounds the actual goal distribution's value using a hindsight-goal value function and Wasserstein distance, then uses a discrete Wasserstein Barycenter solver to optimize that bound.Source: §1 · p. 2
• HGG significantly improves sample efficiency over DDPG+HER on a broad set of robotic manipulation tasks.Ablations report that its exploration strategy is robust across a wide range of hyperparameters.Source: §1 · p. 2

2 Background

• An MDP is specified by states, actions, transitions, rewards, and a discount factor; the agent seeks a policy maximizing expected cumulative reward.The starting state may be given or drawn from an initial-state distribution.Source: §2 · p. 2
• In a goal-oriented MDP, the reward is a sparse binary signal conditioned on whether the goal is achieved.The goal representation uses a tractable mapping from state to goal space, with a threshold for deciding whether a goal is reached.Source: §2 · pp. 2–3
• A universal value function represents values for many goals with one function approximator, defining the value for each state and goal pair.The task distribution jointly draws an initial state and a goal at the start of each episode.Source: §2 · p. 3
• FetchPush illustrates the goal setup: the goal specifies the box position, which is extracted from the system state by the state-to-goal mapping.The robot and box status are included in the state.Source: §2 · p. 3
• Unlike some prior work, HGG does not require a universal simulator capable of resetting the environment to any given state.The paper describes this as making HGG more realizable.Source: §2 · p. 3

3 Related Work

• Goal-conditioned policies support multiple uses, including learning several tasks simultaneously and identifying important decision states.Prior work also reports improved generalization to unseen goals from multi-task learning with goal-conditioned policies.Source: §3 · p. 3
• HER generates reward signals from failure trajectories and has been extended to hierarchical RL, dynamic goal pursuit, imitation, visual robotics, and on-policy RL.The listed extensions combine HER with importance sampling in the on-policy setting.Source: §3 · p. 3
• Curriculum-learning research schedules auxiliary tasks or intrinsic motivation, and guided exploration manipulates initial states or goal positions to create intermediate tasks.The section notes that most curriculum-learning methods are domain-specific and a generalized framework remains open.Source: §3 · pp. 3–4

4 Automatic Hindsight Goal Generation

• HGG addresses the mismatch between HER hindsight goals and distant target goals by seeking intermediate goals that are easier to achieve and may improve performance on the original task.The paper illustrates the problem with a target distribution far from the initial position, where HER goals may not promote optimization effectively.Source: §4 · p. 4

4.1 Algorithmic Framework

• Under a local Lipschitz generalizability assumption, the value for a target task distribution is bounded below by the value for a surrogate distribution minus L times their Wasserstein distance.The paper notes that this continuity condition is required only over a specific region and is considered reasonable when the paired state-goal distance is not too large.Source: §4.1 · p. 4
• The surrogate objective balances policy value on a goal set against distance to the target distribution, while limiting selected goals per replay-buffer trajectory to encourage diversity.The paper describes joint optimization as difficult because the goal distribution is high-dimensional and shifting, and value estimates may be inaccurate during training.Source: §4.1 · p. 5

4.2 Solving Wasserstein Barycenter Problem via Bipartite Matching

• HGG alternates policy optimization using DDPG+HER with hindsight-goal optimization, approximating the latter through bipartite matching between sampled target tasks and replay-buffer trajectories.Selected achieved states from matched trajectories supply the hindsight goals; the paper uses minimum-cost maximum-flow to solve the matching problem and treats the unknown Lipschitz constant as a hyperparameter.Source: §4.1 · p. 5, §4.2 · pp. 5–6

5 Experiments

• The benchmark varied Fetch initial positions and goals, hand-object rotations, and reach-task goal distributions, comparing DDPG+HER with prioritized variants.FetchReach and HandReach do not support initial-state randomization.Source: §5 · pp. 6–7

5.1 HGG Generates Better Hindsight Goals for Exploration

• On FetchPush, HGG goals gradually moved toward the target region, while HER replay goals remained near the initial position for many iterations.Figure 2 distinguishes recent and earlier HGG goals from HER replay goals.Source: §5.1 · p. 7, Figure 2 · p. 7

Performance on benchmark robotics tasks

• HGG substantially outperformed HER on four of eight environments and matched it on four; HGG+EBP performed best on six eligible environments.FetchReach and HandReach were not eligible for EBP variants.Source: Figure 3 · p. 7, Performance on benchmark robotics tasks · p. 8
• With an obstacle, the crafted graph-distance metric provided better goal-generation guidance and significantly improved sample efficiency over ℓ2 distance.The paper identifies learning or obtaining a suitable metric as future work.Source: Performance on benchmark robotics tasks · p. 8, Figure 4 · p. 8

5.2 Comparison with Explicit Curriculum Learning

• HGG substantially outperformed HER even when HER+GOID sampled goals uniformly from an oracle GOID generator.This comparison used a fixed initial state, and the threshold choice followed Florensa et al. (2018).Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 Ablation Studies on Hyperparameter Selection

• Across tested tasks, HGG was robust to Lipschitz L when it was not too small and to distance weight c; the selected values were L = 5.0 and c = 3.0.c trades off initial-state similarity against goal similarity; larger c favors a closer initial state.Source: §5.3 · pp. 8–9
• K was not critical on simple tasks, where K = 1 could be competitive, while larger batches reduced training-result variance on difficult FetchPickAndPlace.The reported comparisons plotted the best results for K ∈ {50, 100}.Source: §5.3 · p. 9, Figure 6 · p. 9

• HGG achieves better goal-oriented exploration than original HER and curriculum learning across a collection of robotic learning tasks.Source: §6 · p. 9
• The method formulates hindsight-goal selection as surrogate optimization and uses a combinatorial solver to generate intermediate tasks.Source: §6 · p. 9
• The authors identify task-specific distance metrics as a future direction for cases where standard Wasserstein distance cannot be applied directly.They suggest controllable representation learning as a route to such metrics.Source: §6 · p. 9

A Proof of Theorem 1

• The proof applies the pointwise value bound under any coupling, takes expectations, and then uses the infimum over couplings.Source: A Proof of Theorem 1 · p. 13
• Theorem 1 bounds V^π(T′) below by V^π(T) − L · D(T, T′), assuming the stated generalizability condition.Source: A Proof of Theorem 1 · p. 13
• D(T, T′) is the Wasserstein distance, defined through the infimum expected distance over joint distributions with marginals T and T′.Source: A Proof of Theorem 1 · p. 13

B.1 Modified Environments

• FetchPush, FetchPickAndPlace, and FetchSlide sample object starts and goals uniformly along task-specific segments.Source: §B.1 · p. 13
• Block and egg rotation tasks sample initial and target z-axis rotations in specified intervals; the pen task retains its original setting.Source: §B.1 · p. 14
• FetchReach samples goals along a segment, HandReach offsets one randomly selected meeting-point dimension by 0.005, and other environment attributes remain at default.Source: §B.1 · p. 14

B.2 Evaluation Details

• Curves are plotted from 10 runs with random task initializations and seeds.Source: §B.2 · p. 14
• The shaded region indicates 60% of the population around the median.Source: §B.2 · p. 14
• Curves use the same hyperparameters except in the ablation section.Source: §B.2 · p. 14
• An episode is successful when the final object position is within δg of the goal, using the reward-function threshold.Source: §B.2 · p. 14

B.3 Details of Experiment with obstacle

• In the obstacle experiment, the custom distance sums the hindsight goal’s ℓ2 distance to the blue path and the distance along that path to the desired goal.The two terms use the ratio specified in Eq. (5).Source: §B.3 · p. 14

B.4 Details of Experiment 5.2

• In the deterministic setting, success rate f(π, g) uses a ball centered at g with radius δg, the same threshold as the reward and success test.Source: §B.4 · p. 14
• The average success-rate oracle is estimated using 102 samples.Source: §B.4 · p. 14
• Figure 8 shows the initial object position and the blue region from which goals are uniformly generated.Source: Figure 8 · p. 15

C.1 Hyper-Parameters

• Training follows benchmark DDPG/HER settings with 20 replay batches per trajectory, HER replay probability 0.8, random-action probability 0.3, and Gaussian noise scale 0.2.Source: §C.1 · p. 15
• The actor and critic each use three 256-unit ReLU layers, with Polyak coefficient 0.95 and action L2 penalty coefficient 1.0.Source: §C.1 · p. 15
• Matching uses Lipschitz constant 5.0, distance weight 3.0, 50 or 100 hindsight goals, one MPI worker, and a 104-trajectory buffer.Source: §C.1 · p. 15

C.2 Details on Data Processing

• Policy-training minibatches use HER; matching uses L*=L(1−γ)dmax, and the buffer is approximated by a FIFO queue of 103 recent trajectories.HGG goals receive N(0, 0.05I) noise in Fetch environments, but not Hand environments because their goal space is not Rd.Source: §C.2 · p. 15

D.1 Additional Visualization of Hindsight Goals Generated by HGG

• HGG’s generated goals progress from locating the object, to moving it beneath the target, and finally picking it up.For HandManipulation tasks, the paper plots distances between exploratory and desired goals and reports similar learning dynamics across experiments.Source: §D.1 · p. 16

D.2 Evaluation on Standard Tasks

• Figure 10 presents learning curves for HGG and HER on the standard task distribution created by Andrychowicz et al. (2017).Source: Figure 10 · p. 16, §D.2 · p. 16

D.3 Additional Experiment Results on Section 5.2

• On FetchPickAndPlace, the HGG–explicit-curriculum comparison is reported to generalize the Section 5.2 result to a different environment.Figure 11 shows the object’s initial position and a blue goal region following the default distribution.Source: §D.3 · p. 16, Figure 11 · p. 17

D.4 Ablation Study

• Figure 12 provides the full version of the ablation study.Source: §D.4 · p. 17, Figure 12 · p. 17