Source-linked AI summary
Goal-conditioned Imitation Learning
Yiming Ding, Carlos Florensa, Mariano Phielipp, Pieter Abbeel
TL;DR
Reward design and exploration make multi-goal reinforcement learning difficult in real-world robotics, where reward instrumentation is costly and HER can learn bottlenecked regions slowly. The paper combines demonstrations, hindsight relabeling, and GAIL in goalGAIL, which converges faster than HER, achieves better final performance than naive goal-conditioned GAIL, and can use state-only demonstrations. These results broaden imitation learning toward practical robotics settings, although the reported setup includes fewer than 20 demonstrations where Behavioral Cloning fails completely.
Problem
Reward design for multi-goal robotic reinforcement learning is costly, while HER may require many samples to discover state-space regions separated by exploration bottlenecks.
Method
The paper combines a few demonstrations with hindsight relabeling and GAIL in goalGAIL, including expert relabeling and demonstrations without expert actions.
Results
goalGAIL converges faster than HER, reaches better final performance than naive goal-conditioned GAIL, and improves with expert relabeling across the evaluated environments.
Takeaways & Limitations
The method supports goal-conditioned imitation learning from state demonstrations alone, broadening its potential use with kinesthetic, third-person, or planner-generated demonstrations.
Takeaways & Limitations
Behavioral Cloning fails completely in the reported low-data regime of fewer than 20 demonstrations.
Abstract
from arXiv · showhide
Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions, which can leverage kinesthetic or third person demonstration. The code is available at https://sites.google.com/view/goalconditioned-il/.
1 Introduction
The paper addresses reward and exploration difficulties in goal-conditioned robotics by combining hindsight relabeling with demonstrations. It proposes goalGAIL to accelerate learning across goals, including from demonstrations without expert actions.
- Real-world robotics makes reward access difficult because reproducing simulation learning environments and instrumentation is challenging.
- Goal-conditioned policies seek to reach any observed state using an indicator reward requiring only the robot’s existing sensors.In continuous robotics spaces, exact repetition of sensory inputs is extremely rare, so the reward is typically not directly observed.
- HER relabels collected trajectories with visited states as goals, producing indicator rewards without requiring a separate reward function.The approach can observe successful relabeled transitions frequently, although earlier variants used special resets or approximate goal rewards.
- HER can learn goals slowly when exploration must cross bottlenecks, such as grasping objects or navigating narrow corridors.Random motion may fail to traverse these bottlenecks efficiently, delaying access to new regions of the goal space.
- The work extends imitation learning to goal-conditioned tasks, proposes goalGAIL, and studies demonstration relabeling to improve convergence without additional rewards.The method also supports demonstrations lacking expert actions and investigates robustness to sub-optimal experts.
2 Related Work
Prior imitation-learning methods commonly follow demonstrations or solve single tasks, while goal-conditioned learning must reach many goals efficiently. The paper combines demonstrations with HER and introduces goalGAIL to address this exploration challenge.
- Imitation learning offers an alternative to reward crafting, but prior work often centers on trajectory following or single-task behavior.
- Many imitation-learning methods cannot outperform the expert without an additional training reward and may not adapt to unseen situations.
- Goal-conditioned tasks require reaching any state upon demand and can be data-hungry when reward shaping is unavailable.HER enables learning in sparse-reward settings, but its breadth-first exploration can make complex policies inefficient to learn.
- The paper investigates a few demonstrations as a way to improve HER’s convergence and proposes goalGAIL using GAIL with off-policy algorithms.
3 Preliminaries
The paper formulates goal-conditioned reinforcement learning in a discounted finite-horizon MDP and combines hindsight relabeling with GAIL-style imitation. Expert trajectories provide additional goal-reaching supervision, including when actions are available for discrimination.
- The underlying environment is modeled as a finite-horizon discounted MDP with states, actions, transitions, rewards, an initial-state distribution, discount factor, and horizon.
- In the goal-conditioned formulation, both policy and reward depend on a goal, with reward 1 when the reached state matches that goal.
- Because goals do not affect transition dynamics, a transition can be relabeled with its next state as the goal and assigned a successful reward.
- The method assumes expert trajectories for sampled feasible goals, with approximately geodesic actions that reach goals as quickly as possible.
- GAIL trains a discriminator to distinguish expert from agent state-action transitions while training the agent to fool it using discriminator output as reward.The paper exploits later work showing that off-policy optimization can retain stability while improving efficiency.
4 Demonstrations in Goal-conditioned tasks
The paper incorporates demonstrations into Hindsight Experience Replay for goal-conditioned policies through Behavioral Cloning, expert relabeling, and goal-conditioned GAIL, including a state-only variant. These methods combine demonstration information with off-policy RL and relabeling to improve learning toward diverse goals.
- Overview: The paper extends demonstration-based learning to goal-conditioned tasks by combining Behavioral Cloning, expert relabeling, and goal-conditioned GAIL with Hindsight Experience Replay.The proposed methods are intended to train policies that can reach different goals rather than merely follow a single expert trajectory.
- 4.1 Goal-conditioned Behavioral Cloning: Behavioral Cloning trains a deterministic goal-conditioned policy by supervised regression on expert state-action-goal tuples.Its loss and gradient can be computed without additional samples from the current policy, allowing combination with off-policy policy updates.
- 4.1 Goal-conditioned Behavioral Cloning: Annealing the Behavioral Cloning contribution or applying a Q-filter allows the agent to eventually outperform the expert.Combining Behavioral Cloning with deterministic policy-gradient updates otherwise loses the improvement guarantees associated with the task reward.
- 4.2 Relabeling the expert: Expert relabeling treats a demonstration transition as valid for future states visited in that demonstration, augmenting training data under a geodesic-trajectory assumption.The method is motivated as especially useful when few demonstrations are available.
- 4.2 Relabeling the expert: In four rooms, standard Behavioral Cloning fails to reach many intermediate demonstration states, whereas Figure 1 compares it with expert relabeling across goals.The figure reports policy performance for the two training conditions using top and bottom rows.
- 4.3 Goal-conditioned GAIL with Hindsight: GoalGAIL conditions the discriminator on the goal, uses its log output with an indicator reward in off-policy DDPG, and anneals the GAIL weight.The algorithm is formalized as goal-conditioned GAIL with Hindsight and includes agent and expert transition relabeling.
- 4.3 Goal-conditioned GAIL with Hindsight: The algorithm samples goal-conditioned rollouts, relabels agent transitions with future states using HER, and relabels expert transitions with states visited later in demonstrations.These steps are listed in Algorithm 1 alongside replay-buffer sampling and policy updates.
- State-only demonstrations: State-only demonstrations can support the method by replacing expert actions in the GAIL formulation with the next state.This is intended to make kinesthetic or third-person demonstrations usable when actions are unavailable.
5 Experiments
The experiments evaluate goalGAIL, Expert Relabeling, robustness to sub-optimal experts, and state-only demonstrations across four continuous goal-conditioned environments. Results show faster or stronger learning than relevant baselines, including when demonstrations lack actions.
- Experimental setup: The study evaluates four continuous goal-conditioned MuJoCo environments using 20 demonstrations, five random seeds, and coverage as the percentage of feasible goals reached.Goals are sampled uniformly, and success requires reaching within ϵ of the desired goal.
- Experimental setup: The four tasks comprise Four Rooms, Pointmass Block Pusher, Fetch Pick and Place, and Fetch Stack Two, with goal spaces ranging from two to six dimensions.The environments include bottlenecked navigation, block manipulation, and stacking tasks.
- Goal-conditioned learning: Across all four environments, goalGAIL converges faster than HER and surpasses standard GAIL, whose performance is capped by the demonstrations.HER eventually reaches comparable final performance but learns slowly, while GAIL learns quickly initially without continuing to generalize beyond demonstrated goals.
- Expert relabeling: Expert Relabeling boosts standard BC's final performance and accelerates BC+HER and goalGAIL convergence across all four environments.Without relabeling, the agent fails to learn many intermediate states visited in demonstrations; BC+HER also learns faster than HER alone and exceeds pure BC's final performance with 20 demonstrations.
- Robustness to sub-optimal experts: Under sub-optimal demonstrations, Behavioral Cloning barely improves over plain HER, whereas discriminator-based methods can leverage substantially noisier experts.The sub-optimal demonstrations use ε-greedy behavior and Gaussian action noise; discriminator methods can reward transitions moving in the right direction without copying one action exactly.
- Using state-only demonstrations: GoalGAIL uses state-only demonstrations without expert actions and outperforms BC+HER in all environments except Fetch Pick and Place.State-conditioned discrimination produces a shaped reward encouraging movement toward the goal.
6 Conclusions and Future Work
The paper presents goalGAIL and expert relabeling as ways to accelerate reward-free goal-conditioned learning, including from state-only demonstrations. It reports robustness to sub-optimal experts, while identifying vision-based low-data learning as future work.
- goalGAIL converges faster than HER and achieves better final performance than naive goal-conditioned GAIL.
- Expert relabeling improves both goalGAIL and goal-conditioned Behavioral Cloning.
- goalGAIL requires only state demonstrations, without expert actions used by other Behavioral Cloning methods.
- goalGAIL remains robust to sub-optimal expert behavior.
- Behavioral Cloning fails completely in the reported low-data vision regime of fewer than 20 demonstrations.
A Hyperparameters and Architectures
The experiments use four environments with specified horizons, discounting, neural-network architectures, and fixed optimization settings. Sub-optimal-expert experiments additionally specify environment-specific noise parameters.
- The four environments use task horizons of 300, 100, 100, and 150, respectively.The environments are Four Rooms, Fetch Pick & Place, Pointmass block pusher, and Fetch Stack Two.
- The Q function, policy, and discriminator use fully connected neural networks with two hidden layers of size 256.DDPG performs policy optimization, and hindsight probability is p = 0.8.
- The initial behavior-cloning loss weight is β = 0.1 and is annealed by 0.9 per 250 collected rollouts.The initial discriminator reward weight is δGAIL = 0.1 and is not annealed.
- Sub-optimal-expert experiments use environment-specific ϵ and σα values.The passage lists ϵ as 0.4, 0.5, 0.4, 0.1 and σα as 1.5, 0.3, 0.2, 0 for the four environments.
B Effect of Different Input of Discriminator
The discriminator study compares three input configurations for goalGAIL across four environments. The configurations vary whether the discriminator receives the current state with the goal, the next state, or the action.
- The study compares discriminator inputs across four environments using three configurations.
- The first configuration uses the current state and goal, (s, g).
- The second configuration uses the current state, next state, and goal, (s, s′, g).
- The third configuration uses the current state, action, and goal, (s, a, g).