PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration

Alexandre Chenu, Olivier Serris, Olivier Sigaud, Nicolas Perrin-Gilbert

arXiv:2211.04786v2cs.ROcs.AI

TL;DR

Complex robotic tasks with sparse or absent rewards are difficult to learn, and demonstrations can be costly to acquire. This paper uses a single demonstration to learn sequential goal-reaching policies that favor states compatible with subsequent goals. DCIL-II is reported to learn simulated robotic behaviors faster than state-of-the-art one-demonstration imitation-learning methods.

  • Problem

    Deep RL struggles with complex robotic tasks that provide reward only upon task completion, while acquiring demonstrations can be costly.

    Source: §I · p. 1

  • Method

    The method extracts sequential low-dimensional goals from one demonstration and uses DCIL-II to learn a goal-conditioned policy that reaches them through states compatible with subsequent goals.

    Source: §I · pp. 1–2

  • Results

    DCIL-II is reported to learn simulated complex articulated behaviors significantly faster than state-of-the-art imitation-learning methods using one demonstration.

    Source: §I · p. 2, DISCUSSION & CONCLUSION · p. 11

  • Takeaways & Limitations

    The reported results cover simulated and underactuated robots, including humanoid locomotion and stand-up and fast running with Cassie.

    Source: 5) Cassie run · p. 9, DISCUSSION & CONCLUSION · p. 11

    Applying DCIL-II to a real robot assumes it can be reset to states selected from the demonstration; learning unrelated skills to return to required states is left for future work.

    Source: DISCUSSION & CONCLUSION · p. 11

Abstract

from arXiv · show

Deep Reinforcement Learning has been successfully applied to learn robotic control. However, the corresponding algorithms struggle when applied to problems where the agent is only rewarded after achieving a complex task. In this context, using demonstrations can significantly speed up the learning process, but demonstrations can be costly to acquire. In this paper, we propose to leverage a sequential bias to learn control policies for complex robotic tasks using a single demonstration. To do so, our method learns a goal-conditioned policy to control a system between successive low-dimensional goals. This sequential goal-reaching approach raises a problem of compatibility between successive goals: we need to ensure that the state resulting from reaching a goal is compatible with the achievement of the following goals. To tackle this problem, we present a new algorithm called DCIL-II. We show that DCIL-II can solve with unprecedented sample efficiency some challenging simulated tasks such as humanoid locomotion and stand-up as well as fast running with a simulated Cassie robot. Our method leveraging sequentiality is a step towards the resolution of complex robotic tasks under minimal specification effort, a key feature for the next generation of autonomous robots.

I. INTRODUCTION

Sparse-reward robotic tasks are difficult to specify and learn, while acquiring demonstrations can be costly. The paper uses one state-based demonstration to define sequential low-dimensional goals and learns to reach each through states compatible with later goals.

I. INTRODUCTION

  • Sparse rewards make complex robotic behaviors difficult for DRL to learn from task success alone.The introduction gives standing up an underactuated humanoid as an example.Source: §I · p. 1
  • Imitation learning can accelerate policy optimization, but typically requires many demonstrations containing states and actions, which can be prohibitively difficult to obtain.Some sources, including motion capture and human guidance, provide state trajectories without actions.Source: §I · p. 1
  • The proposed approach converts one demonstration into sequential low-dimensional goals and learns a goal-conditioned policy to reach them.For humanoid stand-up, the goals can be successive torso positions from the ground to a desired height.Source: §I · pp. 1–2
  • Because a low-dimensional goal may leave parts of the state unspecified, DCIL-II targets success states compatible with reaching the next goal.In the Dubins-car example, a goal specifies position but not orientation.Source: Figure 1 · p. 1, §I · p. 2

A. (Goal-conditioned) Reinforcement Learning

  • Goal-conditioned RL conditions the policy and reward on a goal, and reaching that goal cancels subsequent rewards by making it terminal.The goal-conditioned discount function becomes zero when the goal is reached.Source: A. (Goal-conditioned) Reinforcement… · p. 2

B. Distance-based sparse reward

The method turns a sequence of low-dimensional goals from a demonstration into a sequential goal-reaching problem. It uses an extended state with a current-goal index, propagates value through valid success states, and uses relabeling to support learning from sparse rewards.

B. Distance-based sparse reward

  • Low-dimensional goals can have many success states, some of which are invalid because they do not permit reaching the next goal.The framework distinguishes valid from invalid success states for each goal.Source: B. Distance-based sparse reward · pp. 2–3

C. Relabelling

III. METHODS

  • The proposed framework starts from a sequence of low-dimensional goals extracted from a single demonstration, and DCIL-II implements it.The agent must learn to reach the goals sequentially to perform the complex task.Source: §III · p. 3

A. Problem statement

1) An MDP with implicit sequential goals:

  • The extended state pairs the underlying state with the current-goal index; reaching a goal advances that index, and intermediate goals remain non-terminal.This structure allows value from a later goal to propagate to valid success states for earlier goals.Source: 1) An MDP with implicit sequential goals · p. 3

2) Moving to GCRL to improve exploration:

The section extends goal-conditioned reinforcement learning with index-conditioned sequential goals and HER-style relabeling, enabling value propagation between successive goals. The index preserves which goal in the sequence is active during this propagation.

B. The DCIL-II algorithm

DCIL-II extracts a goal sequence, derives an alternative GC-MDP, and learns a policy through repeated trajectory rollouts. Its trajectories select an indexed goal, reset from the demonstration, and record transitions as the goal is reached or missed.

  • DCIL-II extracts goals from the demonstration, derives the alternative GC-MDP, and repeats a two-step learning loop to obtain a policy for sequential goal reaching.Source: B. The DCIL-II algorithm · p. 5
  • Each trajectory selects a goal index and resets the environment using that index before rollout begins.Source: B. The DCIL-II algorithm · p. 5
  • During rollout, the policy uses state, index, and goal; reaching a goal gives reward 1 and increments the index, while failure keeps the index unchanged.Source: B. The DCIL-II algorithm · p. 5
  • The algorithm stores each transition together with current and next goals and indices, reward, termination, and success information.Source: B. The DCIL-II algorithm · p. 5

1) From a single demonstration to a sequence of goals:

DCIL-II converts one demonstration into a sequence of low-dimensional goals and retains demonstrated states associated with the starts of its goal sub-trajectories.

2) Main Loop:

DCIL-II combines an off-policy actor-critic learner with HER-like relabeling in a repeated trajectory loop. It initializes trajectories from demonstrated states, conditions the policy on the indexed goal, and biases training toward goals with lower success ratios.

  • DCIL-II combines an off-policy actor-critic algorithm, such as SAC, with HER-like relabeling to learn the goal-conditioned policy.Source: 2) Main Loop · p. 5
  • For each trajectory, the algorithm selects an index and resets the agent in its associated demonstrated state.Source: 2) Main Loop · p. 5
  • The policy is conditioned on state, index, and associated goal; a trajectory ends when the sequence is completed, a time limit is reached, or a terminal state occurs.Source: 2) Main Loop · pp. 5–6
  • Index selection can favor goals with lower success ratios using an inverse-proportional discrete distribution.Source: 2) Main Loop · p. 6
  • After each environment step, actor-critic updates use saved transitions, with half of each sampled batch relabeled.Source: 2) Main Loop · p. 6

IV. RELATED WORK

The related work positions DCIL-II at the intersection of learning from a single state-based demonstration, sequential goal reaching, and value propagation. Prior approaches address parts of this combination, but differ in their assumptions and mechanisms.

A. Learning from a single demonstration

  • PWIL and Backplay learn from a single demonstration, while BC and IRL methods fail to imitate complex behaviors from only one demonstration.Backplay can suffer catastrophic forgetting; PWIL does not use sequential goal reaching.Source: A. Learning from a single demonstration · p. 6

B. Sequential goal reaching

  • Feudal HRL constructs goals on the fly, while value-based subgoal methods rely on accurate value functions or broad exploration and reset assumptions.DCIL-II instead targets hard-exploration settings with sparse rewards and resets only along the demonstrated trajectory.Source: B. Sequential goal reaching · p. 6
  • Go-Explore uses trajectory-derived goals to return to visited states, but does not address invalid success states between successive goals.DCIL-II addresses this issue using value propagation.Source: B. Sequential goal reaching · p. 6

C. Value propagation mechanisms

  • Option-based HRL and R-DSC propagate value between options, but these methods do not learn from demonstrations or combine the same goal-conditioned demonstration framework.R-DSC propagates option values to select goals for chaining successive options.Source: C. Value propagation mechanisms · p. 6
  • DCIL-I is the closest predecessor: it also uses one demonstration and sequential goals, with reward bonuses and overshooting to propagate value to the preceding goal.Its approach also targets valid success states.Source: C. Value propagation mechanisms · p. 6

D. Bridging the gap between DCIL-I & DCIL-II

DCIL-I propagates value through a sequence of MDPs, whereas DCIL-II uses an extended-state GCRL framework. The comparison connects DCIL-II’s relabeling and value propagation to a hand-designed chain of options.

D. Bridging the gap between DCIL-I & DCIL-II

  • DCIL-I uses a sequence of MDPs that differ only in reward and discount functions, with success states made terminal.A reward bonus based on the next MDP’s value propagates value backward and encourages valid success states compatible with later goals.Source: D. Bridging the gap between DCIL-I &… · p. 7
  • With HER, DCIL-I propagates its bonus only at demonstration-sequence goals, while DCIL-II’s extended state and modified relabeling prepare for next goals at sequence and relabeled goals.DCIL-I avoids extending the state with the current goal index, but its bonus does not apply to artificial relabeled goals.Source: D. Bridging the gap between DCIL-I &… · p. 7

E. Bridging the gap between DCIL-II & option-based HRL

  • Mseq can be viewed as a semi-MDP whose hand-designed options form a chain, with each goal’s success states terminating one option and initiating the next.The policy is conditioned on goal indices, and the chain switches deterministically to the next option.Source: E. Bridging the gap between DCIL-II &… · p. 7
  • DCIL-II extends Mseq to goal-conditioned reinforcement learning, combining value propagation and exploration mechanisms at the cost of hand-designing options.Option-Critic’s two-stage learning can be unstable, and without explicit goal conditioning its framework cannot use relabeling to aid sparse-goal exploration.Source: E. Bridging the gap between DCIL-II &… · p. 7

F. Bridging the gap between DCIL-II & goal-conditionned skill-chaining

DCIL-II is contrasted with goal-conditioned skill chaining through how each method propagates value to valid success states. The comparison also states the reset and demonstration assumptions underlying DCIL-II.

F. Bridging the gap between DCIL-II & goal-conditionned skill-chaining

  • R-DSC selects valued goals from learned option termination sets, but that value propagation does not shape intra-option policies, so changing policies can break the option chain.DCIL-II instead increases the value of valid success states, encouraging the agent to reach goals through those states.Source: F. Bridging the gap between DCIL-II &… · pp. 7–8
  • DCIL-II raises values of valid success states for both sequence and relabeled goals, and the Dubins car comparison shows broader propagation than DCIL-I.The figure reports value around 2 for DCIL-II’s relabeled goals, compared with around 1 for DCIL-I’s relabeled goals in the sequence.Source: Figure 4 · p. 8

G. Comparison of hypotheses

  • DCIL-II assumes resets are possible at some demonstrated states, unlike reset-anywhere methods, which assume uniformly sampled reset states.PWIL assumes a unique reset, while behavioral cloning requires no reset.Source: G. Comparison of hypotheses · p. 8, 1) Reset · p. 8

1) Reset:

  • DCIL-II learns from one state-based demonstration without expert actions, which matters when actions from sources such as motion capture or human guidance are difficult to collect.The method also assumes a defined goal space for goal-conditioned reinforcement learning.Source: G. Comparison of hypotheses · p. 8, 3) Goal-space definition · p. 8

2) State-only demonstration:

The experiments evaluate five environments with varying complexity, using tasks that require explicit goal-space definitions and state-to-goal mappings for GCRL methods. Dubins Maze and Fetch illustrate how low-dimensional goals can leave important state dimensions unspecified.

3) Goal-space definition:

  • DCIL-II and other GCRL methods require a defined goal space and state-to-goal mapping; Table I compares methods’ problems, mechanisms, and hypotheses.The table covers DCIL-II, DCIL-I, PWIL, BC, R-DSC, and Option-Critic.Source: 3) Goal-space definition · p. 8, Table I · p. 9

V. EXPERIMENTS

  • The experiments assess DCIL-II in five environments spanning different levels of complexity.These include Dubins Maze, Fetch, two Humanoid variants, and Cassie Run.Source: §V · p. 8, A. Environments · p. 8

1) Dubins Maze:

  • In Dubins Maze, the agent controls a Dubins car in a 2D maze, with goals defined by position rather than orientation.The car’s state includes position and orientation, but the goal space is X × Y.Source: 1) Dubins Maze · p. 8

2) Fetch:

  • Fetch is a simulated grasping task with sparse reward for placing an object on a shelf, while its goals specify the end-effector and object positions.The 604-dimensional state includes positions, velocities, and contact indicators; reaching a goal with an unsuitable orientation or velocity may prevent grasping.Source: 2) Fetch · pp. 8–9

A. Environments

  • The Humanoid locomotion variant adds torso x,y positions to the state, uses a 200-step evaluation budget, and divides demonstrations into 15 torso-position goals.The agent is rewarded only for reaching desired torso positions.Source: 3) Humanoid locomotion · p. 9

3) Humanoid locomotion:

The experiments include Humanoid locomotion and stand-up tasks and a simulated Cassie running task, comparing DCIL-II with baselines and ablated variants. In Dubins Maze and Fetch, DCIL-II learns faster than the tested variants.

3) Humanoid locomotion:

  • The Humanoid stand-up task starts with the agent lying on its back, unlike the related locomotion task, which starts standing.The stand-up demonstration reaches a torso height above 1.4 meters and is divided into 12 goals.Source: 4) Humanoid stand-up · p. 9

5) Cassie run:

  • The Cassie Run task uses low-dimensional pelvis-position goals and a demonstration trained for 18 million steps with TQC.The simulated robot’s state includes the positions and orientations of its parts.Source: 5) Cassie run · p. 9

B. Baselines

  • The baselines are random policy initialization, single-demonstration behavior cloning, PWIL, and DCIL-I.Source: B. Baselines · p. 9

C. Implementation details

D. Ablation study

  • DCIL without goal conditioning lacks relabeling for exploration, while DCIL without index lacks reward propagation to encourage valid success states.DCIL-I propagates value only for original successful transitions; DCIL-II also propagates value through relabeled successful transitions.Source: D. Ablation study · p. 10
  • DCIL-II learns to navigate the entire Dubins Maze and solve Fetch faster than the tested variants.The comparison concerns DCIL-I, DCIL without index, and DCIL without goal conditioning.Source: D. Ablation study · p. 10, Figure 5 · p. 10
  • Figure 6 compares maximum returns during training for DCIL-II, random policy, BC, PWIL, and DCIL-I in Humanoid locomotion and stand-up.Source: Figure 6 · p. 10

E. Comparison to baselines

DCIL-II outperforms baselines on humanoid locomotion and stand-up, and sequential pelvis-position goals yield a 15-meter Cassie run with stable pelvis height despite feet trajectories differing from the demonstration. Its real-robot use is limited by the assumption that the agent can be reset to states selected from the demonstration.

E. Comparison to baselines

  • In locomotion, DCIL-II alone exceeds average maximum return 1100 within 2M training steps; in stand-up, DCIL-I and DCIL-II reach around 20000, with DCIL-II faster.The stand-up returns correspond to an agent standing up.Source: E. Comparison to baselines · p. 10, Figure 8 · p. 11
  • In stand-up, BC performs surprisingly well because early actions largely condition the trajectory, reducing the impact of distribution shift.Source: E. Comparison to baselines · p. 10

F. Using DCIL-II to control a simulated Cassie bipedal platform

DISCUSSION & CONCLUSION

  • DCIL-II learns complex articulated behaviors significantly faster than state-of-the-art imitation-learning methods across simulated and underactuated robots.The framework encourages reaching each low-dimensional goal through states compatible with subsequent goals.Source: DISCUSSION & CONCLUSION · p. 11
  • Real-robot use assumes resets to states selected from the demonstration; autonomous return would require learning a large set of unrelated skills.Source: DISCUSSION & CONCLUSION · p. 11
Loading 2211.04786v2…