PaperlayerView exact revision on arXiv

Source-linked AI summary

Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention

Abhishek Gupta, Justin Yu, Tony Z. Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, Sergey Levine

arXiv:2104.11203v1cs.LGcs.RO

TL;DR

Real-world reinforcement learning for dexterous manipulation is challenging when data collection depends on instrumentation and human-provided resets. This paper sequences multiple learned tasks so they can reset one another, and reports reset-free learning in simulation and on hardware, including in-hand reorientation learned in 60 hours.

  • Problem

    Real-world robotic RL has often relied on instrumentation and episodic resets, and robust reset behaviors have been difficult to learn for dexterous manipulation.

    Source: §I · p. 1

  • Method

    MTRF jointly trains task-specific policies with off-policy RL while a task graph sequences them during continuous data collection, allowing tasks to provide resets for one another.

    Source: §I · p. 2, A. Algorithm Description · p. 3

  • Results

    MTRF learned all simulation tasks jointly where reset and perturbation controllers struggled on complex tasks, and learned real-world in-hand reorientation to a 70% success rate in 60 hours.

    Source: B. Reset-Free Learning Comparisons in… · p. 6, C. Learning Real-World Dexterous… · p. 7

  • Takeaways & Limitations

    The approach learned real-world in-hand reorientation, including pickup and repositioning, in about 60 hours and pipe insertion in around 25 hours without human intervention during training.

    Source: §VII · p. 8

Abstract

from arXiv · show

Reinforcement Learning (RL) algorithms can in principle acquire complex robotic skills by learning from large amounts of data in the real world, collected via trial and error. However, most RL algorithms use a carefully engineered setup in order to collect data, requiring human supervision and intervention to provide episodic resets. This is particularly evident in challenging robotics problems, such as dexterous manipulation. To make data collection scalable, such applications require reset-free algorithms that are able to learn autonomously, without explicit instrumentation or human intervention. Most prior work in this area handles single-task learning. However, we might also want robots that can perform large repertoires of skills. At first, this would appear to only make the problem harder. However, the key observation we make in this work is that an appropriately chosen multi-task RL setting actually alleviates the reset-free learning challenge, with minimal additional machinery required. In effect, solving a multi-task problem can directly solve the reset-free problem since different combinations of tasks can serve to perform resets for other tasks. By learning multiple tasks together and appropriately sequencing them, we can effectively learn all of the tasks together reset-free. This type of multi-task learning can effectively scale reset-free learning schemes to much more complex problems, as we demonstrate in our experiments. We propose a simple scheme for multi-task learning that tackles the reset-free learning problem, and show its effectiveness at learning to solve complex dexterous manipulation tasks in both hardware and simulation without any explicit resets. This work shows the ability to learn dexterous manipulation behaviors in the real world with RL without any human intervention.

I. INTRODUCTION

Real-world robotic reinforcement learning has relied on instrumentation and episodic resets, which are especially difficult to provide for dexterous manipulation. This work uses multi-task learning to sequence behaviors that reset one another, and situates that approach among prior reset-free and multi-task methods.

I. INTRODUCTION

  • Real-world RL has depended on instrumentation and episodic resets, while dexterous manipulation makes reliable resets particularly challenging.Prior approaches have used human help, instrumented resets, or a separate robot to restore the object.Source: §I · p. 1

II. RELATED WORK

  • Prior multi-task work has focused on sample efficiency and generalization, while this work uses multi-task RL to address reset-free dexterous manipulation.Related reset-free methods include learned reset or perturbation controllers and sequenced forward/backward controllers; prior multi-task locomotion used domain-specific cyclic dependencies.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • Sequencing multiple learned tasks lets some behaviors reset others, avoiding explicit per-task resets and forming a network of behaviors easier to learn than a complete reset controller.The system is evaluated in simulation and on a four-fingered hand-arm system, including real-world in-hand manipulation and pipe insertion.Source: §I · p. 2

II. RELATED WORK

  • Real-world RL research has used highly instrumented setups for resets and rewards, whereas this work aims to remove explicit environment resets through multi-task learning.Examples of prior reset instrumentation include a scripted second arm and separate servo motors.Source: §II · p. 2

III. PRELIMINARIES

The paper frames reinforcement learning as maximizing expected cumulative reward and argues that learning a repertoire of tasks can make reset-free dexterous manipulation feasible. Its algorithm sequences task policies so successful behaviors can provide valid starts and resets for other tasks.

III. PRELIMINARIES

  • The preliminaries define RL as learning a policy to maximize expected cumulative reward, and note that the framework builds on actor-critic algorithms.The authors state that it can also be used with other standard RL algorithms with minimal modifications.Source: §III · p. 2

IV. LEARNING DEXTEROUS MANIPULATION BEHAVIORS RESET-FREE VIA MULTI-TASK RL

  • Multi-task learning can facilitate reset-free manipulation because diverse tasks can serve as resets for one another, forming a network rather than a forward/reset behavior pair.The example sequences recentering, pickup, and flip-up to resume in-hand reorientation after an object is dropped.Source: §IV · p. 3, Figure 2 · p. 4

A. Algorithm Description

  • MTRF jointly learns one policy per task with off-policy RL and continuous data collection, while a task graph selects which policy to execute and train every T steps.Each task has its own reward and replay buffer, and the graph is queried again after each data-collection interval.Source: A. Algorithm Description · p. 3
  • The method assumes successful outcomes of tasks entering a downstream task are valid initial states for that task.Upstream policies do not explicitly optimize coverage, so they may not expose downstream policies to all of their initial-state distribution; the authors report good performance in practice.Source: A. Algorithm Description · p. 3
  • The designer supplies the task graph and its transitions; learning successful tasks can make them effective resets for other tasks and form a natural curriculum.The paper notes that the graph could instead be learned from experience as future work.Source: A. Algorithm Description · p. 3

B. Practical Instantiation

The practical implementation trains separate task policies, critics, and replay buffers with an actor-critic method. A user-provided state machine periodically selects the policy to execute and update.

B. Practical Instantiation

  • Each task has a separate Gaussian policy, critic, and replay buffer, with policies trained by actor-critic methods such as SAC.Each policy samples training data from its own task-specific replay buffer.Source: B. Practical Instantiation · p. 4
  • A user-provided task-graph state machine is queried every T steps to choose the policy that will execute and train next.The environment transitions from its current state through T time steps of policy data collection before the graph is queried again.Source: B. Practical Instantiation · p. 4
  • During each interval, the selected policy collects task-specific rewards and experience, then its policy and value functions are trained from that task’s replay buffer using SAC.The algorithm initializes the policies, critics, and buffers before repeating this collection-and-training process.Source: B. Practical Instantiation · p. 4

V. TASK AND SYSTEM SETUP

The study evaluates reset-free dexterous manipulation on a 22-DoF hand-arm system in simulation and on hardware. Its task families cover lightbulb insertion and basketball dunking in simulation, and in-hand reorientation and pipe insertion on hardware.

V. TASK AND SYSTEM SETUP

  • The 22-DoF platform pairs a self-designed 16-DoF D’Hand with a 6-DoF Sawyer arm for extended-workspace tabletop manipulation.The hand was built for robustness and ease of long-term operation, including upwards of 100 hours in contact-rich tasks without breakage.Source: §V · p. 4

A. Simulation Domains

  • Simulation evaluates lightbulb centering, pickup, in-hand reorientation, and insertion, alongside basketball repositioning, pickup, positioning over the basket, and dunking.The basketball task has a natural cycle in which tasks can reset one another.Source: A. Simulation Domains · pp. 4–5

B. Hardware Tasks

  • Hardware tasks comprise rotating a three-pronged valve to a desired palm orientation and inserting a cylindrical pipe into a wall-mounted hose attachment.For in-hand reorientation, drops lead into tabletop recentering, pickup, and flip-up tasks before another attempt; dropped pipes are recentered and picked up for another insertion attempt.Source: B. Hardware Tasks · p. 5
  • In the in-hand reorientation setup, a 22-DoF hand-arm system was configured for extended uninterrupted operation, enabling over 60 hours of training without any human intervention.Source: B. Hardware Tasks · p. 5

VI. EXPERIMENTAL EVALUATION

The evaluation compares MTRF with reset-free baselines in simulation and tests real-world dexterous manipulation without human intervention. Across the experiments, tasks can support one another as resets, enabling joint learning of complex behaviors.

A. Baselines and Prior Methods

  • The baselines use soft actor-critic, and evaluation rolls out each final policy from states induced by tasks that can transition to the evaluated task.Comparisons include SAC, a reset controller, and a perturbation controller.Source: A. Baselines and Prior Methods · p. 6

B. Reset-Free Learning Comparisons in Simulation

  • Basketball training transitions progress from recentering and perturbation to pickup and dunking, cycling through recentering, pickup, and placement as learning improves.The task-frequency visualization notes that recentering and pickup can provide resets for other tasks.Source: B. Reset-Free Learning Comparisons in… · p. 6, Figure 9 · p. 7
  • MTRF learns all simulated tasks without explicit resets, more quickly and with higher average success than prior reset-free methods that struggle with the more complex tasks.Only MTRF reaches later tasks in the task graph.Source: B. Reset-Free Learning Comparisons in… · p. 6, Figure 8 · p. 7

C. Learning Real-World Dexterous Manipulation Skills

  • After about 60 hours of autonomous real-world training, MTRF learns in-hand manipulation, with 70% success on in-hand reorientation and more than 70% success across the in-hand tasks.The task sequence includes recentering, pickup, flipping, and reorientation; recentering initially provides resets for pickup.Source: C. Learning Real-World Dexterous… · p. 7, Figure 12 · p. 8
  • On hardware, the final pipe-insertion task learns to more than 60% success; the entire training process takes about 25 hours of real time while learning all tasks simultaneously.The film strip shows repositioning the object, picking it up, and inserting it into the wall attachment.Source: Figure 12 · p. 8, C. Learning Real-World Dexterous… · p. 8, Figure 11 · p. 8

VII. DISCUSSION

The paper uses multi-task reinforcement learning to make reset-free dexterous manipulation possible: different learned tasks provide resets for one another. Simulation and hardware experiments demonstrate joint learning, including real-world in-hand reorientation and pipe insertion without human intervention.

VII. DISCUSSION

  • The technique learns dexterous manipulation behaviors reset-free, without human intervention during training.It leverages multi-task reinforcement learning to make reset-free learning less challenging.Source: §VII · p. 8
  • When tasks are learned simultaneously, different tasks can reset one another and support uninterrupted learning.The system also learns behaviors that let it continue practicing.Source: §VII · p. 8
  • The real-world hand-arm system learned in-hand reorientation in about 60 hours and pipe insertion in around 25 hours, without human intervention or special instrumentation.Source: §VII · p. 8

APPENDIX A. REWARD FUNCTIONS AND ADDITIONAL TASK DETAILS

The hardware appendix specifies reward variables and success conditions for in-hand manipulation and pipe insertion. The pipe-insertion setup divides insertion into two tasks to reduce collisions.

A. In-Hand Manipulation Tasks on Hardware

B. Pipe Insertion Tasks on Hardware

  • Pipe insertion uses separate approach and insertion tasks to reduce collisions in real-world experiments.The reward description identifies object position, D’Hand joint positions, task goal positions or angles, and a lift threshold set to 0.1 m.Source: B. Pipe Insertion Tasks on Hardware · p. 10
  • A pipe-insertion rollout is reported successful when the valve is in-hand and the pipe is inserted.The displayed condition is z > threshold AND d2 < 0.05.Source: B. Pipe Insertion Tasks on Hardware · p. 11

C. Lightbulb Insertion Tasks in Simulation

The simulation lightbulb task adds a bulb-insertion reward to previously defined rewards. Figure-reported rollout success is based on the bulb reaching a position very close to the lamp’s goal.

C. Lightbulb Insertion Tasks in Simulation

D. Basketball Tasks in Simulation

The simulation basketball section adds a basket-dunking reward to the previously defined recentering and pickup rewards. Its reported success condition is a ball positioned very close to the goal above the basket.

D. Basketball Tasks in Simulation

APPENDIX B. ADDITIONAL DOMAINS

Additional simulations test the method on simpler pincer and lifting tasks. The pincer tasks form a cycle linking grasping, drawer filling, and opening the drawer.

APPENDIX B. ADDITIONAL DOMAINS

  • The additional simulation tests include a 2D pincer task and simplified lifting tasks on Sawyer and D’Hand setups.Source: Appendix B · p. 11
  • The pincer sequence consists of grasping an object, filling the drawer with it, and pulling the drawer open.Source: Figure 13 · p. 11
  • After drawer opening, the cycle allows grasping and filling practice to begin again.Source: Figure 13 · p. 11

APPENDIX C. HYPERPARAMETER DETAILS

Table I lists hyperparameters used across all domains.

APPENDIX C. HYPERPARAMETER DETAILS

  • Table I presents hyperparameters used across all domains.Source: Table I · p. 11

APPENDIX D. TASK GRAPH DETAILS

Appendix D provides task graphs for the paper’s domains. The graphs select behaviors using task-specific conditions, including object position, lift, orientation, and prior task where specified.

APPENDIX D. TASK GRAPH DETAILS

  • The hardware in-hand graph selects Inhand for lifted, upright objects and Flipup for lifted objects that are not upright.Otherwise it chooses Perturb, Recenter, or Lift based on object centering and the previous task.Source: Appendix D · p. 12
  • The hardware pipe-insertion graph selects Remove when inserted, Insert2 near the waypoint, and Insert1 when lifted.If none applies, it selects Perturb or Recenter for an off-center object depending on the previous task, and Lift otherwise.Source: Appendix D · p. 12
  • The simulated lightbulb graph selects among Recenter, Perturb, Lift, Flip Up, and Lightbulb Insertion based on centering, lift, and orientation.For off-center, unlifted objects it alternates between Recenter and Perturb according to the previous task.Source: Appendix D · p. 12
  • The simulated basketball graph uses a pickup location out of the hoop’s way and selects Basketball Dunking once the object is lifted.Its graph also selects Recenter or Perturb for off-center, unlifted objects and Lift when centered; the target wrist angle is upright.Source: Appendix D · p. 12
Loading 2104.11203v1…