PaperlayerView exact revision on arXiv

Source-linked AI summary

MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale

Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, Karol Hausman

arXiv:2104.08212v2cs.ROcs.LG

TL;DR

General-purpose robots need many skills, but real-world RL makes acquiring each skill costly. MT-Opt combines example-based success detectors, multi-robot multi-task data collection, and shared multi-task RL. Across 12 real-world tasks trained with data from 7 robots, it generalizes to related new tasks and acquires distinct new tasks faster by leveraging prior experience.

  • Problem

    The paper asks how robots can acquire broad repertoires of skills despite the time required to learn each behavior with reinforcement learning.

  • Method

    MT-Opt combines example-based success detectors, simultaneous multi-robot data collection, and a multi-task deep RL system that shares representations, data, and exploration.

  • Results

    Across 12 real-world tasks, MT-Opt significantly outperforms several baselines, with up to approximately 3× average improvement, and 12-task training exceeds 2-task training by 7% on lift-any and 22% on place-any.

  • Takeaways & Limitations

    Learning multiple related tasks improves data efficiency, supports more complex behaviors, and enables faster acquisition of new tasks by leveraging prior experience.

Abstract

from arXiv · show

General-purpose robotic systems must master a large repertoire of diverse skills to be useful in a range of daily tasks. While reinforcement learning provides a powerful framework for acquiring individual behaviors, the time needed to acquire each skill makes the prospect of a generalist robot trained with RL daunting. In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks. In this framework new tasks can be continuously instantiated from previously learned tasks improving overall performance and capabilities of the system. To instantiate this system, we develop a scalable and intuitive framework for specifying new tasks through user-provided examples of desired outcomes, devise a multi-robot collective learning system for data collection that simultaneously collects experience for multiple tasks, and develop a scalable and generalizable multi-task deep reinforcement learning method, which we call MT-Opt. We demonstrate how MT-Opt can learn a wide range of skills, including semantic picking (i.e., picking an object from a particular category), placing into various fixtures (e.g., placing a food item onto a plate), covering, aligning, and rearranging. We train and evaluate our system on a set of 12 real-world tasks with data collected from 7 robots, and demonstrate the performance of our system both in terms of its ability to generalize to structurally similar new tasks, and acquire distinct new tasks more quickly by leveraging past experience. We recommend viewing the videos at https://karolhausman.github.io/mt-opt/

I. INTRODUCTION

MT-Opt addresses the high cost and engineering burden of real-world robotic reinforcement learning by learning multiple tasks together and sharing data, representations, and exploration. The system uses example-based task rewards and is designed to acquire new and more complex skills efficiently.

  • Motivation: Real-world robotic RL can learn effective skills, but requires substantial on-robot training time and task-specific engineering.QT-Opt, for example, required over 500,000 trials across multiple robots for vision-based grasping.
  • Motivation: Multi-task RL can amortize learning costs when tasks share structure, including visual features and manipulation concepts.The shared structure must be discovered by the learning algorithm for these efficiency benefits to apply.
  • Motivation: Easier tasks can bootstrap harder tasks by providing exploration experience, such as single-item picking aiding multi-item placement.The framework also aims to make later task learning easier through increasingly useful shared representations.
  • Challenges: Realizing these benefits requires solving optimization difficulty, scalable reward definition, and data-sharing challenges in image-based robotic RL.The paper frames these challenges as barriers to large-scale demonstrations of generalizable real-world skills.
  • Contributions: MT-Opt is a general multi-task learning system that addresses these challenges through scalable success classifiers and shared representations, data, and learned policies.Users define rewards through examples, while prior tasks support faster acquisition of new tasks.
  • Contributions: Learning related tasks simultaneously improves data efficiency and enables more complex tasks than a single-task setup.The system also adapts to behaviorally and visually distinct tasks and can learn new tasks with less data than isolated training.

II. RELATED WORK

Prior work develops multi-task, hierarchical, meta-learning, and continual-learning approaches for sharing knowledge across tasks. MT-Opt applies these ideas to real robotic reinforcement learning through shared policies, data, and continuously added tasks.

  • Multi-task learning: Multi-task learning shares structure and data to improve efficiency and has been successful in natural language processing and computer vision.The paper applies this concept to real robotic reinforcement learning, which introduces additional challenges.
  • Multi-task reinforcement learning: Related RL methods combine task policies through gating, task conditioning, parameter mappings, or policy distillation.MT-Opt instead directly learns a shared policy to exploit common task structure.
  • Complex and composite skills: Other approaches address complex skills through hierarchical options, sub-task composition, sample reuse, hindsight relabeling, and demonstrations.Related work also uses autonomous supervision, large-scale data collection, task sampling, and learned task spaces.
  • Meta-learning and continual learning: Meta-learning accelerates training by sharing knowledge across tasks through imitation-learning and reinforcement-learning methods.Continual skill acquisition is also framed as lifelong or continual learning.
  • Continual learning: MT-Opt allows offline data to be shared across tasks and new tasks to be added through new success detectors without recollecting data.This design supports continuous acquisition of new skills.
  • MT-Opt positioning: The system combines distributed off-policy multi-task RL with visual success detectors and a shared policy, data-routing strategies, and exploration transfer.These components are designed for simultaneous learning of multiple robotic manipulation tasks.

A. Multi-Task Reinforcement Learning Algorithm

MT-Opt extends off-policy Q-learning to multiple tasks by conditioning a shared Q-function on task identity and averaging task-specific losses. Its data-sharing formulation allows transitions to contribute to multiple tasks.

  • System loop: The MT-Opt overview iterates among example-based success detection, multi-task episode collection, and offline data expansion and rebalancing.Additional on-policy data can be collected, particularly for new tasks, and the resulting experience supports future task definitions and policies.
  • RL formulation: The multi-task policy selects robot actions from camera-image states while conditioning on a categorical task encoding.Actions include arm position, arm orientation, and gripper commands; rewards depend on the selected task.
  • RL formulation: MT-Opt learns a shared multi-task Q-function that estimates expected future rewards for state, action, and task combinations.It builds on QT-Opt, which learns Q-values by minimizing Bellman error.
  • Q-learning objective: The single-task objective minimizes divergence between predicted Q-values and target Q-values computed from rewards and discounted next-state values.The target value uses stochastic optimization with the cross-entropy method, while expectations use previously seen transitions.
  • Multi-task objective: The multi-task loss averages the task-specific losses over tasks sampled from a categorical task distribution.Transitions generated for one task can be reused for multiple tasks in the multi-task formulation.
  • Data sharing: A basic multi-task Q-learning system does not fully exploit cross-task learning when each task uses only its own data.MT-Opt therefore introduces additional task-impersonation and data-sharing mechanisms.

B. Task Impersonation and Data Rebalancing

MT-Opt shares whole episodes across related tasks, then rebalances task and success/failure composition to exploit reusable data without overwhelming individual tasks with unrelated negatives.

  • Task Impersonation: Task impersonation transforms one collected episode into episodes for a task-dependent subset, generating both successes and failures for task-conditioned training.Unlike hindsight relabeling, the method operates at the whole-episode level and can relabel outcomes across tasks.
  • Task Impersonation: The identity baseline restricts each episode to its original task, whereas maximal sharing exposes every episode to all remaining tasks.These alternatives define the extremes of the data-sharing spectrum.
  • Task Impersonation: Skill-based impersonation shares episodes only among tasks with similar semantics and dynamics, limiting dilution from unrelated negative examples.The paper manually groups tasks into skills, such as placing an object on a plate or in a bowl.
  • Data Rebalancing: Impersonation creates severe imbalance in per-task data and success frequencies, so sequential rebalancing controls task proportions and success/failure ratios in training batches.The process may drastically oversample scarce-task transitions, but the authors report a positive impact on task performance in ablations.

V. REWARDS VIA MULTI-TASK SUCCESS DETECTORS

MT-Opt defines sparse rewards with multi-task visual success detectors trained from user-provided outcome examples, while continuously collecting and rebalancing data across tasks and robots.

  • Success Detectors: A multi-task success detector estimates task success from an episode’s final image under a sparse-reward assumption and is conditioned on task identity.The detector is trained as a single neural network for multiple tasks.
  • Success Detectors: Users provide positive desired outcomes and visually similar negative near-misses rather than demonstrations of how to perform the task.The initial detector is incrementally retrained with manually labeled out-of-distribution images caused by lighting, backgrounds, and novel robot states.
  • Success Detectors: Success and failure videos are captured iteratively in pairs to reduce correlations between labels and incidental workspace features.The paired collection varies factors such as users’ hands, backgrounds, bins, and distractor objects.
  • Continuous Data Collection: The resulting multi-task dataset is imbalanced across exploration-policy distributions and per-task success rates, which impersonation and rebalancing address.The dataset grows over time in both per-task quantity and successful-episode proportion.
  • Continuous Data Collection: The data-collection strategy gathers multiple tasks simultaneously and uses easier general tasks to bootstrap exploration for harder specialized tasks.Policies can occasionally produce successful episodes for harder tasks, enabling continued training and improved data collection.

VII. EXPERIMENTS

The experiments evaluate MT-Opt across 12 vision-based robotic manipulation tasks, comparing shared multi-task policies and data strategies with single-task learning. MT-Opt shows stronger performance, particularly on difficult tasks with limited data, while learned behaviors include challenging semantic manipulation.

  • Experimental setup: The study evaluates MT-Opt on 12 real-world manipulation tasks spanning object acquisition and object manipulation.Tasks include general and semantic lifting, placing objects on divided plates, covering, aligning, and rearranging.
  • Experimental setup: The evaluation deploys offline-RL policies on 7 robots, with each task attempted 100 times.Evaluation uses a standard scene in which all 12 tasks are feasible.
  • Quantitative evaluation: MT-Opt significantly outperforms single-task, shared-policy, and data-sharing baselines, with approximately 3× average improvement in some cases.The comparison covers QT-Opt, QT-Opt MultiTask, and Data-Sharing Multi-Task.
  • Quantitative evaluation: Single-task QT-Opt performs similarly on lift-any but drops to approximately 1% on difficult underrepresented tasks such as lift-can.The single-task baseline cannot be run for placing tasks because those tasks require a separate lifting task.
  • Qualitative evaluation: MT-Opt learns challenging behaviors such as repositioning a carrot from a bin corner before grasping it.Semantic picking becomes harder in crowded small bins where objects can be pressed against bin walls.
  • Data distribution: Tasks with more data, especially on-policy data, achieve higher success rates than underrepresented tasks such as lift-box.The paper suggests that focusing data collection on underrepresented tasks could further improve their performance.

C. Sharing Representations Between Tasks

MT-Opt improves multi-task learning by sharing representations and selectively expanding and rebalancing data across related tasks. These mechanisms improve performance on both directly trained tasks and harder, underrepresented tasks, including a specific lifting task supported by broader-task experience.

  • Sharing Representations Between Tasks: A 12-task policy outperforms a 2-task policy on lift-any and place-any despite using the same offline dataset.The gains are 7% for lift-any and 22% for place-any.
  • Sharing Representations Between Tasks: Training on additional tasks improves performance on lift-any and place-any, consistent with beneficially shared representations.The paper attributes the improvement to additional supervision from the other tasks.
  • Data Sharing Between Tasks: Task impersonation and data rebalancing together provide significant improvement across all evaluation tasks, reaching up to 10× on some tasks.The combined strategy is especially beneficial for underrepresented tasks.
  • Data Sharing Between Tasks: Naively sharing all data across tasks performs worse than both no sharing and MT-Opt’s selective sharing strategy.Data imbalance otherwise favors broad, easier tasks and harms harder underrepresented tasks.
  • Using Easier Tasks to Bootstrap Harder Tasks: For lift-sausage, MT-Opt reaches 39% success versus 3% for a single-task policy trained on 16,600 episodes.Both experiments use identical successful episodes; MT-Opt additionally uses impersonated successes and failures from other tasks.
  • Using Easier Tasks to Bootstrap Harder Tasks: The lift-sausage result demonstrates value from both successful and unsuccessful data collected by other tasks.The broader lift-any task supplies related experience, while rebalanced failures contribute to the performance increase.

F. Learning New Tasks with MT-Opt

MT-Opt acquires new tasks by specializing previously learned skills or proactively adapting shared policies to visually and behaviorally different tasks. In both settings, prior experience and multi-task representations reduce the data or training needed for new capabilities.

  • MT-Opt specializes broad skills into harder, more specific tasks such as lift-sausage through retroactive relabelling of prior data.This extends task acquisition to objects defined by properties including size, location, color, or shape.
  • The system can proactively adapt known tasks to previously unseen behaviors, including cloth picking and object covering.The experiment fine-tunes a 14-task model using data collected for the new tasks.
  • After 11K pick-cloth and 3K cover-object attempts, the extended policy reached 70% and 44% success, versus 33% and 5% when trained only on those tasks.The new tasks had no successful episodes available for offline relabelling before collection began.
  • MT-Opt combines multi-task collection, scalable success detectors, and multi-task deep RL to share representations and exploration across tasks.The framework also generalizes to placing into new fixtures, covering, aligning, and rearranging, while acquiring new tasks quickly.

X. APPENDIX

The appendix details MT-Opt’s task-conditioned Q-function, task impersonation, data balancing, and bootstrapping mechanisms. These components expose both the benefits and practical constraints of sharing experience across tasks.

  • Q-function architecture: MT-Opt conditions a convolutional Q-function on the image, action, state features, and one-hot task ID.The action, state, and task inputs are fused with visual features before producing the Q-value.
  • Q-function architecture: A multi-headed Q-function under-performs on most tasks, with the exception of lift-any tasks, which have the most data.The authors hypothesize that separate heads over-compartmentalize task policies and impede shared representations.
  • Task impersonation: Task impersonation transforms an episode from one task into training episodes for other tasks, using both success and failure labels to define task boundaries.The approach operates at the whole-episode level and limits candidate tasks according to the source episode’s skill.
  • Task impersonation: Skill-based impersonation restricts sharing to tasks with shared semantics and dynamics, such as placing an object on a plate or in a bowl.This finer-grained strategy avoids unrestricted sharing across unrelated task families.
  • Task impersonation: Sharing can create excessive artificial negatives for visually different tasks, diluting scarce intrinsic failures.For transparent-bottle lifting, failures generated from abundant plastic-object data can overwhelm intrinsic failures.
  • Task impersonation: The stochastic fIskill function samples impersonated successes with ps = 1.0 and failures with pf <= 1.0 to balance surplus data.This preferentially retains successful examples while allowing only a fraction of surplus failures into training.

D. Distributed Asynchronous System

The distributed asynchronous system continuously labels, augments, balances, and replays multi-task robotic data. It combines human demonstrations, robot episodes, and cross-task feasibility rules to train success detectors and RL policies at scale.

  • Success detector training: New success detectors are initialized from multi-view human success and failure videos or relabelled robot-policy data.Human demonstrations use 5Hz videos from three camera angles, while an example relabelling route is lift-any data for lift-carrot.
  • Success detector training: Evaluation holdouts are split by robot, reserving all data from a robot for testing rather than randomly selecting 20% of examples.This tests generalization of the success detector across robots.
  • Success detector training: Success-detector training uses cropped, brightened, rotated, and shadowed images, with success and failure examples balanced across tasks.Cross-task feasibility rules automatically generate labels, such as lift-carrot successes becoming failures for other instance-lifting tasks and successes for lift-any.

XII. ROBOT SETUP

The robot setup uses an automated multi-robot workspace to collect diverse multi-task experience continuously. Seven robots, multiple cameras, resettable bins, and a plate fixture support large-scale data collection and evaluation.

  • Data collection system: The system assigns each robot a task and stores completed episodes with task identity for later cross-task training.Episodes can be used to train other tasks through MT-Opt data impersonation.
  • Physical setup: Seven KUKA IIWA arms use two-finger grippers and three RGB cameras in a workspace with two bins and a magnetically attached plate fixture.The actuated resettable bin enables automated workspace resets and shuffling of objects.
  • Physical setup: Automated resets allow data collection 24 hours per day, 7 days a week across multiple robots.The setup shuffles and dumps workbench contents back into the source bin after episodes.
  • Collection throughput: Approximately 3,300 episodes per day are collected on one robot and 23K episodes per day across the seven-robot fleet.Each episode averages approximately 10 steps and takes approximately 25 seconds, including reset time.
  • Evaluation scene: The evaluation scene contains one of three colored plates and nine graspable objects, including one object from each of seven categories.Two additional toy food objects vary across categories or may be outside them.

A. Details of Data Collection to bootstrap a Multi-Task System

The system assembled a large, diverse offline dataset through extended multi-robot collection across varied environments, operating conditions, detectors, training regimes, and policies. The resulting data supported multi-task evaluation and analysis of dataset success distributions and re-balancing strategies.

  • Data collection: Nearly 800,000 episodes were collected over 16 months from three lab locations and 6–7 KUKA robots operating across varied conditions.Collection varied by time of day, background, lighting, and camera pose.
  • Re-balancing evaluation: Table IV compares seven data-impersonation and re-balancing strategies for a shared policy across 12 tasks using identical offline data and computation budgets.Each strategy was evaluated with 100 real-robot attempts per task, totaling 8,400 evaluations; the average-best strategy was not best on every task because dataset imbalance favored different strategies for overrepresented tasks.
  • Data collection: The collection process varied success detectors, reinforcement-learning training regimes, and mixtures of scripted, epsilon-greedy, and on-policy data.These components were iteratively improved or varied over time.
  • Task expansion: Specific pick-and-place tasks were introduced after lift-anything and place-anywhere capabilities, with randomly selected pick tasks followed by randomly selected place tasks.Variations required either picking a specific object or placing an object in a specific location.
  • Dataset analysis: Figure 17 shows effective success-rate distributions for each task across the entire offline dataset, rather than the performance of any particular policy.The distributions aggregate successes collected over time from many policies.
  • Dataset: The resulting offline dataset contained 800,000+ episodes with substantial diversity across tasks, policies, and success-rate dimensions.The dataset was assembled from the collection process described above.
Loading 2104.08212v2…