PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling data-driven robotics with reward sketching and batch reinforcement learning

Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, Ziyu Wang

arXiv:1909.12200v2cs.ROcs.LG

TL;DR

Real-world robotics lacks abundant recorded data and task rewards, limiting deep RL. The paper combines human reward sketching with retrospective labeling of stored experience and offline batch RL. On a real robot, the approach learns policies for varied manipulation tasks, with reported generalization and robustness.

  • Problem

    Real-robot experience is limited by real-time data collection, and task rewards often do not naturally exist, limiting deep RL in robotics.

    Source: §I · p. 1

  • Method

    The approach learns task-specific rewards from human reward sketches, applies them to stored experience, and trains policies offline with batch RL.

    Source: §I · pp. 1–2

  • Results

    The learned policies perform varied manipulation tasks, including lifting and stacking rigid or deformable objects and USB insertion, with reported generalization and robustness.

    Source: §I · p. 3, §V · p. 8

  • Takeaways & Limitations

    Storing robot experience across tasks supports efficient policy learning entirely offline, and training-data diversity appears important to the success of standard state-of-the-art RL algorithms.

    Source: §V · p. 8

    Training still requires a human in the loop, reward sketching may not suit very different tasks, and learned agents remain sensitive to significant setup perturbations.

    Source: §V · p. 8

Abstract

from arXiv · show

We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.

I. INTRODUCTION

The paper addresses the scarcity of real-robot data with reward signals and the difficulty of learning manipulation policies that generalize across objects. It combines human reward sketching, retrospective labeling of stored experience, and offline batch RL to learn policies for varied real-robot tasks.

I. INTRODUCTION

  • Real-robot data is costly to collect in real time, and task rewards often do not naturally exist, limiting deep RL in robotics.Learning policies that generalize across object shapes, sizes, textures, and materials remains challenging.Source: §I · pp. 1–2
  • Reward sketching elicits human judgments of task progress as per-timestep annotations, which train a task-specific ranking reward model for labeling episodes.Annotators indicate successful timesteps with reward high enough to reach the goal region.Source: §I · p. 1, Figure 2 · p. 2
  • NeverEnding Storage retains diverse robot experience, allowing learned rewards to label historical episodes for new tasks with fixed annotation effort per task.The collection includes more than 400 hours of videos, proprioception, and actions from teleoperators and random, scripted, and trained policies.Source: §I · p. 2, Figure 3 · p. 2
  • The reward-labeled dataset supports offline batch RL on typically off-task, off-policy experience, without executing new policies on the robot during training.The workflow couples task-specific reward learning with batch training over the growing dataset.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Integrating human input, reward learning, and policy learning into a scalable closed loop poses substantial engineering challenges.The paper presents the NeverEnding Storage system as part of the engineering needed to organize and query robot experience.Source: §I · p. 2
  • The learned policies handle rigid and deformable object lifting and stacking, USB insertion, mistake recovery, and generalization to new objects and initial conditions.The examples also include robustness to some real-time adversarial interference.Source: Figure 4 · p. 2, §I · p. 3

II. METHODS

The workflow uses demonstrations and accumulated robot experience to build a task-specific reward model, then trains a policy from stored data. The resulting policy is deployed while data collection continues, with occasional careful evaluation.

II. METHODS

  • A human teleoperates the robot to demonstrate the target task, and all robot experience is accumulated in NeverEnding Storage.Demonstrations may be successful or unsuccessful, and the stored experience includes them.Source: §II · p. 3
  • Humans sketch rewards for a subset of episodes, and a reward model is trained from this fixed labeled experience.The selected episodes include task-specific demonstrations and accumulated experience.Source: §II · p. 3
  • The target-task agent is trained using all NeverEnding Storage experience and the reward model’s predicted reward values.Source: §II · p. 3
  • The resulting policy is deployed on the robot while more data is recorded, and agents are occasionally selected for careful evaluation.Source: §II · p. 3

A. Teleoperation

Teleoperation supplies demonstrations that inform reward learning and guide reinforcement learning toward the intended task outcome. Operators can also intervene in agent behavior, while each demonstration records observations and corresponding actions.

A. Teleoperation

  • Demonstrations provide examples of successful behavior that help bootstrap reward learning and use expert knowledge to guide reinforcement learning without autonomous exploration.Unsuccessful examples may also be provided.Source: A. Teleoperation · p. 3
  • An operator can take over from an agent or return control at any time to address particular corner cases.Source: A. Teleoperation · p. 3
  • Each demonstrated timestep pairs an observation with an action; observations include multiple-camera pixels and proprioceptive inputs.Source: A. Teleoperation · p. 3

B. NeverEnding Storage

NeverEnding Storage (NES) organizes robot experience so it can be annotated and queried across tasks. Reward sketching lets an annotator mark progress over time, including on prior-task episodes, though it is best suited to tasks where people can reliably compare timesteps.

B. NeverEnding Storage

  • NES stores experience from teleoperation, human play, scripted policies, and learned policies, with trajectory metadata, tags, and reward sketches available through SQL-type queries.Its metadata includes episode, operator, time, task, and episode-type information, plus a path to the separately stored experience.Source: B. NeverEnding Storage · p. 3

C. Reward Sketching

  • Annotators sketch reward by selecting video frames and assigning values, then drawing a curve while scrubbing through the episode rather than labeling every frame.The interface can support hundreds of frames of reward annotations per minute from a single annotator.Source: C. Reward Sketching · pp. 3–4
  • For a new task, annotators can sketch both its demonstrations and prior-task experience, which the paper says improves state-space coverage.NES can associate multiple task-specific sketches with an episode.Source: B. NeverEnding Storage · p. 3, C. Reward Sketching · p. 4
  • Reward sketching is suited to tasks where people can reliably compare timesteps, but variable-speed tasks and cyclic tasks such as walking can be hard to sketch.The authors identify broader annotation interfaces as future work for addressing a wider variety of tasks.Source: C. Reward Sketching · p. 4

D. Reward Learning

The paper learns task rewards from human sketches and uses them to label stored experience for offline policy learning. Its batch-RL training draws on both the shared repository and task-specific data, with recurrent distributional value and policy networks.

D. Reward Learning

  • A learned reward model assigns predicted rewards to all NES experience, allowing historical data to be used for a new task without manually annotating the entire repository.The model is trained from episodes with reward sketches.Source: D. Reward Learning · p. 4
  • Because annotators agree more on within-episode rankings than on reward scale, the reward model uses intra-episode ranking rather than direct regression of sketched values.Its losses also enforce agreement on sketched success and failure labels and on sufficiently separated sketch rankings.Source: D. Reward Learning · p. 4

E. Batch RL

  • Batch RL learns a new policy from stored data generated by previous policies, without further robot execution, and the authors report that historical data from other tasks is important.They use batches with 75% drawn from all NES data and 25% from the target task’s data.Source: E. Batch RL · p. 4
  • The agent uses recurrent policy and value networks with distributional value learning, following an algorithm similar to D4PG.The policy is updated using DPG, while the critic minimizes a distributional-value loss on sampled observation-action sequences.Source: E. Batch RL · p. 4

F. Execution

Deploying an agent adds new robot experience that can inform reward sketching or later policy learning. The workflow also uses human review to correct reward-model delusions and iterate on the policy.

F. Execution

  • Running a trained agent on the robot collects experience that can be used for future reward sketching or reinforcement learning.Deployment also lets researchers observe performance and judge what should be improved.Source: F. Execution · pp. 4–5
  • The robot records data from its cage and wrist cameras, proprioception, and actions, while training uses varied object shapes, textures, and sizes for deployment-time generalization.The recorded setup includes three cage cameras and three wrist cameras, including wide-angle and depth cameras.Source: Figure 6 · p. 5
  • Early policies can exploit reward “delusions” when reward models lack state-space coverage; annotators sketch those episodes to improve the model before training another policy.The correction cycle is typically repeated two to three times per target task until reward predictions are satisfactory.Source: F. Execution · p. 5

III. EXPERIMENTS

The experiments use three manipulation tasks and evaluate agents under controlled conditions on a physical robot. Training proceeds iteratively, with later agents using more of the collected data and improving on the stacking task.

A. Experimental Setup

  • The experiments cover lift_green and stack_green_on_red on variable-shape rigid objects and lift_cloth on deformable objects, using the rgb and deformable datasets.Episodes last up to 200 steps, or 20 seconds, unless ended earlier for safety.Source: A. Experimental Setup · p. 5
  • The datasets include teleoperated episodes, but only about 600 full teleoperated episodes correspond to lift_green or stack_green_on_red.The majority of teleoperated episodes were collected through interactive teleoperation requiring limited human intervention.Source: A. Experimental Setup · p. 5
  • Reward sketches cover 894 lift_green episodes, 1,201 stack_green_on_red episodes, and 585 lift_cloth episodes, with approximately 90% used for training and 10% for validation.The sketches accumulated across several iterations of the workflow.Source: A. Experimental Setup · p. 5
  • The agent uses camera encoders with spatial-softmax keypoints and concatenated proprioceptive features; each camera is encoded independently before the results are combined.The spatial-softmax encoder produces 64 (x, y) keypoints per camera, and the authors found it important in their experiments.Source: A. Experimental Setup · p. 5, Figure 7 · p. 6
  • With more training data, the stack_green_on_red agent improves; across three iterations it progresses from dropping a picked-up block to attempting a stack and then placing the green block on the red one.The three example agents used approximately 82%, 94%, and 100% of collected data, respectively.Source: Figure 8 · p. 6, A. Experimental Setup · p. 6
  • For lift_green and stack_green_on_red, evaluation uses normal, hard, and unseen conditions, each with ten object initial positions; the harder conditions involve less represented or unseen objects and varied configurations.Lift_cloth is evaluated with randomized initial conditions at each trial, and success is judged by a human operator.Source: A. Experimental Setup · p. 6

B. Results

The learned policies achieve substantial lifting and stacking success, including on rarely seen or unseen objects. Dataset ablations and value-function comparisons identify data diversity and distributional value functions as important to performance.

B. Results

  • The agent succeeds in 80% of lifting and 60% of stacking episodes on the rgb dataset.For rarely seen objects in adversarial positions, success is 80% for lifting and 40% for stacking; for never-seen objects, it is 50% and 40%, respectively.Source: B. Results · pp. 6–7
  • The policies compare favorably with Zhu et al.'s block lifting and stacking rates of 64% and 35%, though the physical setups differ.The cited comparison is not perfectly comparable; the paper also contrasts its varied objects and 6-DoF control with fixed-object results from other methods.Source: B. Results · p. 7
  • Using only task-specific data reduces policy performance to 0% in almost all scenarios, while random-watcher data adds up to 30% improvement, especially for unseen objects.The random-watcher data shows its biggest advantage in the hardest case with unseen objects.Source: B. Results · p. 7
  • Distributional value functions are essential for good performance in batch RL.The paper reports this from the distributional versus non-distributional comparison in the last rows of Tab. II.Source: B. Results · p. 7
  • Qualitative evaluations show agents handling human perturbations, unseen and non-standard objects, and lifting toys; in some cases they move faster and more efficiently than a human operator.The human-operator comparison is illustrated in Fig. 9.Source: B. Results · p. 7

C. Interactive Insertion

The interactive insertion experiment learns a vision-based USB-insertion policy from scratch, with human reward feedback rather than historical data and batch RL. Across three runs, success exceeded 80% within eight hours, and the policy also handled substantial positional changes.

C. Interactive Insertion

  • This experiment learns the policy from scratch in a tight data-collection, reward-learning, and policy-learning loop, without historical data or batch RL.The authors describe this alternative as suitable for less data-demanding applications.Source: C. Interactive Insertion · p. 7
  • In the stack_green_on_red comparison, the agent completes the task faster than a human operator.Fig. 9 shows episode frames for the agent and human at three-second intervals.Source: Figure 9 · p. 7
  • The setup uses 6-DoF velocity control, 15-second episodes at 10 Hz, and wrist-camera images of 84 × 84 pixels for both policy and reward models.Episodes begin within a 6 × 6 × 6 cm position region and 8.6° rotation in each direction; 100 human demonstrations are collected and annotated with sketches.Source: C. Interactive Insertion · p. 7
  • The agent reaches over 80% success within 8 hours of online training, with 65 ± 10 additional reward sketches from the human annotator.The experiment was repeated three times; Fig. 10 shows average success rate over time, and the paper reports solving the task within a single working day.Source: C. Interactive Insertion · p. 7, Figure 10 · p. 7
  • The policy inserts USB keys using pixel inputs and remains robust to unseen positional changes, including a significantly moved socket.The second example uses an agent trained on the unperturbed state; actions are defined with respect to the wrist frame.Source: C. Interactive Insertion · p. 7

IV. RELATED WORK

The related work contrasts this approach with reward engineering, prior preference and demonstration methods, other batch-RL work, and sim2real. It highlights the challenge of learning rewards from pixels and the engineering demands of real-robot and simulation-based approaches.

IV. RELATED WORK

  • Reward learning from inverse RL works best on states or engineered features, while applying it to raw pixels remains challenging; reward sketching is introduced to elicit many useful preferences with little user input.Reward sketching supports perceptual reward learning from arbitrary trajectories, not only successful demonstrations.Source: §IV · p. 8
  • Batch RL is an active research area, with recent work addressing stability; demonstrations are also used in model-free temporal-difference learning to help address exploration.The paper uses demonstrations as part of the agent's experience.Source: §IV · p. 8
  • Behavioral cloning requires consistent, high-quality target-task demonstrations, cannot benefit from heterogeneous data, and generally cannot outperform its demonstrator.Around 15% of human demonstrations in the authors' dataset fail to accomplish the task by episode end.Source: §IV · p. 8
  • Real-robot RL often relies on task-specific state estimation, limited raw-observation data, and highly engineered rewards; QT-Opt generalizes across objects but focuses on grasping.The QT-Opt dataset contains over 580,000 grasps collected over several weeks with seven robots.Source: §IV · p. 8
  • Sim2real avoids direct real-world data collection but requires extensive system identification and still faces a simulation-to-reality domain gap.The paper's data-driven framework does not need to simulate the robot or environment, and therefore has no domain gap.Source: §IV · p. 8

V. CONCLUSIONS

The conclusion presents a data-driven robotics framework combining reward learning, retrospective reward labeling, and batch RL with distributional value functions. Its results emphasize reward sketching, long-term cross-task data, and policy generalization, while noting costs and scope limits.

V. CONCLUSIONS

  • The proposed approach combines reward learning, retrospective reward labeling, and batch RL with distributional value functions, supported by substantial engineering at this scale.The authors plan to release the NeverEnding Storage datasets and canonical agents.Source: §V · p. 8
  • Reward sketching effectively elicits reward functions, and storing experience over time and across tasks supports efficient offline policy learning.The authors identify training-data diversity as essential to standard state-of-the-art RL performance, noting prior failures with only expert data or one agent's history [23].Source: §V · p. 8
  • Results across a wide set of tasks show versatility, with learned agents demonstrating significant generalization and robustness.Source: §V · p. 8
  • Training still requires a human in the loop, reward sketching may not suit very different tasks, and agents remain sensitive to significant setup perturbations.The authors identify these as open questions for future work.Source: §V · p. 8
Loading 1909.12200v2…