Source-linked AI summary
Scaling data-driven robotics with reward sketching and batch reinforcement learning
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, Ziyu Wang
TL;DR
Real-world robotics has limited recorded data and task rewards are not naturally available, while complex tasks are difficult to specify with hand-crafted rewards. The paper learns task-specific reward functions from human progress annotations and applies them to stored experience for offline batch RL. The resulting policies perform a range of manipulation tasks and show generalization and robustness.
Problem
Real-world robotics lacks abundant recorded data and naturally available task rewards, limiting deep RL for tasks where hand-crafted rewards do not scale well.
Source: §I · p. 1Method
The approach learns task-specific rewards from human progress sketches, retrospectively labels stored experience, and trains policies offline with batch RL.
Source: §I · pp. 1–2Results
Across tasks, agents handle rigid and deformable object manipulation and USB insertion, with 80% success for lifting and 60% for stacking on the rgb dataset.
Source: §I · p. 3, B. Results · p. 6Takeaways & Limitations
The results show that stored experience collected across tasks can support offline policy learning, with learned agents exhibiting generalization and robustness.
Source: §V · p. 8Training requires a human in the loop at additional cost, reward sketching may not suit every task, and agents remain sensitive to significant setup perturbations.
Source: §V · p. 8
Abstract
from arXiv · showhide
We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.
I. INTRODUCTION
The paper addresses scarce robot data and unavailable task rewards with a pipeline that learns task-specific rewards from human progress sketches and applies them to stored experience for offline batch RL. It reports manipulation policies spanning rigid and deformable objects and USB insertion, with generalization to new objects and initial conditions.
I. INTRODUCTION
- Reward sketching elicits rapid human judgments of per-timestep progress, which train task-specific reward models for tasks without directly available rewards.The model is trained from sketches and then used to label other episodes.Source: §I · p. 1, Figure 2 · p. 2
- Continuous collection stores over 400 hours of multi-camera video, proprioception, and actions from teleoperators and random, scripted, and trained policies.Learned rewards retrospectively repurpose past experience with a fixed amount of annotation effort per task.Source: §I · p. 2, Figure 3 · p. 2
- Task-specific rewards let batch RL learn new visuomotor policies from stored data that is necessarily off-policy and typically off-task, without robot execution.The approach uses historical experience rather than collecting a new dataset for each target task.Source: §I · pp. 1–2
- Integrating preference elicitation, retrospective reward labeling, stored experience, and offline policy learning into a scalable system requires substantial engineering.The system stores demonstrations and behavior from trained, scripted, and random policies in NeverEnding Storage.Source: §I · p. 2
- The learned policies handle rigid and deformable object lifting and stacking, USB insertion, recovery from mistakes, new initial conditions, and some real-time adversarial interference.The introduction also reports that the policies solve tasks more effectively than human teleoperators.Source: Figure 4 · p. 2, §I · p. 3
II. METHODS
For a new task, the procedure gathers demonstrations and experience, uses human reward sketches to train a reward model, and then trains and evaluates a policy using predicted rewards on stored data.
II. METHODS
- The workflow gathers target-task demonstrations and accumulates all robot experience in NeverEnding Storage before selecting episodes for human reward-sketch annotation.The selected episodes include target-task demonstrations and prior experience.Source: §II · p. 3
- A reward model for the target task is trained from a fixed amount of labeled experience.The training labels are human-provided reward sketches.Source: §II · p. 3
- The target-task agent is trained using predicted reward values on all experience stored in NeverEnding Storage.The reward model can annotate the repository without manual annotation of every episode.Source: §II · p. 3
- The resulting policy is deployed while more data is recorded, and agents are periodically selected for careful evaluation.Newly recorded experience can be annotated and reused in later training.Source: §II · p. 3
A. Teleoperation
Teleoperation provides task demonstrations that support reward learning and guide reinforcement learning toward the intended outcome. During agent control, a human can intervene to address corner cases.
A. Teleoperation
- Demonstrations provide examples of task completion that bootstrap reward learning and use expert knowledge of the intended outcome to guide reinforcement learning.Operators can provide several successful and occasionally unsuccessful examples.Source: A. Teleoperation · p. 3
- A human operator can take over from or return control to an agent at any time, providing data for fixing encountered corner cases.Source: A. Teleoperation · p. 3
- Each demonstrated timestep pairs an action with observations containing raw pixels from multiple cameras and proprioceptive inputs.The robot is controlled with a 6-DoF mouse and gripper button or a handheld virtual-reality controller.Source: A. Teleoperation · p. 3
B. NeverEnding Storage
NeverEnding Storage collects cross-task robot experience and metadata for reuse. Reward sketches provide task-specific supervision, which trains a reward model to annotate stored experience for policy learning.
B. NeverEnding Storage
- NeverEnding Storage (NES) centralizes cross-task robot trajectories and uses queryable metadata, labels, and reward sketches to retrieve data for stages of the training pipeline.Its relational database stores environment and policy metadata alongside recordings from several cameras and sensors.Source: B. NeverEnding Storage · p. 3
C. Reward Sketching
- Reward sketching lets an annotator draw progress curves while scrubbing through episodes, including demonstrations and prior-task experience to broaden state-space coverage.The interface supports hundreds of frame annotations per minute and reserves a region for goal-achieved frames.Source: C. Reward Sketching · p. 4
- Sketching best fits tasks where people can reliably compare timesteps; variable-speed tasks and cyclic activities such as walking are harder to sketch.The authors identify broader annotation interfaces as future work.Source: C. Reward Sketching · p. 4
D. Reward Learning
- A supervised reward model predicts rewards for all NES experience, allowing historical data to be used for a new task without manually annotating the entire repository.Source: D. Reward Learning · p. 4
- Because annotators agree more on within-episode rankings than absolute reward scales, training combines thresholded success labels with relative reward ordering through hinge losses.The ranking condition compares frames within the same episode.Source: D. Reward Learning · p. 4
E. Batch RL
The paper trains policies offline with batch reinforcement learning on stored experience. It uses distributional RL, recurrent value and policy networks, and batches that mix repository-wide with target-task data.
E. Batch RL
- Batch RL learns a new policy from data generated by previous policies without further robot execution, and the agent uses distributional RL without feature pretraining, behavior-cloning initialization, batch-correction terms, or auxiliary losses.The authors find historical data from other tasks important.Source: E. Batch RL · p. 4
- The authors release datasets and canonical agents to encourage further investigation of batch RL algorithms in robotics.They note that other batch RL methods might also yield good results.Source: E. Batch RL · p. 4
- The training algorithm is similar to D4PG, with recurrent value and policy networks, a distributional critic loss, and policy updates using deterministic policy gradients.It samples observation-action sequences, initializes recurrent states with zeros, and updates networks using backpropagation through time.Source: E. Batch RL · p. 4
- Fixed-ratio batches draw 75% of data from all NES experience and 25% from data specific to the target task.Source: E. Batch RL · p. 4
F. Execution
Running a trained agent generates experience for later reward sketching or policy learning and lets researchers judge what to improve. Early reward-model delusions can prompt another annotation-and-training cycle.
F. Execution
- Running a trained agent both produces experience for future reward sketching or RL and helps researchers judge what to improve.Source: F. Execution · p. 4
- The robot records camera, proprioceptive, and action data, and is trained with varied object shapes, textures, and sizes to support deployment generalization.Source: Figure 6 · p. 5
- When reward models assign high rewards to undesired behavior, annotators sketch those episodes and the team trains a new policy; this cycle typically repeats 2–3 times per task.Source: F. Execution · p. 5
III. EXPERIMENTS
The experiments use a persistent, heterogeneous robot-experience dataset to train policies for rigid-object and cloth manipulation. Policies are learned offline, then evaluated on the physical robot under controlled conditions.
A. Experimental Setup
- A Sawyer robot records multi-camera observations, proprioception, and actions, with a six-degree-of-freedom continuous action space and binary gripper control.The setup also includes wrist sensing and a 10 Hz control loop.Source: A. Experimental Setup · p. 5
- The rgb dataset supports lifting and stacking variable-shape rigid objects, while the deformable dataset supports cloth lifting; episodes last up to 200 steps.The datasets include teleoperated, target-task, and scripted-policy experience; their statistics are summarized in Table I.Source: A. Experimental Setup · p. 5, Table I · p. 5
- The random_watcher seeds training with randomized movements and gripper actions, and removing its data degrades final-agent performance.Its behavior is typically free-space movement, with occasional object picking or pushing.Source: A. Experimental Setup · p. 5
- Humans sketch rewards for 894 lift_green, 1,201 stack_green_on_red, and 585 lift_cloth episodes, annotating only a small portion of stored data.Sketches accumulate across iterations, increasingly using agent episodes for annotation.Source: A. Experimental Setup · p. 5
- Agents receive 400,000 update steps per training round, and policies are evaluated on the robot because offline learning alone cannot establish their real-world quality.Evaluation varies initial conditions and difficulty for lifting and stacking, while cloth-lifting initial conditions are randomized each trial.Source: A. Experimental Setup · p. 6
B. Results
The final agents perform lifting, stacking, and cloth handling, including under challenging object conditions. Ablations also show the value of heterogeneous experience and distributional value functions.
B. Results
- For adversarially positioned objects, lifting succeeded at 80% and stacking at 40%; the agents also handled perturbations and sometimes outperformed human operators in speed.The qualitative results include stacking unseen and non-standard objects and lifting toys.Source: B. Results · pp. 6–7, Figure 9 · p. 7
- The reported lifting and stacking rates compare favorably with Zhu et al. [73], though the authors caution that physical setups differ.The paper also contrasts its varied objects and 6-DoF control with prior fixed-object approaches.Source: B. Results · pp. 6–7
- Success rates were 80% for lifting, 60% for stacking, and 74% for cloth; unseen-object rates were 50% for lifting and 40% for stacking.Table II summarizes agent and ablation success rates across tasks and difficulty settings.Source: B. Results · p. 6, Table II · p. 7
- Task-specific-only training reduced performance to 0% in almost all scenarios, while random_watcher data contributed up to a 30% improvement.The largest random_watcher benefit occurred in the hardest unseen-object case.Source: B. Results · p. 7
- Distributional value functions were essential for good batch-RL performance in the reported comparison.Source: B. Results · p. 7, Table II · p. 7
C. Interactive Insertion
The interactive insertion experiment trains a policy from scratch while humans refine rewards online, rather than relying on historical data and batch RL. The agent achieves high success within a single working day.
C. Interactive Insertion
- The policy and reward model use wrist-camera images, and the task uses 6-DoF velocity control with a stateful safety controller.Episodes run for 15 seconds at 10 Hz; the safety controller limits excess forces.Source: C. Interactive Insertion · p. 7
- Training starts from 100 human demonstrations with sketches and randomized robot positions within a 6 × 6 × 6 cm region.The allowed workspace is 8 × 8 × 15 cm, with 17.2° rotation.Source: C. Interactive Insertion · p. 7
- The agent exceeded 80% USB-insertion success within 8 hours, with annotators supplying 65 ± 10 additional reward sketches.The experiment was repeated three times.Source: C. Interactive Insertion · pp. 7–8
- Using only vision during training yields insertion behavior robust to positional changes, including a significantly moved socket despite training on an unperturbed state.Source: C. Interactive Insertion · p. 8
IV. RELATED WORK
Prior robotics RL often relies on task-specific state estimation or engineered rewards, while demonstrations and large datasets address other parts of the learning problem. This paper positions reward learning over diverse stored experience as a way to extend that setting.
IV. RELATED WORK
- Robotics RL has often used task-specific state estimation or narrow raw-observation settings with engineered rewards; this work learns from raw observations without predefined rewards.Source: §IV · p. 8
- QT-Opt generalizes across objects using a large grasp dataset, but its system focuses on one task that is well suited to reward engineering and scripted collection.The authors contrast that scope with diverse data collection and learned reward functions.Source: §IV · p. 8
- Inverse reward learning has succeeded on states and engineered features, but applying it to high-dimensional raw pixels remains challenging.Source: §IV · p. 8
- Behavior cloning requires consistent, high-quality target-task demonstrations and cannot benefit from heterogeneous data; demonstrations can also enter model-free RL through temporal-difference learning.Source: §IV · p. 8
- Existing large-scale robot datasets support demonstration learning, online RL, or prediction, while this paper reports reward-labeling and batch-RL use on challenging domains.Source: §IV · p. 8
V. CONCLUSIONS
The paper combines reward learning, retrospective reward labeling, and offline batch RL, reporting versatile agents with generalization and robustness across tasks. It also identifies human-training costs, task limits for reward sketching, and sensitivity to setup perturbations.
V. CONCLUSIONS
- The approach combines reward learning, retrospective reward labeling, and batch RL with distributional value functions; the authors also release large datasets and canonical agents.They describe substantial engineering and innovation as necessary to implement the system at this scale.Source: §V · p. 8
- Across a wide set of tasks, learned agents show versatility, significant generalization, and robustness; the authors also identify diverse training data as an essential factor in success.They report that reward sketching effectively elicits rewards because humans judge progress toward a goal well, and that stored experience across tasks supports fully offline policy learning.Source: §V · p. 8
- The approach requires a human in the training loop, adding cost; reward sketching may not suit every task, and agents remain sensitive to significant setup perturbations.The authors identify these issues as directions for future work.Source: §V · p. 8