PaperlayerView exact revision on arXiv

Source-linked AI summary

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

Jianlan Luo, Charles Xu, Jeffrey Wu, Sergey Levine

arXiv:2410.21845v3cs.ROcs.AI

TL;DR

Real-world robotic manipulation still lacks practical methods for learning complex dexterous skills. HIL-SERL combines human-in-the-loop vision-based RL with demonstrations, corrections, and system-level design choices, achieving near-perfect success rates and faster cycle times across diverse tasks within 1–2.5 hours of training.

  • Problem

    Realizing RL’s promise for autonomous acquisition of complex, dexterous robotic manipulation skills in real-world settings remains challenging.

    Source: §1 · p. 1

  • Method

    HIL-SERL combines vision-based RL with human demonstrations and corrections, including sparse classifier-based rewards, to learn dexterous manipulation policies.

    Source: §1 · p. 1, §3.3 · p. 6

  • Results

    101% improvement in success rate and 1.8x faster cycle time than imitation-learning methods trained on the same human data, with near-perfect success on nearly all tasks after 1–2.5 hours.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Model-free RL can effectively learn a wide range of complex vision-based manipulation tasks directly in the real world within a practical timeframe.

    Source: §6 · p. 21

    The method’s generalization to unstructured environments was not tested, and its extension to significantly longer-horizon tasks remains uncertain.

    Source: §6 · p. 21

Abstract

from arXiv · show

Reinforcement learning (RL) holds great promise for enabling autonomous acquisition of complex robotic manipulation skills, but realizing this potential in real-world settings has been challenging. We present a human-in-the-loop vision-based RL system that demonstrates impressive performance on a diverse set of dexterous manipulation tasks, including dynamic manipulation, precision assembly, and dual-arm coordination. Our approach integrates demonstrations and human corrections, efficient RL algorithms, and other system-level design choices to learn policies that achieve near-perfect success rates and fast cycle times within just 1 to 2.5 hours of training. We show that our method significantly outperforms imitation learning baselines and prior RL approaches, with an average 2x improvement in success rate and 1.8x faster execution. Through extensive experiments and analysis, we provide insights into the effectiveness of our approach, demonstrating how it learns robust, adaptive policies for both reactive and predictive control strategies. Our results suggest that RL can indeed learn a wide range of complex vision-based manipulation policies directly in the real world within practical training times. We hope this work will inspire a new generation of learned robotic manipulation techniques, benefiting both industrial applications and research advancements. Videos and code are available at our project website https://hil-serl.github.io/.

1. Introduction

HIL-SERL is a human-in-the-loop, vision-based RL system designed to efficiently learn precise and dexterous manipulation skills in the real world. Across challenging single- and dual-arm tasks, it achieves near-perfect performance in practical training times while substantially outperforming imitation learning and prior approaches.

  • HIL-SERL combines a pretrained visual backbone, sample-efficient off-policy RL with human demonstrations and corrections, and a safety-oriented low-level controller.Human corrections are incorporated off-policy and are particularly important for learning challenging tasks that are difficult to acquire from scratch.Source: §1 · pp. 1–2
  • The system handles dynamic manipulation, precision assembly, object handover, and dual-arm coordination across tasks with complex dynamics, high-dimensional spaces, and long horizons.Examples include pan flipping, Jenga-block whipping, motherboard and timing-belt assembly, IKEA-shelf construction, and car-dashboard assembly.Source: Figure 1 · p. 2, §1 · p. 2
  • 101% higher success rate and 1.8x faster cycle time than imitation learning, with near-perfect success on almost all tasks after 1–2.5 hours of real-world training.These results demonstrate that RL can learn complex vision-based manipulation policies directly in the real world within practical training times.Source: §1 · p. 3
  • The paper compares HIL-SERL with state-of-the-art RL methods and ablates its components to explain their contributions, including how RL supports reactive and open-loop control strategies.The analysis argues that the empirical gains result from carefully integrating the system components and examines why RL achieves near-perfect success and general-purpose policy flexibility.Source: §1 · p. 3
  • The work reports, to the authors’ knowledge, the first real-world RL dual-arm coordination from image inputs, along with Jenga whipping and timing-belt assembly.These results extend RL to tasks previously considered infeasible or nearly insurmountable with existing methods.Source: §1 · pp. 2–3

2. Related Work

Prior real-world RL and dexterous manipulation systems provide important foundations but often require long training, task-specific engineering, or limited sensing and task complexity. This work integrates existing methods with demonstrations and human corrections to learn general-purpose, closed-loop vision-based manipulation policies across more challenging tasks.

Algorithms and systems for real-world RL

  • Real-world RL methods improve sample efficiency, reward specification, and reset-free learning, but often require long training; this framework integrates existing methods rather than introducing new algorithms.Binary classifier-based rewards with demonstrations are effective for the paper’s complex tasks.Source: §2 · pp. 3–4
  • Compared with SERL (Luo et al., 2024a), this system adds human corrections to demonstrations and addresses longer-horizon dual-arm and dynamic manipulation tasks.The authors report that corrections are crucial for learning from mistakes, especially on tasks difficult to learn from scratch.Source: §2 · p. 4

Dexterous robotic manipulation

  • Prior insertion methods use state-based models, task-specific compliant tooling, or multistage visual servoing, whereas this method learns task-relevant features and visuomotor policies in a tight closed-loop perception-action system.The learned policy is characterized as output feedback control and supports reactive, high-precision tasks.Source: §2 · p. 4
  • Prior dynamic manipulation studies use motion capture, dynamic primitives, or quasi-dynamic pushing, while this system uses pixel inputs and tackles harder whip-based removal and reactive two-arm timing-belt assembly.The timing-belt task requires precise coordination to adjust the tensioner and belt dynamically, distinguishing it from earlier cable-manipulation work.Source: §2 · p. 4

3. Human-in-the-Loop Reinforcement Learning System

The system combines sparse-reward vision-based RL with distributed replay, modular robot control, pretrained visual representations, and corrective human interventions. Training uses demonstrations, a learned binary reward classifier, and progressively reduced intervention as policy performance improves.

3.1. Preliminaries and Problem Statement

  • The method builds on RLPD, which equally samples prior and on-policy data to update a parametric Q-function and policy for sample-efficient robotic RL.The policy is modeled as a Gaussian distribution over actions conditioned on observations, and the objective maximizes expected discounted reward and trajectory success.Source: §3.1 · p. 5

3.2. System Overview

  • The distributed system separates actor, learner, and replay-buffer processes, combining 20–30 human demonstrations with on-policy data and supporting modular cameras, teleoperation, controllers, and robot arms.The learner samples equally from demonstration and RL buffers, applies RLPD, and periodically sends updated policies to the actor.Source: §3.2 · p. 5, Figure 2 · p. 6

3.3. System Design Choices

  • Key design choices include pretrained ResNet-10/ImageNet visual features, sparse binary rewards from offline-trained classifiers, relative proprioception, safety-oriented control, and a separate discrete gripper critic.The gripper component solves continuous and discrete action MDPs separately, using DQN-style argmax selection for actions such as open, close, and stay.Source: §3.3 · pp. 6–7

3.4. Human-in-the-Loop Reinforcement Learning

  • Human operators intervene through a SpaceMouse when policies become unrecoverable or stuck, storing intervention transitions in both buffers while retaining non-intervention policy transitions in the RL buffer.Interventions are initially frequent and become less frequent as the policy improves; targeted corrections combined with continued autonomous exploration accelerate learning.Source: §3.4 · pp. 7–8, Figure 3 · p. 8

3.5. Training Process

  • Training selects task-appropriate cameras, crops and resizes images to 128x128, trains the reward classifier from roughly 200 positive and 1000 negative samples, and initializes replay with 20–30 demonstrations.Reward-classifier data collection takes around 5 minutes, and the classifier generally exceeds 95% evaluation accuracy; overly long successful interventions can destabilize early value estimates.Source: §3.5 · pp. 8–9

4. Experiment Results

Experiments across seven diverse manipulation tasks evaluate HIL-SERL’s integration of human-guided reinforcement learning with task-specific sensing and control in precise, bimanual, flexible-object, dynamic, open-loop, and closed-loop settings. Additional comparisons assess its performance against diffusion-policy and imitation-based baselines on reactive precision-manipulation tasks.

4.1. Overview of Experiments

  • The experiments covered seven tasks using single- or dual-arm systems, combining visual, proprioceptive, force/torque, gripper, twist, wrench, and discrete gripper-action spaces.Tasks included dynamic manipulation, precision assembly, flexible-object handling, multi-stage assembly, dual-arm coordination, and both reactive and predictive control.Source: §4.1 · p. 9

4.2. Description of Tasks

  • The task suite included contact-rich motherboard assembly, chained IKEA shelf assembly, bimanual dashboard and handover tasks, timing-belt handling, Jenga whipping, and object flipping.Motherboard assembly chained RAM, SSD, USB grasp-insertion, and USB clipping subtasks; long-horizon IKEA assembly used three chained policies with randomized grasp poses and two attempts per subtask.Source: SSD Assembly · pp. 11–12

4.3. Experimental Results

  • HIL-SERL achieved a 100% success rate within 1–2.5 hours of real-world training on nearly all tasks, versus HG-DAgger’s 49.7% average success rate, while average cycle time fell from 9.6 s to 5.4 s.Source: §4.3 · p. 15
  • Intervention rates and durations decreased during HIL-SERL training, whereas HG-DAgger required more frequent corrections, indicating progressively greater policy autonomy.HIL-SERL reached 100% success and zero intervention rate on representative learning curves, while HG-DAgger’s success and intervention trends were inconsistent.Source: Figure 5 · p. 14, §4.3 · p. 15

4.4. Robustness Results

  • The learned policies handled disturbances through moving-target insertion, reactive deformable-belt manipulation, retrying after forced gripper openings, and releasing and regrasping poorly positioned objects.These behaviors were demonstrated qualitatively in RAM insertion, timing-belt assembly, dashboard assembly, object handover, and USB connector grasp-insertion.Source: §4.4 · p. 16, Figure 6 · p. 16

4.5. Additional Baseline Comparisons

  • RL from scratch achieved 0% success on all tested tasks, while demonstrations without online corrections remained substantially worse than HIL-SERL, including 0% on car dashboard assembly.Increasing offline demonstrations from 20 to 200 did not replace the benefit of online human corrections.Source: §4.5 · pp. 16–17
  • BC, residual RL, IBRL, and DAPG underperform on precise or reactive tasks because they depend more heavily on imperfect demonstrations or BC-like behavior.The limitation is especially important for car dashboard assembly, RAM insertion, and other tasks requiring closed-loop reactive behavior.Source: §4.5 · p. 17
  • Off-policy learning dynamically weights human data by relevance, using demonstrations early while allowing autonomous exploration to surpass human-level performance.This balances bootstrapping from demonstrations against avoiding constraint by their limitations.Source: §4.5 · p. 17
  • 27%, 28%, and 56% success rates for Diffusion Policy on RAM insertion, car dashboard panel, and object flipping were significantly below our method and HG-DAgger.Diffusion Policy used 200 demonstrations per task, compared with 20 demonstrations in our method’s offline replay buffers.Source: §4.5 · p. 17

5. Result Analysis

The analysis attributes HIL-SERL’s 100% success across tasks to RL-driven self-correction and funnel-like policy robustification around successful behaviors. It also shows that the unified method learns reactive policies for precision tasks and predictive policies for dynamic manipulation.

5.1. Reliability of the Learned Policies

  • 100% success across all tasks demonstrates HIL-SERL’s high reliability, attributed to reinforcement learning’s self-correction from successes and failures.The analysis contrasts this mechanism with imitation learning, which lacks comparable self-correction with the same amount of data.Source: §5.1 · p. 18
  • Funnel-shaped state visitation progressively concentrates around successful trajectories, where critical states gain both higher Q-values and higher Q-value variance.In RAM insertion, this narrowing funnel connects initial states to the target and indicates increasing policy confidence and precision; Fig. 7 visualizes the training dynamics.Source: §5.1 · pp. 18–19, Figure 7 · p. 18
  • Compared with HIL-SERL, HG-DAgger produces a sparser, more diffuse visitation distribution because it explores only around its current policy.The analysis states that matching RL performance may therefore require more demonstrations, corrections, and human attention to data quality.Source: Figure 7 · p. 18, §5.1 · p. 19

5.2. Reactive Policy and Predictive Policy

  • The learned policies distinguish predictive and reactive behavior: Jenga whipping maintains near-zero action variance, whereas RAM insertion starts near 0.6 and rapidly decreases near the target.Precision assembly tasks require continued sensory-feedback corrections, including breaking and re-establishing contact, while dynamic tasks favor consistent open-loop execution; Fig. 8 illustrates these patterns.Source: §5.2 · p. 19, Figure 8 · p. 20
  • Through environment interaction, the method acquires reactive contact-handling behaviors without explicitly formulating them and learns both reactive and predictive policies within one framework.This adaptability lets the system address manipulation tasks with different behavioral requirements.Source: §5.2 · pp. 19–20

6. Discussion

The discussion concludes that model-free RL can learn diverse, high-performing vision-based manipulation policies directly in the real world within practical timeframes. It highlights applications in flexible manufacturing, future extensions for data generation and faster adaptation, and limitations in long-horizon tasks and unstructured environments.

  • The approach advances robotic manipulation by achieving nearly perfect success rates and faster cycle times than alternative approaches, including imitation learning.These results demonstrate that model-free RL can effectively tackle complex perception-based manipulation tasks through direct real-world training.Source: §6 · p. 21
  • The framework could generalize across manipulation skills and support High-Mix Low-Volume manufacturing, where customization, agility, and flexibility are important.Potential application areas include electronics, semiconductors, automotive, and aerospace production.Source: §6 · p. 21
  • Short, largely autonomous training could generate data for robot foundation models, while pretrained value functions could reduce task-specific training from scratch.Converged policies can be executed to collect data for distillation, and general manipulation value functions can be fine-tuned for specific tasks and embodiments.Source: §6 · p. 21
  • The method’s extension to significantly longer-horizon tasks remains uncertain because sample complexity becomes more pronounced.Improved pretraining or automatic decomposition into shorter subtasks could help address this limitation.Source: §6 · p. 21
  • The experiments used limited randomization and did not test generalization in unstructured environments, leaving robustness under these conditions unresolved.Longer training with desired randomization and vision foundation models pretrained on diverse datasets are proposed as possible remedies.Source: §6 · p. 21

Supplementary Materials … A.1.1. Cropped Images

The supplementary materials describe task setup and policy-training details, including the motherboard-assembly hardware arrangement and task-focused image preprocessing. The setup is documented with hardware and sample camera-input figures.

Supplementary Materials

A. Task Setup and Policy Training Details

A.1. RAM Insertion

A.1.1. Cropped Images

A.1.2. Policy Training Details · A.2. SSD Assembly · A.2.1. Cropped Images

This section documents policy-training details for RAM insertion, the motherboard-assembly hardware setup, and task-focused image cropping for policy inputs.

A.1.2. Policy Training Details

  • Policy training details for the RAM insertion task are reported in Table 2.The section introduces Table 2 as the source of additional training information.Source: §A.1.2 · p. 33
  • Table 2 presents policy training details specifically for the RAM insertion task.The table’s scope is defined by its caption.Source: Table 2 · p. 33

A.2. SSD Assembly

  • Figure 9 shows the hardware setup used for the motherboard assembly task.The setup includes the robot, camera placements, and task arrangement.Source: §A.2 · p. 34
  • The motherboard-assembly hardware description covers the robot used in the task.The robot is one of the setup elements identified in Figure 9.Source: §A.2 · p. 34
  • The setup description also specifies camera placements and the task arrangement for motherboard assembly.These elements are presented together in Figure 9.Source: §A.2 · p. 34

A.2.1. Cropped Images

  • Task-relevant image regions were isolated by cropping the input images, as shown in Figure 11.Figure 11 contains sample input images from the cameras used by the policy.Source: §A.2.1 · p. 34, Figure 11 · p. 34

A.2.2. Policy Training Details · A.3. USB Grasp-Insertion · A.3.1. Cropped Images

This block documents policy-training details for SSD assembly, the motherboard-assembly hardware setup, and image cropping used to focus policy inputs on task-relevant scene regions.

A.2.2. Policy Training Details

  • Table 3 reports additional policy-training details for the SSD assembly task.Source: §A.2.2 · p. 34
  • Table 3 is specifically scoped to policy-training details for SSD assembly.Source: Table 3 · p. 34

A.3. USB Grasp-Insertion

  • Figure 9 presents the motherboard-assembly hardware setup.Source: §A.3 · p. 35
  • The hardware setup includes the robot, camera placements, and task arrangement.Source: §A.3 · p. 35

A.3.1. Cropped Images

  • Images were cropped to focus on task-relevant parts of the scene.Source: §A.3.1 · p. 35
  • Figure 12 shows sample input images from cameras used by the policy.Source: Figure 12 · p. 35

A.3.2. Policy Training Details … A.11.2. Policy Training Details

The appendix documents hardware setups, task-focused image cropping, and policy-training details across USB insertion, cable clipping, IKEA assembly, dashboard assembly, object handover, timing-belt assembly, Jenga whipping, and object flipping tasks.

A.3.2. Policy Training Details

  • USB grasp and insertion policy-training details are reported in Table 4.The table is identified as covering the USB grasp and insertion task.Source: §A.3.2 · p. 35, Table 4 · p. 35

A.4. Cable Clipping; A.4.1. Cropped Images; A.4.2. Policy Training Details

  • The passages support task-focused image cropping and policy-training details for cable clipping; Figure 9 instead shows the robot, camera placements, and task arrangement for motherboard assembly.Figure 13 shows sample input images from cameras used as policy inputs.Source: §A.4.1 · p. 36, §A.4.2 · p. 36, Table 5 · p. 36

A.5. IKEA - Side Panel; A.5.1. Policy Training Details

  • IKEA side-panel assembly is documented through a hardware setup and policy-training details reported in Table 6.The hardware description covers the robot, camera placements, and task arrangement.Source: §A.5 · p. 37, Figure 14 · p. 37, §A.5.1 · p. 37

A.6. IKEA - Top Panel; A.6.1. Policy Training Details

  • IKEA top-panel assembly has policy-training details reported in Table 7, with the hardware setup described as showing the robot, camera placements, and task arrangement.The hardware setup is shown in Figure 14.Source: §A.6 · p. 39, §A.6.1 · p. 39, Table 7 · p. 39

A.7. Car Dashboard Assembly; A.7.1. Cropped Images; A.7.2. Policy Training Details

  • Car dashboard assembly uses a hardware setup, task-focused cropped images, and policy-training details reported in Table 8.Figures 15 and 16 present the hardware arrangement and sample camera inputs, respectively.Source: §A.7 · p. 40, Figure 15 · p. 40, §A.7.1 · p. 40

A.8. Object Handover; A.8.1. Cropped Images; A.8.2. Policy Training Details

  • Object handover uses a hardware setup, task-focused cropped images, and policy-training details reported in Table 9.Figures 17 and 18 show the hardware arrangement and sample camera inputs, respectively.Source: §A.8 · p. 42, Figure 17 · p. 42, §A.8.1 · p. 42

A.9. Timing Belt Assembly; A.9.1. Cropped Images; A.9.2. Policy Training Details; A.10. Jenga Whipping; A.10.1. Cropped Images; A.10.2. Policy Training Details

A.10. Jenga Whipping; A.10.1. Cropped Images; A.10.2. Policy Training Details; A.11. Object Flipping; A.11.1. Cropped Images; A.11.2. Policy Training Details

B. Reward Classifier Training Details

Reward classifiers use a ResNet-10 feature extractor with a two-layer MLP and are trained on teleoperated, SpaceMouse-labeled images using cross-entropy loss. Training uses Adam for 100 iterations, with additional false-positive and false-negative samples collected for some tasks.

B. Reward Classifier Training Details

  • Reward classifiers combine a pre-trained ResNet-10 feature extractor with a two-layer MLP, trained using cross-entropy loss and Adam at a 3e-4 learning rate for 100 iterations.The network was trained on the collected dataset.Source: B. Reward Classifier Training Details · p. 50
  • Teleoperation with a SpaceMouse produced labeled images, assigning 1 to successful task-completion images and 0 otherwise.Some tasks also included additional false-positive and false-negative samples to improve classifier performance; Fig. 25 shows RAM insertion examples.Source: B. Reward Classifier Training Details · p. 50, Figure 25 · p. 50

C. Diffusion Policy Training Details … D.2. Robot Controller

The appendix specifies diffusion-policy baseline training details and describes proprioceptive representations and robot controllers for stable real-world execution. It uses relative end-effector coordinates, frame transformations, task-specific actions, and high-frequency low-level control with bounded pose errors.

C. Diffusion Policy Training Details

D.1. Proprioceptive Information Representation

  • Proprioceptive information is represented relative to each episode’s randomized initial end-effector frame, enabling transfer across locations with consistent relative geometry.This transformation prevents overfitting to specific global locations in the robot base frame.Source: §D.1 · p. 52
  • The policy normally outputs a 6 DoF twist in its observation frame, which is mapped into the robot base frame through the adjoint transformation.The controller software expects actions expressed in the base frame.Source: §D.1 · p. 52
  • For two dynamic manipulation tasks, the policy instead outputs a 3 DoF feedforward wrench that is converted to joint torques using the Jacobian transpose.The wrench is expressed in the current observation frame and sent as setpoints to the low-level controller.Source: §D.1 · p. 52

D.2. Robot Controller

  • Most tasks use an impedance controller running at 1000 Hz and accepting 10 Hz policy-computed setpoints, with additional treatment for contact-rich training stability.The controller converts its objective into joint torques and includes nullspace torques.Source: §D.2 · p. 52
  • Bounding pose error to |e| ≤ Δ limits interaction forces without directly reducing controller gains, preserving accuracy while constraining contact forces.The resulting spring-damper force bound is k_p⋅|Δ| + 2k_d⋅|Δ|⋅f, where f is the control frequency.Source: §D.2 · p. 53
  • For the two dynamic tasks, a feedforward wrench controller runs at 1000 Hz, accepts 10 Hz policy setpoints, and converts desired wrenches into joint torques.Conversion uses the Jacobian transpose and adds nullspace torques.Source: §D.2 · p. 53

E. Policy Training Plots

This section presents additional HIL-SERL policy-training plots for all tasks, showing rapid success-rate gains alongside decreasing cycle time and intervention rate.

E. Policy Training Plots

  • HIL-SERL learning curves across all experimental tasks reached 100% success and 0% intervention rate as cycle time progressively decreased.Metrics are shown as running averages over 20 episodes.Source: Figure 26 · p. 54

Related papers

Loading 2410.21845v3…