PaperlayerView exact revision on arXiv

Source-linked AI summary

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, Majid Khadiv

arXiv:2609.30023v1cs.RO

TL;DR

Demonstration-trained manipulation policies can fail in states poorly covered by training data, while additional demonstrations and full-policy online learning can require substantial human effort. Res-HIL learns residual corrections to a frozen imitation policy, using interventions for direct supervision and reward shaping; across five contact-rich tasks it improves pretrained policies, particularly on challenging long-horizon tasks.

  • Problem

    Demonstration-trained policies may encounter states poorly represented in training data, while collecting additional demonstrations takes substantial human effort and full-policy HIL learning can require extensive interaction and supervision.

    Source: §I · p. 1

  • Method

    Res-HIL keeps a pretrained manipulation policy fixed and learns residual corrections, using human actions for direct residual supervision and intervention occurrence to shape rewards.

    Source: §I · pp. 1–2

  • Results

    Across five contact-rich tasks, Res-HIL improves pretrained policies and is particularly beneficial on challenging long-horizon tasks.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The results suggest pretrained policies and online human feedback provide complementary information for efficient robot policy adaptation.

    Source: §VI · p. 8

    The conclusion identifies scaling Res-HIL to more diverse tasks and reducing required human supervision as future work.

    Source: §VI · p. 8

Abstract

from arXiv · show

Imitation learning enables robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.

I. INTRODUCTION

Demonstration-trained manipulation policies can fail in poorly represented states, while collecting more demonstrations is costly and full-policy online learning can demand substantial interaction and supervision. Res-HIL refines a frozen imitation policy with residual corrections informed by human interventions, and improves pretrained policies on five contact-rich tasks, especially challenging long-horizon ones.

I. INTRODUCTION

  • Demonstration-trained policies may encounter poorly represented states, and collecting more demonstrations requires substantial human effort without targeting deployed-policy failures.Full-policy HIL reinforcement learning can also require substantial robot interaction and human supervision as task complexity and horizon grow.Source: §I · p. 1
  • Res-HIL keeps the pretrained policy fixed and learns residual corrections, using human actions to supervise the residual and intervention occurrence to shape rewards.The approach targets failures of the pretrained policy while building on behavior already captured by its base policy.Source: §I · pp. 1–2
  • Across five contact-rich tasks, Res-HIL improves pretrained policies and is particularly beneficial on challenging long-horizon tasks.Ablations report that direct human-data supervision and intervention-aware reward shaping improve training and intervention efficiency.Source: §I · pp. 1–2

II. RELATED WORK

Res-HIL differs from prior HIL and residual-learning approaches by keeping the pretrained imitation policy fixed and using interventions both to supervise residual corrections and to shape rewards. This combines human guidance with residual adaptation without requiring an auxiliary visual reward model.

A. Human-in-the-Loop Reinforcement Learning

  • Unlike HIL methods that optimize the complete task policy, Res-HIL keeps a pretrained imitation policy fixed and restricts online learning to residual corrections.Prior approaches also use interventions to shape learning, but Res-HIL combines intervention timing for reward shaping with direct supervision from human actions.Source: A. Human-in-the-Loop Reinforcement… · pp. 2–3

B. Residual Reinforcement Learning

  • Residual learning can refine fixed imitation policies, but existing approaches rely primarily on autonomous exploration to discover useful corrections.Res-HIL instead uses targeted interventions to identify corrective residual actions in states where the base policy fails.Source: B. Residual Reinforcement Learning · p. 3

C. Human-in-the-Loop Residual Reinforcement Learning

  • Res-HIL derives learning signals directly from interventions rather than requiring the auxiliary visual reward model used by FTC.Unlike HiL-ResRL, it explicitly constructs a residual target from the full human action relative to the frozen base policy; intervention timing also penalizes preceding autonomous transitions.Source: C. Human-in-the-Loop Residual… · p. 3

III. METHOD

The method combines a demonstration-pretrained, frozen base policy with an online residual actor; the residual actor is optimized while the base policy remains unchanged.

III. METHOD

  • The base policy πBC is pretrained from demonstrations and remains frozen during online learning.The policy receives observations consisting of camera images and proprioceptive measurements, and the residual actor πθ produces the residual action.Source: A. Problem Formulation · p. 3
  • The method specifies a command sent to the robot, but the supplied passage does not state its composition.The nearby method text establishes a frozen base policy and residual actor, without giving the command equation here.Source: A. Problem Formulation · p. 3
  • During online learning, TD3 optimizes the residual actor while the pretrained base policy remains frozen.This preserves the division between the fixed demonstration-trained policy and the learned residual component.Source: A. Problem Formulation · p. 3

B. Human Interventions

Human operators intervene when motion risks failure or unrecoverable task deviation and release control after recovery. The method initializes residual corrections at zero and shapes preceding autonomous rewards around interventions.

B. Human Interventions

  • Operators intervene when motion risks failure, damage, or unrecoverable deviation, then release control once the system is recoverable.This allows the learner to experience correct recovery behavior without continuous teleoperation.Source: B. Human Interventions · p. 3

C. Residual Policy Initialization

  • Zero-initializing the residual output layer makes its initial correction zero, preserving the pretrained policy’s behavior at training start.Only the output layer is zero-initialized; preceding layers retain standard random initialization.Source: C. Residual Policy Initialization · pp. 3–4

D. Reward Shaping

  • Intervention-aware shaping applies a decaying penalty to the I preceding transitions, with the strongest penalty immediately before intervention.Penalties apply only before the first K intervention segments to limit cumulative shaping; failed episodes also receive a terminal penalty.Source: D. Reward Shaping · p. 4, Figure 3 · p. 4

E. Optimization Objective

The actor objective combines TD3 learning with residual-action regularization and behavior cloning. The cloning term uses demonstrations and online human corrections to preserve demonstrated behavior while incorporating interventions.

E. Optimization Objective

  • Twin TD3 critics estimate the return of executed actions, with a target action formed from the combined base and residual policies.Target-network parameters use Polyak averaging, and target actions include clipped policy-smoothing noise.Source: E. Optimization Objective · p. 4
  • Residual-action regularization favors small corrections to the base policy.The residual actor is optimized according to the value of the combined action.Source: E. Optimization Objective · p. 4
  • Behavior cloning uses demonstration and intervention samples to preserve initial demonstrations while incorporating online human corrections.For demonstrated or intervened actions, the target residual is defined relative to the frozen base policy.Source: E. Optimization Objective · p. 4
  • The actor objective combines the TD3 objective, residual-action regularization, and behavior cloning.The residual and cloning terms are weighted by λres and λBC, respectively.Source: E. Optimization Objective · p. 5

A. Robot System and Tasks

The experiments use a UR5 with camera and proprioceptive observations across randomized peg-in-hole, cable-routing, and vent-insertion tasks. Res-HIL is compared with four baselines under specified demonstration and evaluation conditions.

A. Robot System and Tasks

  • Experiments use a UR5, teleoperation space mouse, two gripper-mounted cameras, proprioception, and relative Cartesian end-effector actions.Visual inputs use an ImageNet-pretrained ResNet-10.Source: A. Robot System and Tasks · p. 5
  • The evaluation covers randomized peg-in-hole alignment, sequential cable routing and insertion, and ordered vent insertion.Cable routing is the principal long-horizon task; its two-camera setting omits the obstacle from view, while a third camera provides a global view.Source: A. Robot System and Tasks · p. 5

B. Baselines

  • Res-HIL is compared with frozen ACT, residual RL without interventions, full-policy HIL-SERL, and a naive residual-HIL baseline.Most methods use 20 demonstrations; ACT and ResFiT use 100 demonstrations.Source: B. Baselines · p. 5
  • Concurrent FTC and HiL-ResRL were excluded because controlled real-robot reproduction was infeasible within the available evaluation period.The online methods are compared by elapsed training time and environment transitions, with human-generated transitions reported separately.Source: B. Baselines · p. 5

C. Evaluation Protocol

Across five tasks, Res-HIL delivers high final success from 20 demonstrations and improves on 100-demonstration ACT policies. The evaluation also reports strong 10-minute results and low cycle times for successful episodes.

C. Evaluation Protocol

  • Online training experiments were repeated three times; performance was evaluated on the same 50 predefined randomized initial configurations across methods and repetitions, with success rates averaged over repetitions.Collecting 100 demonstrations took approximately 30 minutes for peg-in-hole, 45 minutes for vent insertion, and 60 minutes for cable manipulation.Source: C. Evaluation Protocol · p. 5

A. Task Success

  • After 10 minutes, Res-HIL has the highest success rate on all five tasks, reaching 100% on easy peg-in-hole and 92% on three-camera cable.Source: A. Task Success · p. 5, Table I · p. 6
  • Res-HIL reaches 100% success on three of five tasks and at least 88% on every task, outperforming 100-demo ACT across all five from 20 demos.Source: A. Task Success · pp. 5–6, Table II · p. 6
  • 5.46 s is Res-HIL’s average cycle time across five tasks, versus 5.83 s for ACT with 100 demonstrations and 7.73 s for ACT with 20.Res-HIL maintains low cycle times while achieving high success rates; HIL-SERL’s faster insertion trajectories do not include cable tasks it did not solve.Source: A. Task Success · p. 6, Table III · p. 6

B. Training and Human Intervention Efficiency

On peg-in-hole tasks, Res-HIL adapts faster than other online methods and reaches high success without a higher proportion of human-generated transitions. Intervention needs decline toward zero on the easy task but persist longer on the hard task.

B. Training and Human Intervention Efficiency

  • On easy peg-in-hole, Res-HIL exceeds 90% success within minutes and reaches 100% shortly after; on the hard variant, it reaches high success earlier than HIL-SERL.Both methods eventually achieve comparable final performance on the hard variant.Source: B. Training and Human Intervention… · p. 6, Figure 4 · p. 7
  • The easy-task intervention ratio approaches zero as training progresses; it remains necessary longer on the hard task but is generally lower than HIL-SERL’s.Because human and autonomous transitions have equal duration, the intervention ratio reflects the fraction of interaction time under human control; faster convergence does not require a higher proportion of human-generated transitions.Source: B. Training and Human Intervention… · pp. 6–7, Figure 5 · p. 7

C. Ablation Study

On easy peg-in-hole, ablations show that zero initialization speeds convergence, while reward shaping and especially BC loss support learning. Across five tasks, Res-HIL is particularly beneficial on challenging long-horizon tasks.

C. Ablation Study

  • The ablation isolates zero initialization, intervention-aware reward shaping, and BC loss on easy peg-in-hole; Fig. 6 tracks intervention ratio.This setting was chosen to isolate component effects while keeping real-world interaction tractable.Source: C. Ablation Study · p. 7, Figure 6 · p. 7
  • Without zero initialization, convergence takes 16 rather than 8 minutes at 100% final success; without reward shaping, it misses 30-minute convergence at 92%, and without BC loss, success is 20%.The component ablation is reported for easy peg-in-hole.Source: Table IV · p. 7, C. Ablation Study · p. 7

VI. CONCLUSION

  • Across five contact-rich tasks, Res-HIL is particularly beneficial on challenging long-horizon tasks where the base policy is insufficient and learning the complete behavior online is difficult.Source: §VI · p. 8
  • Future work will investigate scaling Res-HIL to more diverse tasks and reducing the required human supervision.Source: §VI · p. 8

APPENDIX

The appendix table lists hyperparameters used in all experiments.

  • The appendix includes a table of hyperparameters used in all experiments.Source: Table V · p. 8
Loading 2609.30023v1…