PaperlayerView exact revision on arXiv

Source-linked AI summary

ReF-HIL: Shaping the Critic around Human Action Neighborhoods for Efficient Human-in-the-Loop Reinforcement Learning

Shaoyin Luo, Song Wang, Shibo Xia, Tianle Zhang, Zhaowei Liang, Guanghui Shen, Bin Wang, Dan Wu

arXiv:2609.37131v1cs.RO

TL;DR

Real-world HIL-RL can underuse successful human experience in value learning, while persistent imitation penalties can limit value-driven improvement. ReF-HIL combines an independent reference from successful experience with a human-action neighborhood for policy optimization. Across five real-world manipulation tasks, it improves overall learning efficiency and success over evaluated baselines, reaching 90% autonomous success in 18–63 minutes.

  • Problem

    Real-world robot RL requires costly trials, failure recovery, and resets, while underusing successful human experience can require more interaction and persistent imitation penalties can limit value-driven improvement.

    Source: §I · p. 1

  • Method

    ReF-HIL learns an independent value reference from successful human experience and uses a learned human-action neighborhood to guide value-driven policy optimization without imitation penalties inside it.

    Source: §I · pp. 1–2, B. Reinforcement Learning with… · p. 2

  • Results

    Across five real-world manipulation tasks, ReF-HIL improves overall learning efficiency and success over evaluated baselines, reaching 90% autonomous success in 18–63 minutes and final success rates of 91.7–100%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The findings support using human experience to accelerate value learning while preserving value-driven policy improvement within a learned action neighborhood.

    Source: §VI · p. 8

Abstract

from arXiv · show

Human-in-the-loop reinforcement learning (HIL-RL) offers a promising route to efficient training of robotic manipulation policies by combining autonomous learning with human demonstrations and online corrections. However, insufficient use of successful human experience in value learning prolongs costly real-world training, while persistent imitation penalties can limit value-driven policy improvement. To address these limitations, we propose ReF-HIL, an efficient HIL-RL framework that uses human guidance to accelerate the learning process. Human-Reference-Guided Value Shaping learns an independent value reference from successful human experience to guide online value learning, while incorporating local corrective feedback. A Human Action Fence defines a learned human-action neighborhood, allowing value-driven optimization for better performance without imitation penalties inside while constraining policy and value updates outside. Experiments on five diverse and challenging real-world manipulation tasks demonstrate improved overall learning efficiency and higher success rates compared with the evaluated baselines. Specifically, ReF-HIL reaches 90% autonomous success in only 18-63 minutes of active training and achieves final success rates of 91.7-100%. These results highlight the potential of human-guided reinforcement learning to acquire reliable manipulation skills efficiently in the real world. Project website: https://anonymous.4open.science/w/ReF-HIL-7762/

I. INTRODUCTION

Real-world robotic RL requires costly trials, failure recovery, and resets, making efficient use of experience essential. ReF-HIL combines an independent reference from successful human experience with a Human Action Fence to guide value learning and policy improvement.

I. INTRODUCTION

  • Physical-robot RL entails costly trials, failure recovery, and environment resets, so efficient experience use matters for learning manipulation policies.HIL-RL combines demonstrations for initial experience with online human corrections that guide exploration toward successful behavior.Source: §I · p. 1
  • Underusing successful demonstrations and their outcomes may require more real-world interaction, while persistent imitation penalties can limit value-driven improvement when human actions are suboptimal.Local acceptance or rejection feedback may not reveal which behaviors lead to task completion.Source: §I · p. 1
  • ReF-HIL learns an independent value reference from successful human experience to help prevent online critics from undervaluing reference actions, while interventions lower values of preceding autonomous actions.The Human Action Fence removes imitation penalties inside the learned human-action neighborhood and constrains policy and value updates outside it.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Across five real-world tasks, ReF-HIL reaches 90% autonomous success in 18–63 minutes and achieves final success rates of 91.7–100%, outperforming evaluated baselines in efficiency and success.The framework combines value shaping and a human-action neighborhood, with ablations and focused analyses examining both components.Source: §I · p. 2

A. Human-in-the-Loop Reinforcement Learning

Prior HIL-RL methods use demonstrations and corrections to guide learning, while behavioral priors can become restrictive when policies need to depart from suboptimal human actions. The paper places these ideas in an autonomous manipulation MDP with human demonstrations and online interventions.

A. Human-in-the-Loop Reinforcement Learning

  • HIL-SERL combines demonstrations and online corrections with sample-efficient off-policy RL, while other methods use intervention feedback to penalize or discourage actions,, [17], [18].Interventions express a human judgment that the robot should change its current behavior.Source: A. Human-in-the-Loop Reinforcement… · p. 2
  • Successful human outcomes offer value guidance, but Max-Q can raise preceding autonomous mistakes after human recovery [10]; PACT lowers identified suboptimal segments [9].ReF-HIL instead learns an independent reference from successful human experience to supervise online reference-action values.Source: A. Human-in-the-Loop Reinforcement… · p. 2

B. Reinforcement Learning with Behavioral Priors

  • Behavioral priors can direct exploration,, but persistent imitation penalties may oppose beneficial deviations,; the Human Action Fence constrains updates outside a learned neighborhood while allowing value optimization inside.The neighborhood is learned from demonstrations and online interventions.Source: B. Reinforcement Learning with… · p. 2

III. PROBLEM FORMULATION

  • The manipulation task is an approximate MDP with visual and proprioceptive states, robot actions and dynamics, and rewards that favor success and otherwise impose a step cost.The objective is to maximize expected discounted return under autonomous execution; rewards are zero after successful termination.Source: §III · p. 2
  • Training begins with demonstrations and continues through autonomous interaction with brief human corrections or takeovers, recording a human-control indicator for each transition.Fixed-horizon action values reuse experience to supervise return estimates over multiple prediction horizons, with Bellman targets built from shorter-horizon estimates.Source: §III · pp. 2–3

IV. METHOD

ReF-HIL uses successful human experience to guide online value learning and human actions to focus policy improvement. An independent IQL branch supplies a reference, while a learned score defines the action neighborhood used by the actor.

A. Overview

  • An independent IQL branch supplies a reference policy and action-value estimates that shape the online critic, while the Human Action Fence uses a learned score to define the actor’s action neighborhood.The actor searches for higher-value actions within that neighborhood.Source: A. Overview · p. 3
  • Recovery and brief corrections provide pre-intervention autonomous state-action pairs for local value correction, while takeovers also add successful human suffixes to the successful-experience data.Whole rollouts ending at human-marked unrecoverable states directly supervise both value learners.Source: A. Overview · p. 3
  • Demonstrations and interventions train the human-action score, and online actor–critic updates draw equally from general replay and the human transition buffer.This sampling maintains human guidance as autonomous experience accumulates.Source: A. Overview · p. 3

B. Human-Reference-Guided Value Shaping

Human-Reference-Guided Value Shaping uses an independent IQL branch to anchor online values to successful human experience while incorporating local intervention feedback. Separate reference-action baselines and relative advantages let the method correct autonomous actions without changing the reference baseline.

B. Human-Reference-Guided Value Shaping

  • An independent IQL branch learns action-value vectors from successful and failure data, and evaluates the reference policy’s actions to provide the value reference for online learning.The reference policy is trained by advantage-weighted imitation on successful experience.Source: B. Human-Reference-Guided Value Shaping · pp. 3–4
  • The online critic separates reference-action values B from relative action advantages A, with separate parameter sets and the reference action centered at zero advantage.The reference-action value is set to the baseline B.Source: B. Human-Reference-Guided Value Shaping · p. 4
  • On successful-data states, a one-sided loss penalizes baseline estimates only when they fall below the IQL reference, leaving online experience room to support higher values.The loss uses an elementwise positive part and stop-gradient.Source: B. Human-Reference-Guided Value Shaping · p. 4
  • Intervention feedback ranks preceding autonomous actions below the fixed reference with a margin equal to the step cost, adjusting relative advantages while preserving the reference baseline.Human-marked unrecoverable states also directly supervise both value learners with an assumed no-subsequent-success return, mitigating overestimation that relative correction alone cannot resolve.Source: B. Human-Reference-Guided Value Shaping · p. 4

C. Human Action Fence

The Human Action Fence learns a neighborhood of human actions and focuses policy optimization within it. Outside the neighborhood, it constrains value estimates and updates so rejected actions cannot outrank the reference action.

C. Human Action Fence

  • A learned score H(s, a) defines the human-action neighborhood in which policy optimization is concentrated.The score is regressed to Gaussian-kernel targets for sampled query actions; σ sets the bandwidth in normalized action coordinates.Source: C. Human Action Fence · p. 4
  • The relative threshold ρ is chosen so the reference action remains inside the neighborhood.The definition applies for 0 < ρ ≤ 1.Source: C. Human Action Fence · p. 4
  • Inside the neighborhood, the policy retains learned relative advantages; outside it, the advantage is replaced by a distance penalty relative to the reference-action baseline.Outside-neighborhood values are componentwise below that baseline, preventing value extrapolation from making those actions outrank the reference.Source: C. Human Action Fence · p. 4
  • The policy objective permits value gradients to favor actions differing from the reference inside the neighborhood, while an outside-neighborhood quadratic term pulls actions toward it.This allows local improvement without an action-matching penalty inside the neighborhood.Source: C. Human Action Fence · p. 4
  • TD targets compare policy and reference actions at the next state; critic updates use fenced transitions and retain transitions into annotated unrecoverable states even when their actions are rejected.Rejected actions also receive an auxiliary ranking penalty on their relative advantages, propagating estimated returns and annotated failure costs to preceding actions.Source: C. Human Action Fence · p. 4

A. Experimental Setup

The experiments use a UR5e arm with visual and proprioceptive inputs, online human corrections, and five real-world manipulation tasks spanning contact-constrained motion, precision assembly, and deformable-object manipulation.

1) Implementation Details:

  • The UR5e setup combines RGB images from three wrist-mounted and global cameras with proprioceptive inputs, then executes Cartesian motion commands at 10 Hz through an admittance controller.A frozen ImageNet-pretrained ResNet-18 encodes images; the actor and learner run asynchronously on separate GPUs.Source: 1) Implementation Details · pp. 4–5

2) Task Definitions:

  • Task images label Cartesian translation as XY Z, end-effector z-axis rotation as Rz, and gripper actuation as G; arrows indicate motions or manipulation locations.These labels indicate the controlled action dimensions shown for the tasks.Source: Figure 3 · p. 5

1) Implementation Details:

  • Training starts with 20 successful human demonstrations per task, followed by autonomous interaction with SpaceMouse corrections and up to 10 preceding autonomous state-action pairs for pre-intervention correction.Episodes begin from task-specific randomized end-effector positions; the Human Action Fence threshold is ρ = 0.9.Source: 1) Implementation Details · p. 5

2) Task Definitions:

  • The five tasks cover Push-T, cap unscrewing, gear assembly, plug insertion, and dual-branch cable routing.They range from pushing a block and thread-constrained unscrewing to gear insertion, socket insertion, and routing two deformable branches in a fixed order.Source: 2) Task Definitions · p. 5

3) Baselines:

The comparison uses three HIL-RL baselines under shared demonstrations, spaces, initialization, evaluation, pretraining, and online-update schedules; final policies are evaluated without human intervention.

3) Baselines:

  • Final-policy results are separate evaluations without human intervention, reported as batch means with sample standard deviations rather than independent training runs.The reported table covers performance across five manipulation tasks.Source: 3) Baselines · p. 6, Table I · p. 6
  • HG-DAgger, HIL-SERL, and SiLRI serve as the three comparison baselines.HG-DAgger imitates demonstrations and aggregated corrections; HIL-SERL combines demonstrations and interventions in replay buffers for off-policy RL; SiLRI adapts imitation constraints to intervention uncertainty.Source: 3) Baselines · p. 6
  • All methods use the same 20 demonstrations, observation and action spaces, initial-state randomization, evaluation protocol, pretraining budget, and online-update schedule.Effective update rates are limited by computational throughput.Source: 3) Baselines · p. 6

4) Evaluation Metrics:

The evaluation tracks autonomous success, human intervention, and episode length over active training time. Across five real-world tasks, ReF-HIL achieves higher autonomous success with less reliance on human intervention than the baselines.

4) Evaluation Metrics:

  • Autonomous success rate measures task completion without human intervention; intervention rate is the mean per-episode fraction of human-controlled steps.Mean episode length uses actual steps for successful autonomous episodes and the maximum allowed steps for all others; plotted metrics are 20-episode moving averages.Source: 4) Evaluation Metrics · p. 6

B. Comparative Experiments

  • Across five tasks, ReF-HIL achieves higher autonomous success rates with less reliance on human intervention than the baselines.Online curves track these outcomes against accumulated active training time.Source: B. Comparative Experiments · p. 6, 4) Evaluation Metrics · p. 6
  • 100% autonomous success after approximately 25 minutes on Push-T exceeds HG-DAgger’s 86.7%, while cap unscrewing reaches 100% after approximately 18 minutes versus HG-DAgger’s 76.7%.On cap unscrewing, penalized episode length is 42.4 versus HG-DAgger’s 109.8 steps; on Push-T, it is 69.3 versus 66.6 steps.Source: B. Comparative Experiments · p. 6
  • 95.0% success on gear assembly compares with 38.3% for HG-DAgger, while penalized episode length falls from 118.4 to 40.1 steps.Source: B. Comparative Experiments · p. 6
  • On plug insertion, ReF-HIL achieves 91.7% success while every baseline remains at 5% or below, reaching 90% autonomous success after approximately 41 minutes.Intervention subsequently falls to zero.Source: B. Comparative Experiments · p. 6
  • On dual-branch cable routing, ReF-HIL reaches 90% autonomous success after approximately 62 minutes and finishes at 95.0%, versus at most 5% for the baselines.The task requires two successive grasp-and-route stages, with success feedback only after both branches are seated.Source: B. Comparative Experiments · p. 7

C. Ablation Studies

Ablations show that value shaping is important for completion reliability, while the Human Action Fence provides task-dependent gains. On a selected cable-routing trajectory, shaped critic estimates more closely track recorded returns than estimates without shaping.

C. Ablation Studies

  • Removing value shaping drops cable-routing success from 95.0% to 46.7% and raises normalized episode length from 39.1% to 74.7%.The ablations retain the IQL reference policy and use its actions as TD-backup candidates.Source: C. Ablation Studies · p. 7
  • The fence improves success on four tasks, with its largest gain on cable routing (18.3 percentage points), but removing it raises gear-assembly success from 95.0% to 100%.Its contribution is task dependent; the reported gear-assembly change also reduces normalized episode length from 26.7% to 13.5%.Source: C. Ablation Studies · p. 7

1) Value Learning:

  • 0.136 mean absolute deviation from recorded return with ReF-HIL, compared with 0.648 without value shaping.The comparison uses a selected 57-step autonomous trajectory; shaped estimates track recorded return at 45–55 minutes, while underestimation persists without shaping.Source: 1) Value Learning · p. 7, Figure 6 · p. 7

2) Policy Improvement:

Across five real-world manipulation tasks, ReF-HIL improves learning efficiency and autonomous success, reaching 90% success in 18–63 minutes and final success rates of 91.7–100%. In dual-branch cable routing, the reported comparison is consistent with in-fence imitation limiting later policy refinement; the conclusion notes that fence effectiveness varies by task.

2) Policy Improvement:

  • At approximately 70 minutes, ReF-HIL achieves 90% autonomous success with zero intervention, versus 45% success and 11.3% intervention for in-fence imitation.The separately evaluated IQL reference policy reaches 65% success at its final checkpoint.Source: 2) Policy Improvement · p. 8
  • Inside the fence, imitation penalizes deviations from the reference action even when they increase estimated return, trading value improvement against reference proximity.The paper describes the observed performance divergence as consistent with this constraint limiting later policy refinement.Source: 2) Policy Improvement · p. 8

VI. CONCLUSION

  • Across five real-world tasks, ReF-HIL reaches 90% autonomous success in 18–63 minutes and final success rates of 91.7–100%.The conclusion reports improved overall learning efficiency and higher autonomous success rates than evaluated baselines.Source: §VI · p. 8
  • Fence effectiveness varies across tasks, motivating future constraints that account for human-guidance reliability and online learning progress.The conclusion identifies adaptive constraints as a direction for future work.Source: §VI · p. 8

Related papers

Loading 2609.37131v1…