PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Learn from What a Human Would Avoid? Intervention-Aware World Models with Real-World RL for Dexterous Manipulation

Jiaju Yin, Zhenhui Zhang, Lixin Xu, Heng Zhang, Jun Shao, Yating Feng, Arash Ajoudani, Renjing Xu

arXiv:2609.06009v1cs.RO

TL;DR

Dexterous real-world RL faces costly exploration, while HIL pipelines often use takeovers only as reactive corrections rather than predictive safety supervision. WHIRL learns next-step takeover risk for actor-side shaping; across five real-robot tasks, it improves autonomous success by 15–30 percentage points over the strongest baseline and reduces operator-controlled training steps by up to 84%.

  • Problem

    Demonstrations may miss contact variations, and online RL on a 16-DoF hand makes exploratory mistakes costly; existing HIL pipelines use takeovers without learning where the policy will soon need help.

    Source: §1 · p. 2

  • Method

    WHIRL combines a behavior-prior residual policy with a four-head latent world model that predicts next-step takeover probability for actor-side risk shaping.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    WHIRL improves autonomous success by 15–30 percentage points over the strongest baseline and reduces operator-controlled training steps by up to 84% across five real-robot dexterous tasks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    WHIRL treats operator attention as reusable supervision by learning when the operator would intervene and routing that prediction to the actor.

    Source: §6 · p. 8

    Evaluation is limited to in-region pose and approach randomization, with one seed and one operator per method-task cell.

    Source: §4.4 · p. 7, §5 · p. 8

Abstract

from arXiv · show

Multi-fingered dexterous manipulation remains a frontier for real-world reinforcement learning (RL) due to the high-dimensional action space and the prohibitive cost of hardware failures. While human-in-the-loop (HIL) RL allows operators to intervene before failures occur, current pipelines often treat these interventions as reactive corrections, discarding the rich safety signal inherent in the operator's decision to take control. In this paper, we ask: How can we learn from what a human would avoid? We present WHIRL, a safety-aware RL framework that transforms binary human interventions into forward-predictive signals for proactive risk avoidance. Our approach centers on an intervention-aware latent world model with four prediction heads: dynamics, reward, termination, and a novel per-state intervention-probability head that learns to predict the likelihood of a human takeover at future states. This head provides an actor-side risk-shaping term that discourages the policy from entering "intervention-prone" regions, modeling the operator's internal safety threshold. We evaluate our framework on a 16-DoF LEAP Hand across tasks spanning convex and irregular object grasping, prismatic manipulation, and long-horizon multi-stage tasks. Our results show that predictive risk-shaping enables the system to achieve a 96.7 percent success rate on complex grasping tasks while reducing the operator intervention burden by up to 84 percent in step-weighted terms. By closing the loop between human intuition and predictive world modeling, this work provides a practical safety-aware recipe for training complex dexterous agents in the real world while reducing operator fatigue and hardware-risk exposure.

1 Introduction

Dexterous manipulation is difficult to learn reliably from demonstrations because small contact variations can cause failures, while online exploration on a 16-DoF hand is costly. WHIRL reuses human takeovers as predictive risk signals within a real-world HIL-RL system.

1 Introduction

  • Contact-rich dexterous manipulation remains challenging because pose, finger timing, and slip variations can defeat behavior-cloning policies under deployment shift.Teleoperated demonstrations provide nominal approach, grasp, and placement motions, but cannot exhaustively cover these variations.Source: §1 · p. 2
  • For a 16-DoF hand, online RL is costly because coupled arm–hand actions and stiff, discontinuous contacts make each exploratory mistake consume robot time, resets, and hardware attention.Sim-to-real approaches can reduce the burden when accurate simulation is available but do not remove the need for real-robot adaptation.Source: §1 · p. 2
  • HIL-RL lets an operator take over before unrecoverable failures, but each takeover consumes expert attention.The paper asks whether that attention continues to benefit learning after control returns to the policy.Source: §1 · p. 2
  • WHIRL trains a takeover-probability head alongside dynamics, reward, and termination predictions, turning interventions into a reusable actor-side risk signal.Its contributions also include a mid-rollout teleoperation system and controlled real-robot evaluation across five dexterous tasks.Source: §1 · p. 2

2 Related Work

Prior work covers real-world online RL, human-in-the-loop exploration, latent world models, and dexterous teleoperation. WHIRL combines these lines by predicting future takeovers from real-robot intervention labels and routing that prediction to the actor.

2 Related Work

  • Real-world online RL has advanced through direct actor–learner systems such as SERL [10] and related platforms [8]; for dexterous hands, ResFiT [25] freezes a behavior-cloning prior and learns a sparse-reward correction.HIL methods such as HIL-SERL [14], SiLRI [15], and interactive imitation [16] [17] [18] make exploration safer by adding takeovers to replay, but use each takeover once and discard the rest of the signal.Source: §2 · p. 2
  • WHIRL trains takeover labels as next-step intervention-risk predictions consumed by the actor on every state, rather than as one-use behavior supervision, replay weights, or metrics.The overview distinguishes the dynamics, reward, and termination heads supporting critic learning from the intervention head shaping the actor.Source: §2 · p. 3, Figure 2 · p. 3
  • Latent world models such as Dreamer [19], MBPO [21], and TD-MPC [22] predict environment quantities, while Hi-WM [26] lets an operator intervene inside a learned model.WHIRL instead keeps corrections on the real robot and predicts future human takeover for actor-side risk shaping.Source: §2 · p. 3, Figure 2 · p. 3

3 Method

WHIRL combines mid-rollout human teleoperation, residual RL around a behavior prior, and a four-head latent world model. Its intervention head predicts next-step takeovers for actor-side risk shaping, while the other model predictions support critic learning.

3.1 Human-in-the-Loop Teleoperation System

  • The tri-channel teleoperation stack uses a SpaceMouse for the arm, a glove for the hand, and a pedal for takeovers that label transitions for evaluation and model supervision.The LEAP + Manus retargeter maps non-thumb joints affinely and uses thumb fingertip IK; command rebasing makes the first takeover command match the current hand pose and then applies glove displacement.Source: §3.1 · p. 4

3.2 Residual RL Based on a Behavior Prior

  • Residual RL updates a correction to a frozen teleoperated behavior prior, while replay combines demonstration, autonomous, and intervention transitions with takeover labels.The online hand residual is restricted to a low-dimensional synergy basis to preserve coordinated postures, and the hand regularizer is applied in that space.Source: §3.2 · p. 4

3.3 Intervention-Aware World Model

  • The latent world model predicts next-state dynamics, reward, termination, and next-step intervention probability; the intervention target is the takeover label at the predicted step.A label-availability mask excludes replay segments without logged takeover state, making this a forward predictor rather than a current-state intervention classifier.Source: §3.3 · pp. 4–5
  • Dynamics, reward, and termination predictions support uncertainty-gated critic updates, while the intervention head is excluded from the Bellman target to keep operator habits out of return estimates.The critic uses an uncertainty gate for model-derived targets.Source: Figure 3 · p. 5, §3.3 · p. 5
  • Actor-side shaping penalizes predicted termination, intervention probability, and ensemble disagreement; intervention probability can signal risk before terminal failure.The intervention prediction shapes the actor rather than entering the reward, because it predicts operator behavior rather than task cost.Source: §3.3 · p. 5

4 Experiments

The experiments compare a four-policy ladder across five real-robot tasks, using intervention traces and autonomous success to evaluate WHIRL. The final results report higher task success and lower operator-controlled burden, with ablations probing the source of the gains.

4.1 Tasks

  • The five real-robot tasks span convex and irregular grasping, pick-and-place, drawer pulling, and a three-stage drawer-open, object-place, drawer-close task.Long Horizon is evaluated end-to-end in one rollout.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Baselines

  • The comparison ladder progresses from a frozen behavior prior to ResFiT, model-free Residual RL, and full WHIRL, adding mechanisms at each step.The model-free baseline adds takeover-aware critic training; WHIRL is the full pipeline.Source: §4.2 · p. 6

4.3 Metrics

  • The headline trace metric is a step-weighted, five-episode rolling intervention fraction; autonomous success is measured per method and task over rollouts.The trace is based on the fraction of each episode’s steps under operator control.Source: §4.3 · pp. 6–7

4.4 Training and evaluation details

  • All methods start from the same frozen behavior-cloning prior and train with in-region randomization under a fixed operator takeover rule.Success is evaluated on held-out deployments with the pedal disabled, while each task-method cell uses one seed and one operator.Source: §4.4 · p. 7

4.5 Main Results

  • WHIRL improves over model-free Residual RL by +15–30 percentage points on every task.Fisher’s exact test reaches p < 0.05 on the three pick tasks; Pull Drawer and Long Horizon are suggestive at p ≈0.08.Source: §4.5 · p. 7, Table 1 · p. 7
  • On Pull Drawer, WHIRL reduces the final rolling intervention fraction from 0.113 to 0.018 (84%).Across all five tasks, WHIRL stays below the model-free baseline and reaches the near-zero regime earlier; the metric is step-weighted.Source: Figure 5 · p. 7, §4.5 · p. 8
  • Removing actor-side intervention shaping or the world model delays convergence.The mechanism ablation compares those removals with full WHIRL.Source: Figure 6 · p. 8

5 Limitations

WHIRL’s evidence and deployment scope are bounded by the real-robot HIL setting, and its intervention predictor reflects the labeling operator’s takeover habits.

5 Limitations

  • On Pull Drawer, risk shaping shows a brief early cost before crossover, consistent with benefits emerging after the intervention head fits enough replay..Source: §5 · p. 8
  • Evaluation covers in-region pose and approach variation, not new objects, layouts, or fine in-hand manipulation.The Pull Drawer and Long Horizon success gaps are suggestive at current sample sizes.Source: §5 · p. 8
  • The intervention head inherits the labeling operator’s takeover threshold and habits, so substantially more or less cautious operators may require recalibration.Training also uses one seed and one operator per cell.Source: §5 · p. 8

6 Conclusion

WHIRL turns human takeovers into reusable risk predictions for actor-side shaping. Across five real-robot dexterous tasks, it improves autonomous success over the strongest baseline and reduces operator-controlled training steps.

6 Conclusion

  • WHIRL improves autonomous success by 15–30 percentage points over Residual RL model-free and reduces operator-controlled training steps by up to 84%.The paper presents this as a practical recipe for dexterous HIL-RL: keep corrections on the real robot, predict when the operator would intervene, and route that prediction to the actor rather than the Bellman target.Source: §6 · p. 8

A Hand Retargeter Implementation Details

The hybrid hand retargeter maps glove commands to LEAP joints using affine maps for non-thumb fingers and a constrained thumb solve. Its intervention rebase preserves the current hand pose when control changes hands.

Non-thumb fingers

  • Non-thumb index, middle, and ring joints use per-joint affine maps whose gains and offsets are fit offline from three reference postures.The reference postures are fully open, half-closed, and fully closed.Source: A Hand Retargeter Implementation Details · p. 11

Thumb

  • The 4-DoF thumb solve minimizes residuals for fingertip targeting, thumb–index opposition, and frame-to-frame smoothing.The solver uses damped Gauss–Newton steps, up to 40 iterations per control step, and typically converges in fewer than 10 iterations.Source: A Hand Retargeter Implementation Details · p. 11

Intervention rebase

  • After takeover, the first commanded pose equals the current robot hand pose, and later commands apply only the retargeted posture’s joint-space increment since takeover.The rebase is deterministic in the current glove sample while continuous control retains the solver’s temporal warm-start.Source: A Hand Retargeter Implementation Details · pp. 11–12

B World Model and RL Hyperparameters

The world model uses real replay transitions for its critic and a one-step latent design for auxiliary predictions. WHIRL keeps takeover-probability shaping in the actor objective, separate from task reward and Bellman targets.

B World Model and RL Hyperparameters

  • The backbone critic learns from real replay transitions with a clipped-double-Q Bellman target, with an uncertainty-gated world-model auxiliary target added on top.The next actor action is sampled from the current actor; the frozen behavior-prior action and encoded real next observation also enter the target description.Source: B World Model and RL Hyperparameters · p. 12
  • Table 2 reports world-model and residual-RL hyperparameters, with values shared across the five tasks unless noted.The ablations retain these settings except for their named removals.Source: Table 2 · p. 12, B World Model and RL Hyperparameters · p. 12

C Design Choices for the Intervention-Aware World Model

  • Each world-model head is a small MLP on the same encoded latent, and the one-step design avoids multi-step imagination that compounded dynamics error on the 16-DoF hand.Dynamics supplies an epistemic mask, reward and termination form a one-step Bellman target, and intervention prediction supplies forward supervision for the actor.Source: C Design Choices for the… · pp. 12–13
  • The takeover-probability head shapes the actor objective alone, avoiding a reward bonus that could redefine task optimality or leak prediction errors into critic value estimates.Setting α_i to zero removes intervention shaping while preserving the dynamics, reward, and termination scaffold, enabling a separate ablation of actor-side shaping.Source: C Design Choices for the… · p. 13

D Detailed Training and Evaluation Protocol

The protocol reports task-level confidence intervals and Fisher tests, then details the shared operator procedure, demonstration data, and single-seed evaluation choices. These controls support within-task comparisons while bounding how broadly results can be interpreted.

D Detailed Training and Evaluation Protocol

  • Each task uses 15–30 teleoperated demonstrations both to fit the frozen behavior-cloning prior and seed replay, with no additional offline data added during online training.One operator follows a fixed rule: intervene near workspace-bound violations or object drops, then release once the failure mode is averted.Source: D Detailed Training and Evaluation… · pp. 13–14
  • Each task-method cell uses one training seed and one operator; trailing five-episode smoothing addresses within-seed noise, but trace metrics lack across-seed bands.The baseline ladder shares each task’s seed/operator pair to isolate row-to-row mechanisms, while the success-rate table uses checkpoints closest to the nominal budget.Source: D Detailed Training and Evaluation… · pp. 13–15
  • WHIRL’s Wilson 95% CI is [89, 100]% for Pick Cube, [83, 99]% for Pick LEGO, [74, 97]% for Pick & Place, [76, 99]% for Pull Drawer, and [64, 95]% for Long Horizon.Fisher one-sided p-values versus Residual RL are 0.026, 0.026, 0.008, 0.091, and 0.078, respectively, in the same task order.Source: D Detailed Training and Evaluation… · p. 14
  • The three pick tasks, each at N = 30, reach p < 0.05, while Pull Drawer and Long Horizon, each at N = 20, remain in the suggestive regime at p ≈0.08–0.09.The report uses per-task one-sided uncorrected Fisher tests and applies no multiple-comparison correction across the five tasks.Source: D Detailed Training and Evaluation… · p. 14

E Raw Per-Episode Intervention Traces

Figure 7 shows noisier per-episode intervention fractions for WHIRL and WHIRL without the world model across five tasks; both methods converge to near-zero intervention need.

E Raw Per-Episode Intervention Traces

F Hardware, Workspace, and Data Collection

The setup uses a Franka FR3 with a 16-DoF LEAP Hand, camera observations, and distinct arm and hand control rates. Task-specific workspace, reset, horizon, demonstration, and training details are specified alongside a tri-channel teleoperation rig.

F Hardware, Workspace, and Data Collection

  • A Franka FR3 with a 16-DoF LEAP Hand uses wrist-mounted D405 and third-person D435 RGB cameras at 30 Hz.The arm runs at 20 Hz in Cartesian impedance mode; the hand and Manus glove stream at 120 Hz.Source: F Hardware, Workspace, and Data… · p. 15
  • Table 3 lists task randomization, reset pose, workspace bounds, and episode horizon for the five tasks.Pick Cube is the convex-object control, and Long Horizon is the three-stage task.Source: F Hardware, Workspace, and Data… · p. 15, Table 3 · p. 15
  • Behavior priors use 15–30 demonstrations; residual RL runs for approximately 8k steps on pick/drawer tasks and 50k on Long Horizon.Workspace bounds derive from raw teleoperation episodes with a 2 cm margin.Source: F Hardware, Workspace, and Data… · p. 15
  • The tri-channel rig pairs a SpaceMouse for arm control, a Manus glove for hand control, and a foot pedal for takeover.Figure 8 identifies the three teleoperation channels.Source: Figure 8 · p. 15
Loading 2609.06009v1…