PaperlayerView exact revision on arXiv

Source-linked AI summary

DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation

Yifan Han, Zhongxi Chen, Yuxuan Zhao, Congsheng Xu, Yanming Shao, Yichuan Peng, Yao Mu, Wenzhao Lian

arXiv:2603.09121v1cs.ROcs.AI

TL;DR

Dexterous VLA post-training remains challenging because multi-finger control is high-dimensional and contact-rich, while offline adaptation struggles with these tasks. DexHiL combines coordinated arm-hand human interventions with corrective-data weighting; after three online refinement stages, it reports 20% and 30% success-rate improvements across two tasks over an equivalent-data offline baseline.

  • Problem

    Offline post-training for VLA models struggles to adapt to high-DOF, contact-rich dexterous manipulation and its execution distributions.

    Source: §I · p. 1

  • Method

    DexHiL combines coordinated arm-hand human-in-the-loop intervention with intervention-aware weighting of corrective data for VLA post-training.

    Source: §I · p. 2

  • Results

    20% and 30% improvements in success rates across two distinct tasks, respectively, followed three online refinement stages versus an offline baseline trained with equivalent data volume.

    Source: §I · p. 2

  • Takeaways & Limitations

    Real-robot experiments establish DexHiL as an effective and practical post-training recipe for improving dexterous VLA policies.

    Source: §V · p. 8

Abstract

from arXiv · show

While Vision-Language-Action (VLA) models have demonstrated promising generalization capabilities in robotic manipulation, deploying them on specific and complex downstream tasks still demands effective post-training. In parallel, Human-in-the-Loop (HiL) learning has proven to be a powerful mechanism for refining robot policies. However, extending this paradigm to dexterous manipulation remains challenging: multi-finger control is high-dimensional, contact-intensive, and exhibits execution distributions that differ markedly from standard arm motions, leaving existing dexterous VLA systems limited in reliability and adaptability. We present DexHiL, the first integrated arm-hand human-in-the-loop framework for dexterous VLA models, enabling coordinated interventions over the arm and the dexterous hand within a single system. DexHiL introduces an intervention-aware data sampling strategy that prioritizes corrective segments for post-training, alongside a lightweight teleoperation interface that supports instantaneous human corrections during execution. Real-robot experiments demonstrate that DexHiL serves as an effective post-training framework, yielding a substantial performance leap, outperforming standard offline-only fine-tuning baselines by an average of 25% in success rates across distinct tasks. Project page: https://chenzhongxi-sjtu.github.io/dexhil/

I. INTRODUCTION

Dexterous VLA post-training is difficult because high-DOF, contact-rich hand control challenges offline adaptation and real-robot execution. DexHiL combines arm-hand human intervention with intervention-aware weighting to refine policies.

I. INTRODUCTION

  • Dexterous VLA post-training must address high-dimensional contact-rich control, offline-data limitations, and execution drift that can push policies into OOD states.Offline datasets may overrepresent repetitive successes, while minor open-loop execution errors can accumulate into failure.Source: §I · pp. 1–2
  • DexHiL integrates online human interventions with offline training and reweights corrective data to improve sample efficiency and policy convergence.The framework prioritizes corrective segments from expert interventions for error recovery and task refinement.Source: Figure 1 · p. 1, §I · pp. 2–3
  • 20% and 30% improvements in success rates across two distinct tasks, respectively, followed three online refinement stages versus an offline baseline trained with equivalent data volume.Ablations identify intervention-aware weighting as the primary driver for overcoming sample-efficiency bottlenecks in high-DOF manipulation.Source: §I · p. 2
  • DexHiL also introduces learning-based retargeting to map human hand movements to dexterous robotic hands with adaptive, real-time control.The authors describe the mapping as more adaptive and real-time than traditional optimization methods.Source: §I · p. 2
  • Its teleoperation system supports smooth, real-time human corrections during post-training to address intervention discontinuity in high-DOF hand manipulation.The framework describes these interventions as a means of collecting high-quality error corrections during execution.Source: §I · p. 2

II. RELATED WORK

Prior dexterous data-collection, VLA adaptation, and human-correction approaches each leave gaps in precise arm-hand coordination or effective downstream post-training. DexHiL is positioned as an online post-training framework addressing these limitations.

A. Dexterous Manipulation Data Collection System

  • Existing dexterous data collection uses simulated human-hand data or teleoperation, but imprecise alignment and limited finger correspondence hinder incremental interventions and accurate grasp control.Optimization-based teleoperation can struggle with precise finger motion, while network fitting may focus mainly on the thumb.Source: A. Dexterous Manipulation Data… · p. 3

B. Vision Language Action model for dexterous manipulation

  • Offline fine-tuning of dexterous VLA and VLG models faces distribution shifts, low sample efficiency, and convergence difficulty in high-dimensional, contact-intensive tasks.The cited approaches typically collect offline real-world datasets before adapting to downstream robots and tasks.Source: B. Vision Language Action model for… · p. 3

C. Human-in-the-Loop Corrections for Robot Learning

  • Human-in-the-loop correction has largely been limited to parallel gripper-arm setups; DexGrasp-VLA corrects only the arm, leaving hand motion to a separate grasping network.The related-work passage links this lack of arm-hand coordination to reduced effectiveness for contact-rich tasks.Source: C. Human-in-the-Loop Corrections for… · p. 3

III. METHOD

DexHiL combines an arm-hand teleoperation interface with an intervention-aware post-training pipeline. Its hand-retargeting method maps human keypoints to robot joints while using staged optimization to preserve a fuller grasp repertoire.

III. METHOD

  • DexHiL pairs real-time arm-hand teleoperation and intervention with intervention-aware weighting to refine policies and mitigate covariate shift during real-world execution.The framework targets robust error-recovery behavior in contact-rich scenarios.Source: §III · p. 3

A. Interactive Human-in-the-Loop Teleoperation System for Dexterous Manipulation

  • A handheld ArUco cube and monocular camera provide real-time 6D pose tracking, while dual-path mapping handles arm motion and finger articulation.The paths jointly maintain trajectory consistency and gesture-level controllability despite human-robot kinematic mismatch.Source: A. Interactive Human-in-the-Loop… · p. 3

1) Hand Joint Retargeting:

  • Hand retargeting maps glove-captured keypoints to actuated robot joints, applies coupling constraints, and uses forward kinematics for fingertip positions; training separates non-thumb fingers from a thumb residual stage.The first stage avoids pinch-like collapse by learning a four-finger motion manifold; the second freezes those parameters and optimizes the thumb with geometric regularizers and an interfingertip kinematic term.Source: 1) Hand Joint Retargeting · pp. 3–4

2) Arm Pose Mapping:

Arm pose mapping uses the ArUco cube’s motion relative to an anchor pose to set the robot end-effector target, then derives arm joint angles with inverse kinematics.

2) Arm Pose Mapping:

  • The ArUco cube pose is a 6-DoF pose expressed in the camera frame, and the cube-to-end-effector transformation is assumed known.Source: 2) Arm Pose Mapping · p. 4
  • The cube’s relative motion is applied to the robot end-effector anchor pose to compute the target end-effector pose in the base frame.At intervention, the system records the current end-effector and cube poses as anchors.Source: 2) Arm Pose Mapping · p. 4
  • The robot’s arm joint angles are derived from the target pose via inverse kinematics.Source: 2) Arm Pose Mapping · p. 4
  • The marker-based technique is described as robust to ambient lighting and efficient to detect in cluttered environments.Source: 2) Arm Pose Mapping · p. 4

3) Asynchronous Multi-threaded Control and Intervention:

The control system runs autonomous policy inference alongside faster human-guided arm and hand teleoperation. A human can take over the entire robot when imminent task failure is detected.

3) Asynchronous Multi-threaded Control and Intervention:

B. Human-in-the-Loop Post-training pipeline

DexHiL combines online human intervention data with offline data in a post-training pipeline that initializes, iteratively refines, and filters policy training data. Its weighting mechanism emphasizes sparse corrective samples, while filtering retains terminal recovery segments after the last takeover.

B. Human-in-the-Loop Post-training pipeline

1) Intervention-aware Weighting Mechanism:

  • The intervention-aware weighting mechanism increases the target distribution’s proportion of intervention samples to emphasize sparse, high-value corrective data.The method frames the weighting as importance sampling; the implementation sets P*(intervention) = 0.5.Source: 1) Intervention-aware Weighting… · p. 4

2) Policy Update Mechanism:

  • Policy refinement proceeds through offline warm-up, online interactive learning, and strategy-driven data filtering.Warm-up fine-tunes on D0 to initialize π0; later iterations deploy the policy under human supervision, aggregate corrective data, and warm-start updates from prior parameters.Source: 2) Policy Update Mechanism · pp. 4–5
  • Real-world rollouts illustrate tissue extraction through fingertip alignment and vertical retraction, and plush-toy grasping through synchronized multi-joint flexion.Source: Figure 3 · p. 5
  • The policy’s weighted imitation objective prioritizes gradients from high-value intervention data during iterative updates.DexHiL is instantiated with Being-H0.5 [11], whose action head uses a Flow Matching objective for imitation loss.Source: 2) Policy Update Mechanism · p. 5
  • For intervention trials, filtering keeps only segments from the final human intervention to task completion and discards earlier segments.The stated rationale is that earlier actions can be sub-optimal and multiple interventions can produce incoherent trajectories, risking policy oscillation or multimodal distribution conflict.Source: 2) Policy Update Mechanism · p. 5

IV. EXPERIMENTS

The experiments are organized to test whether DexHiL improves dexterous-hand VLA performance over time and how hardware-specific algorithmic designs contribute to complex manipulation performance.

IV. EXPERIMENTS

  • The experiments address whether DexHiL improves dexterous-hand VLA performance over time and how hardware-specific designs contribute to complex manipulation performance.Source: §IV · p. 6

1) Tasks:

The experiments assess end-to-end grasping and fine-grained tissue extraction, then describe the robot platform, data collection, and training setup.

1) Tasks:

  • The two tasks were designed to evaluate full-system HiL effectiveness and fine-grained dexterous-hand manipulation.Source: 1) Tasks · p. 6
  • Plush toy grasping tests end-to-end policy performance and the hand’s grasping capability.Source: 1) Tasks · p. 6
  • Tissue extraction requires precise dexterous retargeting to pull a thin, deformable tissue reliably.Source: 1) Tasks · p. 6

2) Implementation Details:

  • The platform pairs a Franka Research 3 arm with a DexHand021; training starts from 60 offline trajectories and adds 10 trajectories per task each round.New in-the-loop trajectories are compared with an equal-volume offline-pool condition to isolate the effect of interactive collection.Source: 2) Implementation Details · p. 6
  • Being-H0.5 is fully trained for 60k iterations on 8 NVIDIA H100 GPUs, then fine-tuned on human-interaction data using one H100 GPU.Source: 2) Implementation Details · p. 6

B. Baselines and Evaluation Protocol

The evaluation compares DexHiL with data-matched offline and online baselines on two physical-robot tasks. Results report higher success, less human labor, and mechanisms tied to corrective trajectories and hand retargeting.

B. Baselines and Evaluation Protocol

  • The data-budget-matched comparison uses full-parameter fine-tuning, with offline baselines containing 40, 50, or 60 trajectories per task and 20 physical-robot trials per task.Success requires extracting more than half a napkin or lifting the plush toy entirely off the table; all policies share pretrained weights.Source: B. Baselines and Evaluation Protocol · p. 6

C. Experiment Results

  • Fig. 4 compares four gestures: Dex-Retargeting shows discontinuous contact adjustments, while GeoRT struggles to form stable, coordinated grasp postures.The paper reports more accurate, smooth, and coordinated poses from its method, which it connects to the 95% Tissue Extraction result.Source: Figure 4 · p. 6, C. Experiment Results · pp. 7–8
  • By Round 3, intervention segments take 3s versus 10s offline, reducing total human labor by 35% (13 min versus 20 min).Source: C. Experiment Results · p. 7
  • By Round 3, Tissue Extraction reaches 95% success with DexHiL versus 80% for DAgger* and 75% for the offline Baseline; Plush Toy Grasping reaches 65% versus 20% and 35%, respectively.Source: C. Experiment Results · p. 6, Table I · p. 7
  • Corrections at impending failures preserve recovery trajectories for two bottlenecks: failed tissue-edge pinches and premature grasping before the wrist reaches its target.Precise pinching supports rapid gains in Tissue Extraction, while DexHiL outperforms the baselines on coordination-intensive grasping.Source: C. Experiment Results · p. 7
  • Corrective trajectories produce loss spikes at the start of training rounds, which the paper attributes to added out-of-distribution recovery states that intervention-aware weighting prioritizes.The training-loss analysis contrasts DexHiL with DAgger* and reports the loss increase after human corrective trajectories are added.Source: C. Experiment Results · p. 7, Figure 5 · p. 7

V. CONCLUSION AND FUTURE WORK

DexHiL combines arm–hand teleoperation with intervention-aware sampling for dexterous VLA post-training. Real-robot experiments show consistent task-success gains, while future work will examine hand representations.

V. CONCLUSION AND FUTURE WORK

  • DexHiL is an arm–hand integrated human-in-the-loop post-training framework for dexterous VLA.Source: §V · p. 8
  • The framework combines a lightweight teleoperation interface with intervention-aware sampling that emphasizes corrective segments as post-training data.Source: §V · p. 8
  • Real-robot experiments demonstrate consistent task-success gains, and planned work will study hand tokenizers after hardware and teleoperation optimizations.Source: §V · p. 8

Related papers

Loading 2603.09121v1…