PaperlayerView exact revision on arXiv

Source-linked AI summary

HACo: Learning Haptic Active Compliance for Force-Aware Dexterous Manipulation

Naisheng Ye, Yinzhe Zhou, Junkai Zhao, Yuhang Lu, Checheng Yu, Zhenjie Yang, Pengwei Wang, Hongyang Li

arXiv:2609.36596v1cs.RO

TL;DR

Contact-rich dexterous manipulation needs policies that translate physical feedback into motion while regulating loads across changing contacts. HACo learns controller-executable compliant actions from tactile and joint-torque feedback, achieving an 83% mean success rate versus 35% for the strongest evaluated baseline. The authors identify teleoperation, demonstrable-motion scope, and limited tactile coverage as remaining challenges.

  • Problem

    Existing policies may overlook complementary fingertip tactile and joint-torque feedback, while common action targets can encode excessive loading or omit motion constrained by contact.

    Source: §I · pp. 1–2

  • Method

    HACo learns controller-executable compliant actions from compliance-regulated teleoperation and grounds action generation in tactile and joint-torque feedback using gated haptic cross-attention.

    Source: §I · p. 2

  • Results

    83% mean success rate versus 35% for the strongest evaluated baseline across five real-world dexterous tasks.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The results establish a practical route to closed-loop, force-aware dexterity across diverse real-world tasks.

    Source: §V · p. 8

    Current learning remains bounded by teleoperation-demonstrable motions, and palm and phalange contacts are indirectly inferred from joint torque rather than directly observed.

    Source: §V · p. 8

Abstract

from arXiv · show

Contact-rich dexterous manipulation requires policies that translate physical feedback into motion commands while regulating interaction loads across evolving multi-contact interactions. This requires haptic observations of contact state and action supervision showing how commands should adapt. Existing policies often overlook complementary fingertip tactile and joint-torque feedback, while common action targets either encode excessive loading or omit motion constrained by the object. We introduce HACo, a Haptic Active Compliance policy that learns force-regulating actions directly from haptic feedback. Compliance-regulated teleoperation converts operator inputs into controller-executable compliant actions that preserve motion intent while regulating loads. HACo learns these actions directly, using command-state discrepancy as auxiliary compliant-intent supervision. It combines local fingertip tactile responses with joint-torque feedback capturing load transmission through the articulated hand, including contacts beyond tactile coverage. A Compliance Grounding Module uses gated haptic cross-attention to ground action generation in the evolving haptic state, enabling closed-loop force regulation without explicit online contact modeling. We evaluate HACo on a real-world benchmark covering multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation. Across 20 trials per task, HACo achieves an 83% mean success rate, compared with 35% for the strongest evaluated baseline. These results demonstrate active compliance across diverse force-sensitive dexterous manipulation tasks.

I. INTRODUCTION

Contact-rich dexterous manipulation must regulate both motion and interaction force as contacts change. HACo combines complementary haptic sensing with compliance-regulated action learning and evaluates the policy on five real-world tasks.

I. INTRODUCTION

  • Changing contacts can make geometrically correct trajectories fail through excessive pressure, insufficient traction, or poorly regulated torque.Existing targets can either omit the command–state discrepancy needed to sustain interaction force or preserve operator intent while inducing excessive force.Source: §I · pp. 1–2
  • Compliance-regulated teleoperation produces controller-executable actions that preserve task-directed motion while reducing excessive loads, with command–state discrepancy providing compliant-intent supervision.HACo conditions action generation on evolving haptic feedback through gated cross-attention, without explicit force prediction, force tracking, or online analytical contact modeling.Source: §I · p. 2
  • 83% mean success rate versus 35% for the strongest evaluated baseline across five real-world tasks spanning force-sensitive dexterous manipulation.The benchmark includes multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation.Source: §I · p. 2
  • HACo couples fingertip tactile sensing with joint-torque feedback to capture local contact responses and loads transmitted through the hand.Joint torque also reflects contacts beyond tactile coverage.Source: §I · p. 2
  • The paper introduces a real-world benchmark covering multi-contact friction, tangential interaction, fragile curved-surface contact, rotational torque, and deformable-object manipulation.These task categories are used to evaluate force-sensitive dexterous manipulation.Source: §I · p. 2

II. RELATED WORK

Related work covers complementary physical sensing for dexterous manipulation and approaches to compliance control. HACo combines tactile and torque observations and learns regulated action references directly.

A. Haptic Perception for Dexterous Manipulation

  • Prior haptic policies use tactile signals and proprioceptive proxies; HACo combines fingertip tactile responses with joint-torque feedback for local contact and hand-transmitted loads.The combination covers loads transmitted through the articulated hand, not only fingertip contact responses.Source: A. Haptic Perception for Dexterous… · p. 2

B. Compliance Control for Contact-Rich Manipulation

  • Compliance approaches shape motion–force relations or learn compliance profiles, while HACo learns regulated controller-executable references and uses their discrepancy from observed hand state as auxiliary supervision.Unlike Contact-Grounded Policy, HACo learns regulated references directly rather than predicting robot state and tactile feedback to obtain compliance-controller targets.Source: B. Compliance Control for Contact-Rich… · pp. 2–3

III. METHOD

HACo maps multimodal observations to controller-executable compliant actions. Compliance-regulated teleoperation adjusts nominal arm and hand references before execution and records the resulting compliant actions as training targets.

III. METHOD

  • At each step, HACo uses visual, language, robot-state, tactile, and joint-torque histories to predict a horizon of controller-executable compliant actions.The arm end-effector and dexterous-hand references are sent to their corresponding low-level controllers.Source: §III · p. 3

A. Compliance-Regulated Teleoperation

  • An outer-loop regulator modifies motion-retargeted nominal arm and hand references before execution, preserving task-directed motion while preventing unnecessarily large interaction loads.The adjusted compliant actions are recorded as ground truth for learning.Source: A. Compliance-Regulated Teleoperation · p. 3
  • For the arm, wrench components exceeding an admissible range drive a Cartesian admittance response, while in-range input vanishes and displacement relaxes under virtual dynamics.Virtual mass, damping, and stiffness specify the admittance dynamics.Source: A. Compliance-Regulated Teleoperation · p. 3
  • For the hand, fingertip forces are clipped in magnitude, and virtual contact stiffness maps the required unloading to fingertip displacement and joint-space adjustments.The objective balances unloading against small, smooth joint motion; kinematic coupling can also move other fingertips, and joint-position and update limits apply.Source: A. Compliance-Regulated Teleoperation · p. 3

B. Active Compliance Learning

HACo learns controller-executable compliant commands rather than nominal commands or observed states alone, and uses their command–state discrepancy as auxiliary compliant-intent supervision. The action expert is trained to predict both outputs from haptic-conditioned observations.

B. Active Compliance Learning

  • The compliant command is the learning target because it preserves task-directed motion while incorporating force-regulating adjustments absent from nominal commands or observed configurations.Nominal commands may induce excessive contact force, while observed states omit motion constrained by contact.Source: B. Active Compliance Learning · p. 3
  • Under sustained contact, the time-aligned command–state discrepancy captures the force-regulating component and is supervised as compliant intent, not executed as an independent residual.The discrepancy is dominated by the contact-dependent reference offset introduced by compliance regulation.Source: B. Active Compliance Learning · pp. 3–4
  • Conditional flow matching trains the action expert on compliant-action and compliant-intent trajectories, with separate prediction channels and jointly optimized losses.The compliant trajectory is the executable target; the auxiliary intent loss preserves force-regulating information in the command–state discrepancy.Source: B. Active Compliance Learning · p. 3
  • Cartesian admittance and hand force regulation turn retargeted nominal references into controller-executable compliant references while robot state and haptic observations are recorded.These references provide compliant-action supervision from teleoperation.Source: Figure 3 · p. 4
  • For asynchronous execution, Real-Time Chunking conditions each new action chunk on a committed prefix, while only compliant actions—not predicted compliant intent—are sent to controllers.Training uses sampled prefixes up to 12 steps; inference uses a 10-step prefix.Source: B. Active Compliance Learning · p. 4

C. Kinematics-Aligned Haptic Perception

HACo combines fingertip tactile signals with joint-torque histories to represent local contact and load transmission along the articulated hand. It aligns these signals within each digit before modeling interactions across digits.

C. Kinematics-Aligned Haptic Perception

  • Fingertip wrench and deformation features describe local contact, while joint torques capture broader, spatially ambiguous loads transmitted through the hand.The modalities are aligned according to hand kinematics before interactions across digits are modeled.Source: C. Kinematics-Aligned Haptic Perception · p. 4
  • For each digit, a torque encoder and a tactile encoder produce representations that are fused into one token before cross-finger information exchange.The tactile branch combines wrench history with the current deformation map; the torque branch embeds joint-torque histories.Source: C. Kinematics-Aligned Haptic Perception · p. 5
  • A cross-digit Transformer processes ten finger tokens with unavailable measurements masked, producing structured haptic memory for the Compliance Grounding Module.The platform measures 44 hand-joint torques and ten fingertip tactile streams; wrench and torque histories span nine steps, while the deformation branch uses the current 240×240 map.Source: C. Kinematics-Aligned Haptic Perception · pp. 4–5

D. Compliance Grounding Module

The Compliance Grounding Module lets action features query structured haptic tokens through gated haptic cross-attention. Its learnable residual gate controls haptic influence and is initialized so the module begins as an identity mapping.

D. Compliance Grounding Module

  • The module connects structured haptic tokens to evolving action features through gated cross-attention, rather than simply appending haptic features to visual or action tokens.This design explicitly aligns contact observations with action generation while avoiding potential disruption to pretrained cross-modal dependencies.Source: D. Compliance Grounding Module · p. 5
  • Action features query the haptic representation during generation, grounding action generation in the evolving haptic state.The gated cross-attention is the module’s mechanism for connecting contact observations with the action representation.Source: D. Compliance Grounding Module · p. 5
  • A learnable residual gate regulates each adapted layer’s reliance on haptic evidence.The gate is zero-initialized so the module begins as an identity mapping and preserves pretrained computation at fine-tuning outset.Source: D. Compliance Grounding Module · p. 5

IV. EXPERIMENTS

The experiments compare HACo with existing policies, examine haptic modalities and action-target design, and assess haptic incorporation into a pretrained action model. Each task–method pair receives 20 physical rollouts, with results averaged across five tasks.

IV. EXPERIMENTS

  • The evaluation asks whether HACo outperforms existing policies on force-sensitive dexterous tasks.Each task–method pair is tested over 20 physical rollouts, with results averaged across five tasks.Source: §IV · p. 5
  • The experiments investigate the contributions and complementarity of tactile and joint-torque feedback, and the roles of compliant-action targets and compliant-intent supervision.Source: §IV · p. 5
  • The study also tests how haptic features should be incorporated into the pretrained action model and evaluates wrist-camera observations.Source: §IV · p. 5

A. Experimental Setup

The experiments use a dual-arm dexterous platform with synchronized visual, robot-state, tactile, and joint-torque observations. Policies are trained on task-specific demonstrations and evaluated through physical rollouts.

A. Experimental Setup

  • HACo runs on two UR5 manipulators with Sharpa hands and receives three RGB views, robot kinematics, fingertip tactile feedback, and joint torque.It replans 40-step action chunks in closed loop.Source: A. Experimental Setup · p. 5
  • Each policy is trained on 100 demonstrations per task; HACo and its ablations use the same training configuration.The policies are optimized for 30,000 steps, with HACo initialized from GR00T N1.7.Source: A. Experimental Setup · pp. 5–6
  • Each task–method pair receives 20 physical rollouts, and task success rates are macro-averaged across the five tasks.A rollout counts as successful when the task is completed.Source: A. Experimental Setup · p. 6

B. Real-World Dexterous Force Benchmark

The benchmark comprises five multi-stage bimanual tasks designed to test distinct force-sensitive interactions. Each requires completing the intended motion while regulating the physical interaction that enables it.

B. Real-World Dexterous Force Benchmark

  • Insert poker cards tests multi-contact friction by requiring a card to slide into an occupied grasp without bending or dropping either card.The task pairs alignment and insertion with preservation of both cards.Source: B. Real-World Dexterous Force Benchmark · p. 6
  • Open book tests tangential-force control, while drawing on a balloon tests fragile curved-surface contact without slip, excessive deformation, or rupture.Opening the book requires regulating tangential-force direction and magnitude to turn and hold pages without damage.Source: B. Real-World Dexterous Force Benchmark · p. 6
  • Unscrewing a cap tests rotational torque, and squeezing toothpaste tests controlled compression of a deformable container.The tasks respectively require removing a cap without drops or spills and dispensing toothpaste onto toothbrush bristles.Source: B. Real-World Dexterous Force Benchmark · p. 6

C. Comparison with Existing Policies

HACo outperforms the evaluated policies on the five-task benchmark, and ablations examine haptic sensing, compliance learning, grounding, and wrist-camera observations. The comparisons favor combining haptic feedback with action-aligned compliance learning.

C. Comparison with Existing Policies

  • Under the stated evaluation protocol, HACo attains an 83% mean success rate, exceeding strongest baseline T-Rex (35%) by 48% and GR00T (15%) by 68%, with the advantage holding across all five tasks.Source: C. Comparison with Existing Policies · p. 6
  • T-Rex averages 57.5% on sustained rotational or compressive tasks versus 20% on tasks with more abrupt force transitions; GR00T + Tactile reaches 22%.The passage attributes the modest GR00T + Tactile improvement to concatenation without explicit alignment between contact evidence and the evolving action representation.Source: C. Comparison with Existing Policies · p. 6
  • Each Fig. 6 row shows seven temporally ordered frames of one HACo execution, from contact establishment to task completion.The figure presents representative rollouts on the real-world dexterous force benchmark.Source: Figure 6 · p. 7

D. Ablation Studies

  • Removing all haptic feedback lowers mean success from 83% to 27%; tactile alone reaches 68% versus 45% for torque alone, while adding torque to tactile raises cap-removal success from 70% to 95%.Coupled encoding raises mean success from 70% to 83%.Source: D. Ablation Studies · p. 7
  • Removing compliant-intent supervision while retaining compliant action targets lowers mean success from 83% to 73%; replacing the compliant target with the nominal command lowers it further to 59%.The action target contributes more, while compliant-intent supervision remains complementary; HACo also reduces mean fingertip force by 19% relative to the nominal-action variant.Source: D. Ablation Studies · p. 7
  • Visuo–Haptic Fusion and Action-Suffix Fusion reach 41% and 47%, versus 66% for ungated haptic cross-attention and 83% for the complete Compliance Grounding Module.The passage describes dedicated cross-attention as aligning haptic observations with the evolving action representation.Source: D. Ablation Studies · p. 7
  • Removing wrist cameras reduces mean success by 7%, from 83% to 76%; the passage describes haptic feedback as partly compensating for visually occluded contact states.Wrist views remain complementary.Source: D. Ablation Studies · p. 7

V. CONCLUSION

The conclusion presents HACo as a practical route to closed-loop force-aware dexterity and identifies open challenges in demonstrations, learning beyond teleoperation, and tactile coverage.

V. CONCLUSION

  • HACo combines compliant action targets, command–state discrepancy supervision, and kinematics-aligned tactile and torque feedback through gated haptic cross-attention; it achieves an 83% mean success rate across five real-world tasks.The reported mean is 48% above the strongest evaluated baseline.Source: §V · p. 8
  • The authors identify force-feedback teleoperation, learning beyond motions demonstrable through teleoperation, and dense whole-hand tactile coverage as open challenges.Without dense coverage, palm and phalange contacts are indirectly inferred from joint torque.Source: §V · p. 8
Loading 2609.36596v1…