PaperlayerView exact revision on arXiv

Source-linked AI summary

TacOT: Learning Contact-Rich Dexterous Manipulation from Human Demonstrations via Tactile-Guided Optimal Transport

Xingting Li, Yifan Han, Zijian Lin, Wei Hou, Chuqiao Lyu, Shoujie Li, Wenbo Ding

arXiv:2610.04363v1cs.RO

TL;DR

Contact-rich dexterous manipulation is difficult to transfer from human demonstrations because visual and motion similarity can pair interactions with different contact states. TacOT uses action–tactile DTW to guide soft optimal transport alignment, improving success over action-guided OT on four real-world tasks. The results support tactile dynamics as a semantic signal for human–robot correspondence.

  • Problem

    Visual or motion similarity can associate human and robot demonstrations with similar motion but different contact states and force patterns.

    Source: §I · p. 1

  • Method

    TacOT combines action and tactile DTW to identify interaction-consistent demonstration pairs and uses them to guide soft optimal transport alignment in a shared policy representation space.

    Source: §I · p. 2

  • Results

    TacOT improves closed-loop success rates over action-guided OT by up to 17 points in-distribution and 20 points under targeted human-to-robot transfer across four real-world tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Tactile dynamics provide an effective semantic signal for establishing reliable human-to-robot correspondence in contact-rich dexterous manipulation.

    Source: §V · p. 8

Abstract

from arXiv · show

Learning contact-rich dexterous manipulation from human demonstrations provides a scalable source of interaction data, yet transferring such skills to robots remains challenging due to unreliable human--robot correspondence. Existing human-to-robot transfer methods typically rely on visual appearance or motion similarity, which may associate similar motions with different contact states and force patterns. Tactile dynamics provide interaction-aware cues to distinguish manipulation processes with similar motions but different contact states. We introduce tactile-guided optimal transport (TacOT), a framework for human-to-robot contact-rich manipulation. TacOT leverages action--tactile dynamic time warping to identify human--robot demonstration correspondences with consistent interaction dynamics and uses these correspondences to guide soft optimal transport alignment in a shared policy representation space. This enables human demonstrations to provide contact-rich supervision for robot policy learning without requiring predefined frame-level human--robot pairing. Across four real-world dexterous manipulation tasks, TacOT improves closed-loop success rates over action-guided OT by up to 17 points on in-distribution tasks and 20 points under targeted human-to-robot out-of-distribution transfer. Further analyses show that tactile-guided correspondence selects demonstration pairs with more consistent contact dynamics and produces latent representations that better reflect interaction-state evolution. These results demonstrate that tactile dynamics provide an effective semantic signal for establishing reliable human-to-robot correspondence in contact-rich dexterous manipulation.

I. INTRODUCTION

TacOT addresses unreliable human–robot correspondence in contact-rich manipulation by using action–tactile dynamics to align demonstrations. It improves success over action-guided OT across real-world tasks.

I. INTRODUCTION

  • Human demonstrations offer interaction diversity, but dexterous robot demonstrations are slow, labor-intensive, and difficult to scale across objects, scenes, and contact conditions.Critical dexterous-manipulation variation includes how contact is established, maintained, and released, beyond appearance and motion.Source: §I · p. 1
  • Similar wrist motions can reflect different contact phases, so visual or motion similarity may align demonstrations with different finger contacts and force distributions.TacOT treats temporally extended action–contact patterns, rather than individually similar frames, as the relevant correspondence unit.Source: §I · pp. 1–2
  • TacOT combines action and tactile DTW distances to identify interaction-consistent human–robot pairs, then reduces their latent transport costs while retaining soft coupling across pairs.The alignment is jointly optimized with a multimodal flow-matching objective in a shared policy space anchored by robot data.Source: §I · p. 2
  • The paper formulates contact-rich human-to-robot imitation as interaction-consistent correspondence learning, using joint action–tactile dynamics to guide soft optimal transport alignment.Tactile dynamics serve as a supervision signal for matching demonstrations, not only as policy observations.Source: §I · pp. 1–2
  • 17 percentage points in-distribution and 20 percentage points under targeted human-to-robot transfer: TacOT improves success rates over action-guided OT across four real-world tasks.The framework combines a matched-sensing human–robot platform and multimodal flow-matching policy for joint learning from visual, motion, and tactile data.Source: §I · p. 2

II. RELATED WORK

Prior work broadens human and tactile supervision, but human-to-robot dexterous transfer still requires representation alignment. TacOT uses matched-sensing systems to collect multimodal human and robot demonstrations independently.

II. RELATED WORK

  • Dexterous imitation methods broaden supervision, while tactile-transfer work learns reusable representations; TacOT additionally uses tactile trajectories to match human–robot demonstrations during policy learning.Human-to-robot transfer still requires representation alignment and often robot-specific adaptation data.Source: §II · p. 2
  • For dexterous manipulation, similar motions can involve different contact states; TacOT combines end-effector motion, hand actions, and tactile evolution to identify correspondences consistent in motion and contact.Earlier human–robot approaches described here use visual and action-space alignment or end-effector trajectory pseudo-pairs.Source: §II · p. 2

III. SYSTEM AND METHOD

  • TacOT collects synchronized visual, motion, and tactile observations from matched human and robot systems, then converts them into unified representations for policy learning and latent alignment.The overview identifies hardware collection, representation alignment, flow-policy learning, and action–tactile-guided latent alignment as the method stages.Source: §III · p. 2, Figure 3 · p. 3

A. Hardware Design and Data Collection

  • The human system records natural hand–object interactions, while teleoperation maps human motion to executable robot commands; the datasets are collected independently without shared timestamps or recorded pairings.Both systems collect visual, motion, and tactile observations; the robot system records robot-side trajectories, fingertip forces, and dual-view RGB.Source: A. Hardware Design and Data Collection · p. 3

B. Human-Robot Representation Alignment

Human and robot observations are converted into shared visual, motion, and tactile representations despite embodiment-specific kinematics. The conversions standardize spatial references, hand actions, and fingertip forces.

B. Human-Robot Representation Alignment

  • Human and robot raw representations differ because of embodiment-specific kinematics, so demonstrations are converted into unified visual, motion, and tactile representations.Both systems record synchronized multimodal observations at 30 Hz, including wrist-view and third-person RGB images.Source: B. Human-Robot Representation Alignment · p. 3
  • Twelve semantically corresponding human joints are mapped to XHand angular ranges and order, yielding a unified 12-DoF hand representation.End-effector actions use camera-frame position offsets and wrapped roll–pitch–yaw increments; hand actions use absolute XHand joint angles.Source: B. Human-Robot Representation Alignment · p. 3
  • Fifteen fingertip-force channels are transformed into a shared canonical frame and consistent finger-axis order, with action and tactile signals normalized independently.The 15 channels come from three-axis force measurements at five fingertips.Source: B. Human-Robot Representation Alignment · p. 3

C. Multimodal Policy Learning via Flow Matching

TacOT encodes multimodal observations into context tokens used by an action Transformer to predict action chunks through conditional flow matching. During deployment, the policy acts from current observations without demonstration trajectories or alignment procedures.

C. Multimodal Policy Learning via Flow Matching

  • Each observation combines dual-view RGB, camera-frame pose and 12-DoF hand state, and current 15-dimensional fingertip forces.Modality-specific encoders and eight learnable context tokens feed a multimodal observation Transformer.Source: C. Multimodal Policy Learning via Flow… · p. 4
  • Encoded context tokens serve as cross-attention memory for the action Transformer and as representations for human–robot alignment.The policy generates action chunks conditioned on current multimodal observations.Source: C. Multimodal Policy Learning via Flow… · pp. 3–4
  • Conditional flow matching trains the action Transformer to predict a conditional velocity field for action chunks.Training samples flow time and Gaussian noise for the demonstrated action chunk.Source: C. Multimodal Policy Learning via Flow… · p. 4
  • At inference, the robot generates action chunks by integrating the learned velocity field from Gaussian noise; deployment requires neither demonstration trajectories nor DTW or OT.Demonstration trajectories are used during training to measure interaction consistency and guide latent alignment.Source: C. Multimodal Policy Learning via Flow… · p. 4

D. Action–Tactile-Guided Latent Alignment

TacOT uses joint action–tactile dynamics to identify interaction-consistent human–robot correspondences and guide latent alignment. The selected pairs receive lower transport costs while other matches remain possible.

D. Action–Tactile-Guided Latent Alignment

  • Action and tactile DTW distances are computed for every human–robot chunk pair, robustly normalized by median and interquartile range, then fused.Local costs use weighted action MSE and tactile MSE; tactile channels are standardized with fixed per-embodiment training-set statistics.Source: D. Action–Tactile-Guided Latent… · p. 4
  • Each robot chunk selects one preferred human chunk using the fused action–tactile correspondence, which discounts that pair’s latent transport cost.The selection does not make the coupling one-to-one.Source: D. Action–Tactile-Guided Latent… · p. 4
  • Encoded context tokens are flattened and L2-normalized before latent transport costs are computed between human and robot chunks.TacOT uses action and tactile trajectories because similar motions can correspond to different contact states and interaction outcomes.Source: D. Action–Tactile-Guided Latent… · p. 4
  • TacOT discounts the latent transport costs of action–tactile-selected human–robot chunk pairs, favoring those matches while preserving soft many-to-many coupling.The correspondence guides alignment without excluding other pairs.Source: D. Action–Tactile-Guided Latent… · p. 4
  • The alignment loss uses entropy-regularized optimal transport between uniform empirical human and robot latent representations.The joint objective updates the observation encoder through both losses, while only the flow-matching loss updates the action decoder; DTW and OT add no inference overhead.Source: D. Action–Tactile-Guided Latent… · pp. 4–5

IV. EXPERIMENTS

The experiments assess TacOT on four real-world dexterous manipulation tasks using closed-loop success rates, including in-distribution comparisons and targeted transfer to robot-unseen configurations represented in human demonstrations.

IV. EXPERIMENTS

  • The evaluation uses closed-loop success rates across four real-world dexterous manipulation tasks.It includes in-distribution baseline comparisons and targeted human-to-robot transfer to configurations unseen during robot training but present in human demonstrations.Source: §IV · p. 5
  • The experiments analyze action–tactile correspondence and latent representations alongside the task evaluations.These analyses examine correspondence, latent representations, and varying numbers of human and robot demonstrations.Source: §IV · p. 5

A. Experimental Setup

The setup evaluates Tissue, Wiping, Egg, and Block on the same UR5–XHand platform and closed-loop protocol, with task-specific binary success criteria. Episode-level splits support in-distribution evaluation and transfer to configurations represented in human demonstrations but excluded from robot training.

A. Experimental Setup

  • The four tasks are Tissue, Wiping, Egg, and Block, evaluated on the same UR5–XHand platform with closed-loop control.They cover flexible-object handling, sustained surface interaction, fragile-object manipulation, and geometry-sensitive insertion.Source: A. Experimental Setup · p. 5
  • Success is binary and task-specific: placing tissue in a designated region, removing wiping marks, placing an egg intact in a container, or fully inserting a block.The same criteria apply to ID and OOD conditions.Source: A. Experimental Setup · p. 5
  • Human-augmented methods use 60 robot ID episodes and 200 human demonstrations, while Robot-only uses the same robot episodes without human data.The human demonstrations comprise 100 ID and 100 OOD episodes; each OOD configuration is represented in human demonstrations but excluded from robot training.Source: A. Experimental Setup · p. 5
  • ID evaluation uses 30 rollouts per method–task pair, and OOD evaluation uses 15 rollouts per pair.The primary metric is complete-task binary success rate.Source: A. Experimental Setup · p. 5
  • Table I reports in-distribution real-world evaluation results.The caption identifies the table’s scope but does not provide its cell values.Source: Table I · p. 5
  • TacOT uses a 16-layer Transformer observation encoder and an 8-layer Transformer action decoder, each with hidden dimension 256 and 8 attention heads.Its weights are wa = 0.7, wc = 0.3, α = 0.7, and λ = 0.1; deployment replans after executing the first 8 predicted actions at 30 Hz.Source: A. Experimental Setup · p. 5
  • TacOT is compared with Robot-only, Co-train, Standard OT, MimicPlay, EgoMimic, and two action-guided OT variants.The action-guided OT variant with tactile observations shares TacOT’s policy architecture and OT objective but uses action-only DTW correspondence; the other removes tactile policy inputs.Source: A. Experimental Setup · p. 5

B. Real-Robot Manipulation Evaluation

TacOT achieves best or tied-best in-distribution success across four tasks, while comparisons indicate tactile inputs benefit both policy execution and correspondence selection.

1) In-Distribution Performance:

2) Human-to-Robot Transfer:

The transfer protocol tests robot-unseen configurations represented in human demonstrations; offline analysis finds that TacOT retrieves pairs with less tactile mismatch while maintaining comparable action mismatch.

2) Human-to-Robot Transfer:

  • Targeted transfer evaluates configurations absent from robot training but represented in human demonstrations, comparing Robot-only, action-guided OT, and TacOT.Source: 2) Human-to-Robot Transfer · p. 6

1) Offline Correspondence Analysis:

  • The offline comparison uses identical robot queries and human candidate pools, with action-guided OT selected by action DTW and TacOT by action–tactile matching score.Source: 1) Offline Correspondence Analysis · pp. 6–7
  • TacOT reduces mean tactile DTW mismatch versus action-guided OT on Tissue from 4.227 to 2.325, Wiping from 2.644 to 1.158, Egg from 3.963 to 1.457, and Block from 3.129 to 1.835.Source: 1) Offline Correspondence Analysis · p. 7
  • TacOT's retrieved pairs have lower tactile mismatch across all four tasks while retaining a comparable action-mismatch range, and lower mean action and tactile mismatch than Standard OT and Co-train.Source: 1) Offline Correspondence Analysis · p. 7

2) Offline Model Analysis:

TacOT's learned latent space better preserves interaction dynamics and improves successive replanning consistency across the four tasks.

2) Offline Model Analysis:

D. Demonstration Efficiency Analysis

TacOT's benefit from human demonstrations is greatest when robot data is limited, while additional robot demonstrations consistently improve performance. The evaluation remains bounded to a fixed robot embodiment and RGB observations without explicit 3D geometry.

D. Demonstration Efficiency Analysis

  • With 60 robot demonstrations fixed, adding human data raises Block success from 47% to 73%, while increasing robot demonstrations consistently improves performance across all tasks.The results suggest robot demonstrations provide stronger task-specific execution supervision, while human demonstrations provide complementary behavioral information.Source: D. Demonstration Efficiency Analysis · p. 8
  • From 30 to 100 human demonstrations, Wiping success rises from 47% to 80% and Block success from 53% to 73%; added human-data benefits shrink as robot demonstrations increase.Source: D. Demonstration Efficiency Analysis · p. 8

V. CONCLUSION

  • TacOT achieves the best or tied-best in-distribution success rates among evaluated methods and improves transfer to human-demonstrated configurations unseen in robot training.Tactile dynamics also yield more contact-consistent matches and stronger alignment between latent similarity and interaction contact dynamics.Source: §V · p. 8
  • The current evaluation uses a fixed robot embodiment and RGB observations without explicit 3D geometry.Future work will extend TacOT to heterogeneous morphologies and incorporate depth and multi-view geometry for more general transfer settings.Source: §V · p. 8

Related papers

Loading 2610.04363v1…