PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation

Wenqiao Li, Qianyou Zhao, Jiawen Hao, Xuezhou Zhu, Tengyu Liu, Kaifeng Zhang, Chuan Wen, Siyuan Huang

arXiv:2609.34182v2cs.ROcs.AI

TL;DR

Robot tactile demonstrations are difficult to scale, while human tactile interactions provide a more diverse source despite embodiment differences. UVTA synchronizes human tactile motion capture, aligns human and robot representations, and jointly predicts action and tactile futures. Across five real-robot tasks it achieves a 70% average success rate versus 29% for the strongest visual-tactile baseline, with gains continuing through 1,000 human demonstrations per task.

  • Problem

    Dexterous-hand teleoperation makes robot tactile demonstrations costly to scale and limits the diversity of represented objects, environments, and contact patterns.

    Source: §I · p. 2

  • Method

    UVTA aligns human and robot trajectories in tactile and action representations, jointly predicts future action and tactile trajectories, and executes only robot actions at deployment.

    Source: §I · p. 2, §VI · p. 7

  • Results

    70% average success rate across five real-robot tasks compares with 29% for the strongest visual-tactile baseline and 42% for the ablated architecture; performance remains unsaturated at 1,000 human demonstrations per task.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Human tactile demonstrations provide scalable physical supervision for dexterous robot learning, with performance improving as human data scales through 1,000 demonstrations per task.

    Source: §VI · p. 7

    Tactile sensing is limited to fingertips, leaving palm and dorsal-hand contacts unobserved because available whole-hand gloves cannot reliably suppress motion-induced noise during large hand movements.

    Source: APPENDIX IV LIMITATIONS AND FUTURE WORK · p. 10

Abstract

from arXiv · show

Dexterous manipulation requires tactile feedback. However, robot tactile demonstrations are difficult to scale,because dexterous-hand teleoperation provides limited tactile feedback to the operator. In contrast, human demonstrations offer a substantially more scalable source of diverse tactile interactions. Motivated by a simple premise: hands can change, but the underlying physics of interaction does not. We leverage human tactile data to improve dexterous manipulation policies. Specifically, we first build a tactile motion-capture system that synchronously records images, tactile signals, and hand motions. Using this system, we construct the UVTA dataset spanning five contact-rich tasks, with 1,000 human demonstrations covering diverse interaction patterns and 150 robot demonstrations per task. To transfer the underlying physics of human interaction to robot control, we propose a Unified Visual-Tactile-Action Model that maps both embodiments into aligned tactile and action representations and jointly predicts future action and tactile trajectories. The joint objective enables human demonstrations to supervise contact-aware representation learning, while only robot actions are executed during deployment. In real-robot evaluations across five tasks, our method achieves an average success rate of 70%, outperforming the strongest visual-tactile baseline, which achieves 29%, and an architecture ablation, which achieves 42%. Performance improves consistently with additional human demonstrations and exhibits no saturation at 1,000 demonstrations per task, validating the effectiveness of scalable human tactile data for dexterous manipulation. Project page is available at https://uni-vta.github.io/.

I. INTRODUCTION

Robot tactile demonstrations are difficult to scale, while human interactions offer more diverse contact experience despite differences between human and robot hands. UVTA combines synchronized human tactile data with robot demonstrations and jointly predicts action and tactile futures, achieving stronger real-robot results than the reported baselines.

I. INTRODUCTION

  • Robot tactile policy data are costly to scale because dexterous-hand teleoperation is demanding and gives operators limited tactile feedback.These constraints limit the diversity of objects, environments, and contact patterns represented in robot datasets.Source: §I · p. 2
  • Human and robot hands differ in morphology, kinematics, sensing layouts, and action spaces, while contact formation, deformation, slip, release, and object response retain shared physical structure.Video alone does not fully capture local contact dynamics.Source: §I · p. 2
  • A wearable system synchronously records wrist-centric images, fingertip tactile measurements, and hand motion for five contact-rich tasks.The UVTA dataset contains 1,000 diverse human demonstrations and 150 robot demonstrations per task.Source: §I · p. 2
  • 70% average success rate exceeds the strongest visual-tactile baseline’s 29% and the ablated architecture’s 42% across five real-robot tasks.Performance improves with more human demonstrations and shows no saturation at 1,000 demonstrations per task.Source: §I · p. 2
  • The model aligns human and robot trajectories into tactile and action representations, jointly predicts future action and tactile trajectories, and executes only robot actions at deployment.Human demonstrations supervise contact-aware representation learning.Source: §I · p. 2

II. RELATED WORK

Prior approaches use human data through visual representations or motion transfer, but direct policy transfer depends on recoverable motion or aligned actions. UVTA combines human tactile motion-capture and robot teleoperation data, with synchronized sensing and a shared wrist-frame alignment.

II. RELATED WORK

  • Prior human-data transfer often depends on recoverable motion or aligned actions; UVTA co-trains human tactile and robot trajectories with shared action and future-tactile objectives, then deploys robot-action predictions.The approach differs from staged transfer of human touch through aligned human–robot action spaces.Source: §II · p. 2

III. DATA COLLECTION SYSTEM AND DATASET

  • The collection system records wrist fisheye images, fingertip tactile signals, and 22-DoF joint angles, then aligns the signals with Sharpa Wave Hand in kinematic and contact space.The dataset overview reports 1,000 human and 150 robot demonstrations per task.Source: §III · p. 3, Figure 3 · p. 3

A. Tactile Motion-Capture System

  • FSR arrays measure fingertip pressure, trackers and a linkage exoskeleton recover wrist and finger motion, and the wrist image, tactile, pose, and joint streams are resampled at 30 Hz.The camera and tracker are calibrated to the Sharpa Wave wrist frame.Source: A. Tactile Motion-Capture System · pp. 3–4, Figure 2 · p. 3

B. Dataset

The dataset pairs varied human contact sequences with robot demonstrations that provide executable commands. Both embodiments supervise action and future-tactile prediction in a shared representation, while human motions remain training targets rather than robot commands.

B. Dataset

  • Across five contact-rich tasks, each has 150 robot demonstrations and 1,000 human demonstrations, with human recordings varying object instances, scene backgrounds, and glove appearance.Robot collection fixes scene layout, initial configuration, and table height for each task.Source: B. Dataset · p. 4
  • Robot demonstrations provide direct supervision for executable wrist and finger commands, while human demonstrations contribute more varied motion and contact sequences across objects and scenes.Both sources supervise action and future-tactile prediction; human motions are training targets, not directly executable robot commands.Source: B. Dataset · p. 4

IV. METHOD

  • The model aligns both embodiments in a unified cross-embodiment space and conditions action generation and future-tactile prediction on the same wrist-centric visual-tactile observation.Training uses balanced co-training; inference converts only robot-normalized action predictions into wrist and finger commands.Source: §IV · p. 4

A. Problem Formulation

The model learns from synchronized visual-tactile observations to predict future robot or human interaction trajectories, aligning their representations across embodiments. Its action and tactile prediction branches share an observation condition, while tactile predictions are not used for control.

A. Problem Formulation

  • The model jointly predicts future action and tactile trajectories from synchronized wrist images, tactile tokens, and actions in either embodiment.The prediction horizon is 16 steps; robot actions are executable teleoperation commands, while human actions come from consecutive motion-capture states.Source: A. Problem Formulation · pp. 4–5
  • At test time, only the predicted robot action is sent to the controller.The model parameters comprise a visual encoder, action head, and tactile head.Source: A. Problem Formulation · p. 5

C. Model Architecture

  • The model performs early feature-level fusion by direct concatenation, with VTPM supplying a 20-D tactile token normalized using statistics specific to embodiment e.Source: C. Model Architecture · p. 5
  • The action branch generates a 16-step, 31-D chunk using conditional diffusion, with relative wrist targets and absolute finger-joint targets.Each action step contains a 9-D wrist target and 22 finger-joint targets.Source: C. Model Architecture · p. 5
  • The tactile world head is a two-layer MLP that maps the 404-D interaction representation through a 512-D hidden feature to a 320-D output.It directly regresses the complete future tactile trajectory from the noise-free condition.Source: C. Model Architecture · p. 5
  • The tactile target is shifted one step ahead and aligned with the action horizon.Keeping tactile prediction outside the diffusion trajectory prevents tactile denoising errors from entering iterative action generation, while its regression loss still updates the shared visual encoder.Source: C. Model Architecture · p. 5

D. Training Recipe

Training uses a single co-training run over human and robot data, with sampling that balances their contributions. The recipe normalizes each embodiment separately, corrects tactile baselines per recording, and jointly optimizes the model.

D. Training Recipe

  • A weighted sampler gives human and robot datasets equal probability mass, sampling with replacement so dataset frame counts do not set their mixture ratio.The task-specific model trains in one co-training run on the union of both datasets rather than sequential training stages.Source: D. Training Recipe · p. 5
  • Separate min–max normalization places human and robot channels in the same numerical range, and saved robot statistics decode deployment predictions.Tactile signals are baseline-corrected per recording before normalization: using the first five-frame mean for human data and the first frame for robot data.Source: D. Training Recipe · p. 5
  • The denoising objective averages wrist and hand errors separately and weights the two groups equally, preventing the 22-D hand block from dominating the 9-D wrist block.Each MSE is averaged over batch, time, and dimensions within its group.Source: D. Training Recipe · p. 5
  • Training samples a diffusion step uniformly from 50 noise levels and jointly optimizes all modules for 300 epochs with batches of 250 using AdamW.The tactile-loss weight is 0.2 in all experiments.Source: D. Training Recipe · p. 5
  • During rollout, 16-step DDIM sampling produces a 16-step robot action chunk, but the controller executes eight waypoints before incorporating new visual-tactile feedback.The tactile-head output is not used for control; wrist predictions are anchored to the live pose after reversing robot normalization and converting rotations to SO(3).Source: D. Training Recipe · pp. 5–6

V. EXPERIMENTS

The experiments are organized around three questions: the value of human tactile data, its scaling effects, and the importance of architecture and world-model training design.

V. EXPERIMENTS

  • RQ1 asks what value human tactile data provides for contact-rich dexterous manipulation.The passage frames this as an experimental question, not a reported finding.Source: §V · p. 6
  • RQ2 asks whether scaling human tactile data improves real-robot performance.The passage states the question but does not report its answer.Source: §V · p. 6
  • RQ3 asks how important the architecture and training design of the world model are.The passage states the question but does not report its answer.Source: §V · p. 6

A. Experiment Setup

Across five contact-rich tasks, the evaluation compares the full policy with baselines and ablations, while scaling tests vary human demonstrations at a fixed robot-data budget.

A. Experiment Setup

B. Main Results and Ablations

  • 70% average success across five tasks beats T-Rex by 41 percentage points, and the method performs best on every task.Table 1 reports the baseline comparison and ablations across the five tasks.Source: B. Main Results and Ablations · p. 6, Table 1 · p. 7
  • Removing future-tactile prediction lowers average success from 70% to 42%, and removing tactile input as well lowers it further to 26%.The ablations indicate that future tactile supervision and current tactile observations contribute distinct information.Source: B. Main Results and Ablations · pp. 6–7, Table 1 · p. 7
  • 84% average success with 1,000 human trajectories follows 0%, 8%, 29%, and 51% at 0, 250, 500, and 750 trajectories, respectively.The robot-demonstration budget is held at 150 per task; the curve remains unsaturated at 1,000 trajectories.Source: B. Main Results and Ablations · p. 6, Figure 5 · p. 7
  • Proprioceptive conditioning yields 18% average success, with the policy relying on current state and producing behavior resembling replay of a nearly fixed demonstration.The authors connect this reliance to reduced adaptation to changing visual and tactile observations.Source: B. Main Results and Ablations · p. 7, Table 1 · p. 7

VI. CONCLUSION

The conclusion presents UVTA as a way to use scalable human tactile demonstrations for dexterous manipulation. Across five real-robot tasks, its policy outperforms the strongest visual-tactile baseline and continues to benefit as human data increases.

VI. CONCLUSION

  • UVTA combines 1,000 diverse human demonstrations and 150 robot demonstrations per task across five tasks.The dataset is collected with a system recording wrist-centric vision, fingertip tactile measurements, and hand motion synchronously.Source: §VI · p. 7
  • The model aligns human and robot embodiments and jointly predicts future action and tactile trajectories, using human data for contact-aware learning while retaining robot-specific execution.The conclusion frames this as transferring shared interaction physics across differing hand embodiments.Source: §VI · p. 7
  • 70% average success across five real-robot tasks compares with 29% for the strongest visual-tactile baseline; removing future-tactile prediction reduces performance to 42%.Performance continues improving with human data and remains unsaturated at 1,000 demonstrations per task.Source: §VI · p. 7

APPENDIX I ADDITIONAL SYSTEM AND DATA DETAILS

The appendix details how human and robot visual, motion, and tactile streams are calibrated and synchronized, then describes dataset composition and shared implementation choices.

A. Cross-Embodiment Calibration

  • Camera-to-wrist calibration aligns the human and robot visual frames, while tracker measurements and constrained inverse kinematics map human wrist and hand motion into robot-relative targets.The construction avoids requiring a shared global workspace.Source: A. Cross-Embodiment Calibration · p. 9

B. Tactile Processing and Synchronization

  • Four sensing regions on each of five human fingertips yield the same 20-D tactile layout used for robot observations; streams are timestamped and resampled to 30 Hz.Tactile readings are baseline-corrected and normalized using embodiment-specific statistics.Source: B. Tactile Processing and… · p. 9

C. Dataset Composition

  • The dataset contains 5,750 demonstrations across five tasks: 5,000 human and 750 robot demonstrations, with human motion targets used for representation learning but never issued as robot commands.Training samples pair a synchronized wrist image and 20-D tactile token with a 31-D motion target.Source: C. Dataset Composition · p. 9

APPENDIX II IMPLEMENTATION DETAILS

  • Training balances human and robot dataset sampling, computes normalization statistics separately by embodiment, and uses only robot statistics to decode deployment actions.The tactile head regresses a 16 × 20 future trajectory and updates the shared visual encoder, but its prediction is not passed to the controller.Source: APPENDIX II IMPLEMENTATION DETAILS · p. 9

A. Closed-Loop Execution

UVTA generates robot action chunks and replans from refreshed visual-tactile observations during contact. Evaluation uses milestone-based scores that award partial task progress rather than only full completion.

A. Closed-Loop Execution

  • At each update, UVTA samples a 16-step robot action chunk using 16 DDIM reverse steps, then executes eight waypoints before observing again and replanning.Wrist predictions are anchored to the live wrist pose, while finger predictions are absolute joint targets.Source: A. Closed-Loop Execution · p. 9

B. Stage-Wise Success Rate

  • Each method and ablation is evaluated over 10 real-robot rollouts per task, with randomized object positions and heights.The reported rate is a stage-wise score, not the fraction of fully successful trials.Source: B. Stage-Wise Success Rate · p. 9
  • Milestone scores are absolute rather than additive: a rollout scores 0 with no milestone and 1 only upon full task completion.For page turning, multiple pages receive 0.7 while exactly one page receives 1.0.Source: B. Stage-Wise Success Rate · p. 9

C. Human-Data Scaling Protocol

The scaling study varies human-data volume under fixed robot data and evaluation settings, while the limitations identify constraints in tactile coverage and experimental scale.

C. Human-Data Scaling Protocol

  • Human demonstrations increase from 0 to 250, 500, 750, and 1,000 per task while robot data stays fixed at 150; the study covers three tasks.The same policy architecture, training objective, and evaluation procedure are used at every data scale.Source: C. Human-Data Scaling Protocol · p. 9
  • Fig. 6 shows representative successful rollouts for five tasks, with time progressing left to right in each task row.The rows are Flip Page, Screw Light Bulb, Toggle Switch, Ball Classification, and Liquid Transfer with a Dropper.Source: Figure 6 · p. 10
  • Evaluation scores the highest milestone achieved, gives zero when none are achieved, and uses 1.0 for full task success.Source: Table 3 · p. 10

APPENDIX IV LIMITATIONS AND FUTURE WORK

  • Fingertip-only sensing leaves palm and dorsal-hand contacts unobserved, limiting the interaction patterns represented in the dataset.Extending coverage requires improved sensor integration and compensation for motion-induced artifacts.Source: APPENDIX IV LIMITATIONS AND FUTURE WORK · p. 10
  • Experiments cover five laboratory tasks, one robot platform, task-specific policies, at most 1,000 human demonstrations per task, and one model scale.The results therefore do not establish behavior with substantially larger and more diverse datasets, larger models, or new robot embodiments.Source: APPENDIX IV LIMITATIONS AND FUTURE WORK · p. 10

Related papers

Loading 2609.34182v2…