PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data

Hyesung Lee, Si-Hwan Heo, Sungwook Yang

arXiv:2609.16504v1cs.RO

TL;DR

Human videos lack robot-executable actions and tactile measurements for dexterous manipulation. UniDex-ViTac uses human-guided simulation to train a reference-free visuo-tactile policy, achieving higher reported success than the point-cloud-only baseline in simulation and physical trials.

  • Problem

    Human videos offer dexterous manipulation demonstrations but lack robot-executable actions and tactile measurements.

    Source: §I · p. 1

  • Method

    Object-specific residual RL specialists adapt human references in simulation to generate robot action targets paired with contact observations for training a deployable generalist.

    Source: §I · p. 1, D. Robot Demonstrations from… · p. 3

  • Results

    68.3% macro-average success in simulation versus 55.5% for the point-cloud-only baseline, and 73/110 physical successes versus 60/110, an 11.8-percentage-point increase.

    Source: §V · p. 7

  • Takeaways & Limitations

    The results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions.

    Source: §V · p. 7

    The current evaluation is limited to a single grasp-and-lift skill and four binary fingertip signals, with a discrepancy between simulated contact regions and physical tactile-sensor regions.

    Source: §V · p. 7

Abstract

from arXiv · show

Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human-object interaction references to a robotic arm-hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions. Project page: https://unidex-vitac.github.io/

I. INTRODUCTION

Human videos avoid complex robot teleoperation but lack directly executable robot kinematics and tactile measurements. UniDex-ViTac uses simulated robot interactions to turn human references into action–contact demonstrations for training a reference-free visuo-tactile generalist.

I. INTRODUCTION

  • Human videos lack robot-executable kinematics and contact measurements, complicating the learning of contact-aware policies for dexterous hands.The paper frames the problem as translating human movements to robot kinematics while obtaining tactile information absent from passive recordings.Source: §I · p. 1
  • Object-specific residual RL specialists adapt human interaction references in simulation and generate robot action targets paired with contact observations.Successful rollouts train one generalist for reference-free execution.Source: §I · p. 1, Figure 1 · p. 2
  • The policy encodes four binary fingertip contacts through spatial fingertip labels and a separate contact token, alongside point clouds and proprioception.At test time it does not require human references, object identities, or ground-truth object poses.Source: §I · p. 1
  • The sim-to-real implementation uses 50 annotated human demonstrations and 10,000 simulated trajectories, then evaluates on six seen and five unseen physical objects.The physical evaluation uses neither real-robot demonstrations nor policy fine-tuning.Source: §I · p. 1

II. RELATED WORK

Prior work connects human-video references, simulated demonstrations, shared policies, and visuo-tactile representations. UniDex-ViTac combines these directions by generating pooled specialist trajectories with geometry and contact observations for a shared ACT policy.

A. Learning Dexterous Manipulation from Human Videos

  • Human-video references provide hand–object geometry, but transferring them to robots requires accounting for embodiment and contact dynamics.Simulation-based pipelines use successful rollouts to connect geometric references to deployable policies.Source: A. Learning Dexterous Manipulation… · p. 2

B. Specialist Distillation and Sensory Policy Learning

  • Specialist-to-generalist learning separates object-specific control from a shared deployment policy; this work trains one ACT policy on pooled specialist trajectories.Reference-guided adaptation remains in data generation, while the pooled trajectories contain geometry and contact observations.Source: B. Specialist Distillation and Sensory… · p. 2

C. Visuo-Tactile Policy Learning

  • UniDex-ViTac encodes fingertip contact through spatial point labels and a separate contact token for arm–hand grasp-and-lift manipulation.Its robot-side contact observations come from human-reference-guided simulated interactions and are retained in the offline dataset.Source: C. Visuo-Tactile Policy Learning · p. 2

III. METHOD

The method learns one deployable policy from human grasp-and-lift videos by using object-specific specialists to generate paired sensory observations and final task-space targets. Human hand and object annotations provide the references, with five demonstrations selected for each of ten objects.

A. Problem Formulation

  • The objective is one policy conditioned on scene point clouds, robot proprioception, and binary fingertip contacts, trained from specialist rollouts.Specialists use privileged simulated state and human references; their rollouts pair sensory observations with final task-space targets.Source: A. Problem Formulation · p. 2

B. Human–Object Interaction References

  • DexYCB annotations provide hand parameters and object poses that form human–object interaction references mapped to robot wrist and fingertip targets.The object trajectory conditions the specialist and supplies its tracking objective.Source: B. Human–Object Interaction References · pp. 2–3
  • The dataset retains five demonstrations for each of ten DexYCB objects, totaling 50, and excludes sequences unsuitable for the simulated tabletop setup.Excluded initial placements include unsupported or unstable configurations.Source: B. Human–Object Interaction References · p. 3

C. Object-Specific Residual RL

Object-specific residual RL specialists adapt mapped human references to the robot through bounded task-space corrections. Training uses privileged observations, smoothed residual commands, and randomized references and object properties.

C. Object-Specific Residual RL

  • One PPO specialist per object adapts mapped references to robot kinematics and contact dynamics using bounded task-space corrections.Specialists are trained in Isaac Lab with 8,192 parallel environments per policy.Source: C. Object-Specific Residual RL · p. 3
  • The privileged observation includes joint, object, and fingertip motion information, while object features include physical properties and six-degree-of-freedom pose.Actor and critic receive the privileged observation; the passage also identifies binary contacts and continuous contact-force features.Source: C. Object-Specific Residual RL · p. 3
  • Residual commands are smoothed and accumulated into task-space corrections bounded at ±0.1 m translation, ±40° wrist orientation, and ±0.05 m fingertip positions.The raw task-space residual command is predicted at 20 Hz.Source: C. Object-Specific Residual RL · p. 3
  • Training broadens the simulated distribution by translating and rotating references and randomizing object physical properties and scale.Each specialist is trained for 1,000 policy updates.Source: C. Object-Specific Residual RL · p. 3

D. Robot Demonstrations from Specialist Rollouts

Successful specialist rollouts form an offline dataset pairing robot observations, including contact, with final corrected action targets. A single ACT-based generalist uses point-cloud, proprioceptive, and contact inputs to predict action chunks.

D. Robot Demonstrations from Specialist Rollouts

  • 10,000 retained trajectories, with 1,000 per object, enter the offline dataset only after each candidate succeeds in five perturbed verification rollouts.A trial requires grasping the object, lifting it at least 20 cm, and holding it for at least 3 s.Source: D. Robot Demonstrations from… · p. 3
  • Each trajectory pairs sensory observations containing robot-side contact with the specialist’s final corrected task-space target, supporting reference-free execution.The pooled trajectories comprise observation–action sequences.Source: D. Robot Demonstrations from… · p. 3

E. Unified Visuo-Tactile Generalist

  • The generalist observation combines a point cloud, proprioception, and binary fingertip contacts, with fingertip labels and a separate four-bit contact token.The point cloud includes 508 camera points and four FK fingertip positions; contact activation is spatially anchored at the fingertips.Source: E. Unified Visuo-Tactile Generalist · p. 3, Figure 2 · p. 4
  • The action representation specifies a wrist target and four wrist-relative fingertip targets, totaling 21 components and shared by specialist demonstrations and generalist predictions.A damped least-squares arm IK solver tracks the wrist target, while analytical hand IK tracks fingertip targets.Source: E. Unified Visuo-Tactile Generalist · pp. 3–4
  • The ACT-based Transformer predicts 30-step action chunks conditioned on encoded point-cloud, proprioceptive, and contact inputs, with a CVAE latent used during training.At deployment, the CVAE encoder is discarded, the latent is fixed to the prior mean, and temporal ensembling combines overlapping chunk predictions.Source: E. Unified Visuo-Tactile Generalist · p. 4, Figure 3 · p. 4

F. A Four-Bit Tactile Interface

A shared four-bit interface represents fingertip contact in simulation and on hardware, using force thresholds in simulation and calibrated pressure sensors on the robot.

F. A Four-Bit Tactile Interface

  • Each fingertip has one contact bit, set in simulation when its total contact force exceeds 1.0 N and stored in the generalist demonstrations.A bit indicates above-threshold contact with the object or environment.Source: F. A Four-Bit Tactile Interface · p. 4
  • On hardware, each finger’s pressure sensors are calibrated with a constant 1.0 N force applied for 10 s, and its bit activates when any sensor exceeds its threshold.The real sensing region can extend beyond the sensor footprint.Source: F. A Four-Bit Tactile Interface · p. 4

IV. EXPERIMENTS AND RESULTS

The experiments address specialist-to-generalist transition, observation and contact-fusion design, and deployment on seen and unseen objects.

IV. EXPERIMENTS AND RESULTS

  • RQ1 examines specialist data generation and the transition to a generalist.Source: §IV · p. 4
  • RQ2 compares generalist observation configurations, while RQ3 examines contact-fusion design.Source: §IV · p. 4
  • RQ4 evaluates physical deployment on seen and unseen objects.Source: §IV · p. 4

A. Experimental Setup

The setup evaluates simulated policies across ten objects and physical policies across six seen and five unseen objects, using matched observation definitions and a shared arm–hand IK interface.

A. Experimental Setup

  • Simulation training and evaluation use Isaac Lab, ten selected objects, and initial perturbations up to ±15 cm in the xy-plane and ±15° yaw.Evaluation uses a simulated depth camera at the physical camera pose and the 20 cm lift, 3 s hold success criterion.Source: A. Experimental Setup · pp. 4–5
  • Specialist and generalist simulation evaluations use five seeds and 100 trials per seed and object, reporting pooled per-object rates and their unweighted mean.This yields 500 evaluation trials per object.Source: A. Experimental Setup · p. 5
  • State-based baselines receive proprioception, one-hot object identity, and ground-truth object pose, whereas PCD+Contact uses labeled fingertip points and a separate contact token.All generalists train on the same 10,000 specialist trajectories; cross-architecture comparisons also vary training configuration, while ACT variants assess observation design.Source: A. Experimental Setup · p. 5
  • Physical evaluation uses a Franka Panda arm, a 16-DoF hand, a RealSense L515, and the same observation definitions as simulation.The nominal command rate is 20 Hz with a 30-action horizon; six seen and five unseen objects receive ten trials per object per policy.Source: A. Experimental Setup · p. 5

B. From Specialists to One Generalist (RQ1)

Object-specific specialists supply successful trajectories for one generalist, whose 68.3% final evaluation exceeds the Multi-Task RL training endpoint. The generalist’s macro-average remains below the specialists’ result.

B. From Specialists to One Generalist (RQ1)

C. Observation Configurations (RQ2)

The point-cloud configuration gains from adding contact, and the fusion comparisons show a larger improvement from combining spatial fingertip labels with a contact token than from either path alone.

C. Observation Configurations (RQ2)

  • 68.3% PCD+Contact macro-average success compares with 55.5% for PCD Only, a 12.8-point difference.The augmented representation adds labeled FK fingertip points and a separate contact token.Source: C. Observation Configurations (RQ2) · p. 6
  • State-based DP and BC-T achieve 45.2% and 20.2%, respectively.These baselines use privileged object-state inputs and differ in architecture from the proposed sensory policy.Source: C. Observation Configurations (RQ2) · p. 6

D. Contact-Fusion Design (RQ3)

  • 56.8% One-hot Only and 56.4% Token Only compare with 68.3% PCD+Contact and 55.5% PCD Only.The single paths add 1.3 and 0.9 percentage points, while both paths yield the full 12.8-point increase; complementary conditioning is offered as a possible explanation.Source: D. Contact-Fusion Design (RQ3) · p. 6
  • One-hot Only ranks higher on mustard bottle, while PCD+Contact ranks higher on potted meat can and power drill.The ranking varies by object.Source: D. Contact-Fusion Design (RQ3) · p. 6

E. Transfer to Physical Objects (RQ4)

In physical grasp-and-lift trials, PCD+Contact outperformed PCD Only overall and on unseen objects, though object-level outcomes varied. The conclusion notes that evaluation covers one skill and uses a sparse tactile interface.

E. Transfer to Physical Objects (RQ4)

V. CONCLUSION AND LIMITATIONS

  • The results support feasibility of retaining simulated robot-side contact observations when learning a unified sensory policy from human-video-guided interactions.Source: §V · p. 7
  • Evaluation covers grasp-and-lift only; four binary contacts omit force magnitude and detailed contact location, and simulated and physical contact regions differ.Source: §V · p. 7

Related papers

Loading 2609.16504v1…