Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction

1 Introduction

• Human videos offer abundant, diverse demonstrations, but lack the tactile feedback needed for contact-rich interaction, while robot demonstrations and tactile data collection are costly to scale.The stated sim-to-real challenges include perception noise, actuation delays, and inaccurate contact modeling.Source: §1 · p. 2
• DEX-X uses simulated physical interactions as a tactile completion engine, reconstructing contact signals absent from human videos for tactile-aware policy learning.This combines human-video-derived motion priors with tactile-aware reinforcement learning.Source: §1 · p. 2
• 93% success on real-world cube picking and 53% on table cleaning were achieved without real-world fine-tuning, with generalization to unseen object geometries on picking tasks.The framework also demonstrates zero-shot transfer across grasping and tool-use tasks on a 29-DoF dexterous hand-arm platform.Source: §1 · p. 2
• The training framework uses a privileged state-based expert with demonstration references, object states, and tactile contact information, then distills it into a multi-task visual-tactile policy.The deployable policy uses a unified contact point cloud that embeds fingertip tactile feedback into scene geometry and fuses vision, force, and proprioception.Source: §1 · p. 2, Figure 2 · p. 3

Dexterous Manipulation from Human Demonstrations

• Prior human-demonstration work spans demonstration-guided learning, human grasp affordances, and imitation or policy learning from videos, with embodiment transfer remaining a central challenge.Approaches include motion retargeting, trajectory optimization, and reinforcement learning over reconstructed hand-object trajectories [8] [7] [13] [14].Source: §2 · p. 3

Sim-to-Real Transfer for Dexterous Manipulation

• Sim-to-real dexterous manipulation includes grasping and contact-rich tool use, but transfer remains challenging when contact dynamics, tactile sensing, or actuation differ [24] [25].Prior directions include simulation, policy distillation, force-aware transfer, and demonstration-guided tool-use methods [28] [3] [29] [30] [31].Source: §2 · p. 3

Visual-Tactile Manipulation

• Visual-tactile work explores tactile representations, force-aware policies, and spatially grounded sensing; DEX-X instead reconstructs video demonstrations as simulated interactions for sustained tool-object-environment interaction.Robot Synesthesia [38] is identified as especially close in embedding tactile information into a unified point-cloud representation.Source: §2 · p. 3

3 DEX-X: Learning Visual-Tactile Manipulation from Human Videos

• The framework has three stages: reconstruct and retarget human interactions, train a privileged-state policy with tactile feedback, then distill it into a deployable controller.The final controller uses point clouds, tactile observations, and proprioception for zero-shot real-world transfer.Source: §3 · p. 4

3.1 Problem Formulation

• The problem is to learn a multi-task visual-tactile policy that reproduces monocular human-demonstrated behaviors under varying initial conditions despite the mismatch between video supervision and contact-rich robot sensing.The approach reconstructs interactions in simulation to obtain missing tactile supervision and aims for deployment without further fine-tuning.Source: §3.1 · p. 4

3.2 Motion Prior Data Extraction with Spatial Augmentation

• Monocular videos are reconstructed at 30 Hz into hand-object trajectories, then retargeted to robot arm and hand motions and augmented with shared planar translation and yaw perturbations.Object poses use FoundationPose [41], while hand poses use WiLoR [42] with temporal-consistency and hand-object-penetration refinement; the retargeted references condition training.Source: §3.2 · p. 4

3.3 State Expert Training

• A closed-loop state-based expert is trained with reinforcement learning using retargeted references for initialization and motion conditioning, while exploring contact-rich interactions with fingertip force feedback.The expert acts as a privileged teacher for policy distillation; simulated tactile sensing provides five fingertip force magnitudes with latency, noise, and dropout randomization.Source: §3.3 · p. 5

3.4 Policy Distillation and Sim-to-Real Transfer

• DAgger distills the expert into a multi-task student that combines task references, proprioception, fingertip force feedback, and a learned point-cloud feature for real-time closed-loop deployment.The point cloud includes scene points, robot-hand keypoints, and tactile surface points; deployment uses real-time depth and tactile observations.Source: §3.4 · p. 5

4.1 Experimental Setup

• The benchmark covers six task categories, from pick-up and tool use to in-hand manipulation and bimanual handover, on a 29-DoF hand-arm platform.Training used PPO across 4096 parallel environments; policies were deployed at 30 Hz.Source: §4.1 · p. 6

4.2 Q1: Multi-Task Dexterous Manipulation in Simulation

• 65.9% average success across six categories exceeded ManipTrans at 21.9% and kinematic retargeting at 5.2%; DEX-X reached 61.7% versus DAPG's 40.6% on five single-hand categories.Imitation-based policies often lost stable contact during extended interactions, whereas DEX-X used simulated fingertip force feedback for closed-loop adaptation.Source: §4.2 · p. 6

4.3 Q2: The Role of Scene and Tactile Representations

• Point clouds with local 3D force achieve the best average performance, while removing force generally reduces performance, especially for point-cloud observations.Figure 4 compares depth versus point-cloud scenes and force versus no-force conditions; dark and light bars denote strict and relaxed criteria.Source: Figure 4 · p. 7

4.4 Q3: Real-World Deployment and Generalization

• 28/30 cube picks, 24/30 cup pours, 22/30 cup lifts, and 16/30 squeegee manipulations succeeded without additional fine-tuning.The lower squeegee success reflects the challenge of maintaining a stable grasp and sustained contact over a longer interaction horizon.Source: §4.4 · p. 7
• On Figure 5's 30 trials per task, cube picking reached 93% final success and squeegee manipulation 53%, with squeegee failures concentrated in grasp and post-grasp events.The figure tracks cumulative success through grasp acquisition, object grasp, post-grasp stability, and final manipulation.Source: Figure 5 · p. 8
• 23/30 trials succeeded on an unseen thin cube versus 28/30 on the training cube; larger geometry changes yielded 7–8/30 on an unseen square cube and two toy ducks.The supplied Table 3 caption identifies the comparison as zero-shot object generalization on cube picking.Source: Table 3 · p. 7, §4.4 · p. 8

• DEX-X reconstructs monocular human demonstrations as simulated hand-object interactions, using contact dynamics to provide tactile supervision absent from the videos.It combines human-video-derived motion references with tactile-aware reinforcement learning.Source: §5 · p. 8
• The learned policies outperform imitation and retargeting baselines across grasping and contact-rich tool-use tasks, transfer without additional fine-tuning, and generalize zero-shot to unseen object geometries.Real-world sensory ablations also support the importance of both visual and tactile feedback for robust deployment.Source: §5 · p. 8
• The paper presents simulated physical interaction as a bridge from abundant human video demonstrations to deployable dexterous robot policies.It identifies this approach as a path toward scaling robot skill learning from human video data.Source: §5 · p. 8

6 Limitations

• Robust long-horizon interaction and broad generalization remain challenging; performance is sensitive to perception errors, controller mismatch, and large geometry or category shifts.The deployed policy still relies on retargeted motion references; broader skills will require more diverse demonstrations, simulated interactions, and flexible task conditioning.Source: §6 · p. 8
• Tactile input is limited to per-fingertip force magnitudes, excluding pressure, shear, slip, and contact-patch signals.The paper notes that higher-resolution tactile sensing could support more precise contact reasoning and force regulation.Source: §6 · p. 8

A Supplementary Videos and Demos

• The supplementary materials provide full-length rollout videos and demonstrations of diverse tasks on the project website.The paper recommends these videos for a more comprehensive view of system capabilities.Source: A Supplementary Videos and Demos · p. 13

B Analysis of Sim-to-Real Transfer

• Simulation and real-world rollouts exhibit similar fingertip contact timing, force magnitudes, and force redistribution during cube grasping.Real-world force measurements are noisier, but the overall force-profile structure remains largely preserved.Source: B Analysis of Sim-to-Real Transfer · p. 13, Figure 6 · p. 13
• 2.83 cm mean sim-to-real tracking error, with real-world trajectory standard deviations of 0.46–0.82 cm across episodes.The discrepancy grows from 1.1 cm during approach to 2.7 cm during grasp and 3.8 cm during lift; object-placement variation and camera-calibration error also contribute.Source: B Analysis of Sim-to-Real Transfer · pp. 13–14, Figure 7 · p. 14
• The residual wrist-tracking gap is attributed mainly to arm compliance, object-placement variation, and camera-calibration error, rather than large differences in learned motion.The real robot lifts approximately 5.7 mm higher along the z-axis; improved arm-dynamics and camera-extrinsic calibration could reduce the gap.Source: B Analysis of Sim-to-Real Transfer · p. 14

C Hand-Object Reconstruction Details

• Hand-object reconstruction estimates object and hand poses from monocular video, then jointly refines trajectories to address interpenetration and temporal jitter.The joint objective combines penetration, registration, and smoothness terms, activating penetration loss after optimizing registration and smoothness.Source: C Hand-Object Reconstruction Details · pp. 14–15

D Retargeting Details

• Retargeting maps reconstructed wrist, hand, and object targets into the simulation workspace and optimizes robot trajectories subject to physical-robot kinematic limits.Variants with mean end-effector position tracking error above 8 cm are excluded from policy training.Source: D Retargeting Details · pp. 15, 17
• Spatial augmentation applies shared planar translations and yaw rotations to wrist, hand, and object trajectories, preserving their relative interaction geometry.The sampled yaw perturbation is ∆ψ ∼U(−10°, 10°).Source: D Retargeting Details · pp. 15–16, Figure 8 · p. 17
• Stage 1 optimizes arm joints 2–7 to track the reconstructed wrist with the hand held nominal and base yaw fixed; Stage 2 jointly optimizes all 22 hand joints and arm joints 1–7.Stage 2 starts from the Stage-1 solution and releases base yaw for additional reach.Source: D Retargeting Details · p. 16
• The Stage-2 objective weights hand-body keypoint tracking by finger and keypoint level, emphasizing thumb, index finger, and distal keypoints.Arm and hand joint-velocity terms regularize frame-to-frame motion; hand limits are tightened to the empirically measured reachable range.Source: D Retargeting Details · pp. 16–17

E Training Algorithms

• The state expert uses PPO with an asymmetric actor-critic: its actor receives 557 observation dimensions, while the critic adds 148 privileged-state dimensions.The 20 tactile-related actor channels include five fingertip-force magnitudes; the 15 contact-position channels are disabled and zeroed.Source: E Training Algorithms · pp. 17–18, F Tactile Observation and Noise Model · p. 19
• DAgger distillation trains a student on a reduced teacher observation plus a visual-tactile point cloud, producing a 481-dimensional input and the same 29-dimensional action.The student removes BPS, fingertip-distance, and noisy object-pose blocks, while retaining task-conditioning motion references; the teacher-student mixture coefficient decays by 0.85 per iteration.Source: E Training Algorithms · p. 18

F Tactile Observation and Noise Model

• The expert’s effective tactile feedback is five fingertip-force magnitudes, trained with multiplicative Gaussian noise, per-finger dropout, and stochastic hold-last latency.The 15 contact-position channels are disabled in the default configuration; contact-position noise applies only when those observations are enabled.Source: F Tactile Observation and Noise Model · p. 19

G Reward Details

• The state-expert reward combines wrist, hand, and object tracking with contact shaping, action regularization, and terminal success rewards.Tracking rewards are bounded exponential functions of position, orientation, or velocity errors; arm collisions receive an additional penalty, while joint limits are enforced by the controller.Source: G Reward Details · pp. 19–20, Table 7 · p. 20

H Domain Randomization

• Object-pose perturbations emulate real-world pose-estimation errors through frame-wise noise, episode-wise bias, latency, and dropout.The state expert uses a 557-dimensional actor observation and an additional 148-dimensional privileged critic state.Source: H Domain Randomization · p. 20, I Observation Space · p. 20

I Observation Space

• The student removes BPS, fingertip-distance, and noisy current-object-pose blocks from the teacher vector observation, retaining the target object pose in its motion reference.It additionally receives scene points, hand keypoints, and fingertip-surface tactile points encoded by a shared PointNet into a 64-dimensional feature.Source: I Observation Space · p. 21

Arm-dynamics calibration

• Joint 1–3 simulated damping was reduced by approximately 40% until cross-correlation exceeded 0.996 and estimated lag fell below 10 ms.Calibration replayed 0.2 Hz sinusoidal trajectories with 0.3 rad amplitude on each arm joint independently, comparing commanded and achieved positions on the real FR3 and in simulation.Source: J Sim-to-Real Alignment Details · p. 21

Deploy safety

• Safety limits constrain joint positions, joint velocities, and arm- and hand-action deltas; episode resets ramp through current pose, home pose, then demo start.The two-phase reset ramp is intended to avoid sudden motions at episode boundaries.Source: J Sim-to-Real Alignment Details · p. 21

Control-loop timing

• The 30 Hz control loop allocates approximately 33 ms per step, including depth capture, kinematics, policy inference, ROS 2 publishing, and SDK communication.The listed operations take approximately 5, 2, 3, 1, and 1 ms, respectively, with remaining time for scheduling jitter.Source: J Sim-to-Real Alignment Details · p. 21

State-expert actor observations

• The state-expert actor observation layout contains 557 dimensions.The passage identifies this as the state-expert actor observation layout.Source: Table 9 · p. 22

Privileged critic state

• The privileged critic-state layout contains 148 dimensions.The passage identifies this as the privileged critic-state layout.Source: Table 10 · p. 22