PaperlayerView exact revision on arXiv

Source-linked AI summary

DexCast: Learning Real-World Dexterous Control from Human Videos

Zihan Gao, Yuhong Zhang, Mengzhe Wang, Kaisheng Liu, Shengpeng Li, Siyao Xiao, Runqing Cheng, Xiao Lin, Zhifang Liu, Haoqian Wang

arXiv:2610.14691v1cs.ROcs.AI

TL;DR

Teleoperated demonstrations are costly and tied to particular robot embodiments and scenes, motivating human videos as an alternative source of dexterous-manipulation supervision. DEXCAST transfers skills from monocular videos through reconstructed HOI and a motion-aware hand representation, and outperforms baselines across six real-world tasks. Its HOI reconstruction is limited to rigid objects and can fail under rapid motion or severe occlusion.

  • Problem

    Dexterous imitation learning often depends on costly, time-consuming teleoperated demonstrations tied to specific embodiments, viewpoints, and scenes.

    Source: §I · p. 1

  • Method

    DEXCAST transfers skills from human videos to real-world control without robot demonstrations, using HOI reconstruction and DMPM’s compact hand-motion representation.

    Source: §I · p. 2, 1) DMPM Architecture · p. 3

  • Results

    DEXCAST outperforms all baselines across six real-world tasks, with average success rates of 87.5% and 86.7% under two viewpoints.

    Source: B. Real-world Evaluation · p. 5

  • Takeaways & Limitations

    DEXCAST provides a video-derived route to real-world dexterous policy learning, replacing labor-intensive demonstration collection with HOI-based training data.

    Source: §V · p. 7

    HOI reconstruction is limited to rigid objects and susceptible to tracking failures under rapid motion or severe occlusion; DMPM is embodiment-specific and requires separate skeleton design and training for each hand.

    Source: §V · p. 7

Abstract

from arXiv · show

Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.

I. INTRODUCTION

DEXCAST addresses the cost and embodiment-specificity of teleoperated demonstrations by transferring dexterous skills from human videos to real-world robot control. It combines topology-aware hand-motion representations with real-world evaluation across six tasks.

I. INTRODUCTION

  • Teleoperated robot demonstrations are costly, time-consuming, and often tied to particular embodiments, viewpoints, and scenes; high-dimensional actions and complex contacts make dexterous data collection especially difficult.Limited haptic feedback and sampling rates can also disrupt fluid human motion patterns in teleoperated demonstrations.Source: §I · p. 1
  • DEXCAST transfers dexterous skills from human videos without on-robot demonstrations, while DMPM encodes skeletal structure to model inter-joint coupling and coordinated motion.The paper reports pretraining DMPM on over 5M frames of HOI data.Source: §I · p. 2
  • DEXCAST averages up to 87.5% success and supports zero-shot cross-scene deployment across variations in object pose, camera viewpoint, and robot initial state.The framework was evaluated on six real-world dexterous manipulation tasks using a five-fingered robotic hand, without real-robot demonstrations.Source: §I · p. 2

A. Dexterous Hand Motion Representation

  • Prior motion-representation approaches motivate transferable dexterous control, but do not explicitly model dependencies among finger joints; DEXCAST incorporates the robotic hand’s skeletal structure.The cited approaches include latent subtrajectory representations [20], quantized action chunks [21], shared latent actions across embodiments [22], and alternative hand-motion tokenization.Source: A. Dexterous Hand Motion Representation · p. 2

B. Geometry-Aware Visuomotor Policies

The section situates DEXCAST among geometry-aware visuomotor policies and video-based dexterous learning, then outlines its approach: convert HOI videos into expert demonstrations and condition a visuomotor policy with DMPM.

B. Geometry-Aware Visuomotor Policies

  • Recent policies condition action generation on 3D geometry, interaction structure, temporal scene models, or geometric equivariance.Examples include sparse point-cloud features in DP3 [23], scene features and language in 3D Diffuser Actor [24], and SIM(3) equivariance in EquiBot [26].Source: B. Geometry-Aware Visuomotor Policies · p. 2

C. Learning from Human Videos

  • Video-based methods recover hand trajectories or structured HOI, but prior approaches may omit object states, require robot demonstrations, remain simulation-only, or lack closed-loop real-world observation conditioning.The passage contrasts estimated hand trajectories,,, with structured HOI trajectories,,.Source: C. Learning from Human Videos · p. 2

A. Overview

  • DEXCAST aims to learn real-world dexterous control from RGB-D HOI videos without real-robot demonstrations, converting videos into structured expert demonstrations for policy training.DMPM’s frozen encoder supplies temporal-context and topology-aware motion representations to the visuomotor policy.Source: A. Overview · pp. 2–3

B. Demonstration Generation

  • The demonstration-generation process derives expert demonstrations from HOI videos through human-to-robot transfer and data augmentation.The section introduces these two stages as the route from video data to imitation-learning supervision.Source: B. Demonstration Generation · p. 3

1) Human-to-Robot Demonstration Generation:

DEXCAST turns reconstructed human hand-object interaction into robot demonstrations, augments their spatial coverage, and uses DMPM to represent coordinated hand motion for policy conditioning.

1) Human-to-Robot Demonstration Generation:

  • Monocular HOI videos are converted into temporally consistent hand-object trajectories and refined with a residual RL policy to obtain physically plausible poses.The resulting trajectories are transformed into expert demonstrations; these include hand and object point clouds, proprioceptive state, and imitation targets.Source: 1) Human-to-Robot Demonstration… · p. 3

2) Data Augmentation:

  • Object poses are perturbed within a 20 cm×20 cm workspace region and a 60° angular range to broaden spatial coverage while preserving interaction geometry.The augmentation introduces controlled variations in object placement, robot initialization, and workspace configuration.Source: 2) Data Augmentation · p. 3

1) DMPM Architecture:

  • DMPM uses a VAE to encode robotic hand-motion sequences into a compact latent representation and decode them back into motion.Its encoder represents the hand as a directed skeletal graph, restricting feature interactions to local neighborhoods to model inter-joint coupling and coordinated motion.Source: 1) DMPM Architecture · p. 3
  • The non-autoregressive Neural Motion Field decoder represents motion as a continuous function of time conditioned on a latent code, allowing the full sequence to be sampled in parallel.Sampling is performed by querying the decoder at different timestamps.Source: 1) DMPM Architecture · p. 3
  • DMPM’s training objective combines weighted pose and velocity reconstruction losses with a KL penalty that regularizes the latent distribution toward a standard Gaussian prior.The velocity weight controls the trade-off between pose precision and dynamical consistency; joint metrics balance the importance of different joints.Source: 1) DMPM Architecture · p. 3

2) Training Data:

DMPM is pretrained on a large, diverse corpus of retargeted hand motions assembled from public HOI datasets.

2) Training Data:

  • The DMPM pretraining corpus contains over 5M frames from seven public HOI datasets, with MANO poses retargeted to the robotic hand by matching wrist-to-fingertip vectors.The datasets are GRAB, TCDHands, DexYCB, H2O, ARCTIC, HOT3D, and HoCap [39].Source: 2) Training Data · p. 3

D. 3D Diffusion Policy with DMPM

DEXCAST conditions a diffusion policy on a compact, topology-aware representation of robotic-hand motion, alongside geometric and proprioceptive inputs. The policy iteratively denoises actions using temporally aligned observations and predicts action chunks.

D. 3D Diffusion Policy with DMPM

  • A pretrained DMPM encoder represents hand-motion context, while separate point-cloud branches and an MLP provide geometric and proprioceptive conditions to the diffusion policy.The 32-frame motion sequence contains joint positions and velocities; its latent embedding captures inter-joint dependencies and temporal context.Source: D. 3D Diffusion Policy with DMPM · p. 3
  • The geometric condition fuses robotic-hand and object point-cloud features, and the three conditions enter the diffusion network through FiLM layers.The point-cloud branches use SIM(3)-equivariant Point-Net backbones.Source: D. 3D Diffusion Policy with DMPM · p. 3
  • Starting from Gaussian noise, the denoising network iteratively refines a noisy action over K steps to recover the clean action.The update uses step-dependent coefficients and Gaussian noise.Source: D. 3D Diffusion Policy with DMPM · p. 3
  • The policy uses 32 observation-history frames to predict the next 32 action steps, for a total horizon of 64 steps.The action chunk size is fixed at 64 because expert demonstrations are derived from high-frame-rate human video.Source: D. 3D Diffusion Policy with DMPM · p. 4

A. Experimental Setup

The experimental platform pairs a UR5 arm and five-fingered XHAND with a side-mounted RGB-D camera for object tracking.

A. Experimental Setup

  • The system uses a 6-DoF UR5, a 12-DoF five-fingered XHAND, and a side-mounted ORBBEC GEMINI 335 RGB-D camera; camera extrinsics are calibrated to the robot base frame.Source: A. Experimental Setup · p. 5

2) Task Design:

The evaluation covers six manipulation skills, with demonstrations generated from one human video per task and augmented for training.

2) Task Design:

  • The six tasks cover pushing a tabletop button, grasping a mango, wiping a whiteboard, flipping a cup upright, pouring coffee, and stacking cups.Source: 2) Task Design · p. 5, 1) Hardware Setup · p. 5
  • Each task uses 40 training demonstrations, including original reinforcement-learning rollouts and augmented data; trajectory horizons range from 150 to 300 steps.All expert demonstrations for a task come from the same human video recorded at 30 fps.Source: 2) Task Design · p. 5

1) Hardware Setup:

3) Baselines:

The experiments compare DEXCAST with 3D-based and state-based policies, deploy it using buffered observations, and evaluate real-world performance across tasks and distribution shifts.

3) Baselines:

  • The baselines include DP-3D, ACT-3D, CordViP, and a state-based policy, with point-cloud observations and training-cloud counts matched across methods.Models are fine-tuned for 10k iterations per task with a 64-step horizon, 32 observation steps, and 32 action steps.Source: 3) Baselines · p. 5

4) Deployment Details:

  • At deployment, object poses are estimated from RGB-D observations, and buffered joint states supply temporally aligned point clouds and proprioception for repeated 32-step predictions.The robot executes each predicted chunk at 8 Hz without considering velocity.Source: 4) Deployment Details · p. 5

B. Real-world Evaluation

  • DEXCAST achieves average success rates of 87.5% and 86.7% across six real-world tasks under two viewpoints and outperforms all baselines.The evaluation randomizes object pose and arm initial state; Table I reports success rates for each viewpoint and the averages.Source: B. Real-world Evaluation · p. 5, Table I · p. 6
  • On Pour and Stack, DEXCAST reports stronger adaptability to unseen object shapes than the baselines, while interaction-centric geometry is associated with reduced background dependence and stronger spatial perception.Source: B. Real-world Evaluation · p. 6

C. Ablation Study

The ablations examine the contributions of DexCast components and DMPM design choices. They report complementary benefits from the components, richer DMPM hand-state representations, and reconstruction trade-offs across skeleton structure, clip length, and latent size.

1) Ablation of DexCast:

  • Removing augmentation reduces success rate by 61.7%, while the SIM(3)-equivariant architecture and DMPM are individually effective and complementary.The authors also report that DMPM’s gains largely stem from large-scale pretraining.Source: 1) Ablation of DexCast · p. 6, Table IV · p. 7

2) Ablation of the DexHand Motion Perception Module:

V. CONCLUSION

DEXCAST uses 4D HOI representations and RL refinement to produce policy-learning data from human videos, and conditions dexterous manipulation with a compact motion-aware DMPM representation. Its current reconstruction and DMPM designs have stated scope limits.

V. CONCLUSION

  • DEXCAST replaces labor-intensive demonstration collection with policy-learning data produced from 4D HOI representations and RL refinement of human-video trajectories.The framework uses a 3D diffusion policy enhanced with DMPM for real-world dexterous manipulation.Source: §V · p. 7
  • DMPM provides a compact, motion-aware conditioning representation for dexterous manipulation.Source: §V · p. 7
  • Current 4D HOI reconstruction is limited to rigid objects and can be affected by rapid motion or severe occlusion; DMPM requires separate skeleton design and training for each robotic hand.Source: §V · p. 7

Related papers

Loading 2610.14691v1…