Source-linked AI summary
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, Xiaolong Wang
TL;DR
DexMV addresses the challenge of teaching high-DoF robot hands complex dexterous manipulation from scalable human demonstrations. It pairs video capture with simulation, translates estimated 3D human hand-object motion into robot demonstrations, and reports that demonstrations substantially improve learning and enable tasks RL alone cannot solve.
Problem
Complex dexterous manipulation remains difficult for robots because high-DoF, nonlinear multi-finger hands require substantial RL data and may learn unnatural behavior.
Method
DexMV pairs human manipulation videos with aligned simulation tasks, estimates 3D hand-object poses, translates human motion into robot demonstrations, and benchmarks imitation-learning algorithms.
Results
Human demonstrations improve performance by a large margin and solve complex manipulation tasks that reinforcement learning alone cannot solve.
Takeaways & Limitations
Human videos provide useful demonstrations for learning dexterous manipulation policies across complex tasks and diverse object instances.
Abstract
from arXiv · showhide
While significant progress has been made on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipulation. In this paper, we propose a new platform and pipeline DexMV (Dexterous Manipulation from Videos) for imitation learning. We design a platform with: (i) a simulation system for complex dexterous manipulation tasks with a multi-finger robot hand and (ii) a computer vision system to record large-scale demonstrations of a human hand conducting the same tasks. In our novel pipeline, we extract 3D hand and object poses from videos, and propose a novel demonstration translation method to convert human motion to robot demonstrations. We then apply and benchmark multiple imitation learning algorithms with the demonstrations. We show that the demonstrations can indeed improve robot learning by a large margin and solve the complex tasks which reinforcement learning alone cannot solve. More details can be found in the project page: https://yzqin.github.io/dexmv
1 Introduction
DexMV addresses the difficulty of dexterous robot manipulation by using scalable human-video demonstrations, a paired simulation platform, and a translation pipeline for imitation learning. It targets complex, diverse tasks and reports improved performance and generalization to unseen object instances.
- Motivation: RL-only training for high-DoF, tendon-actuated hands requires substantial data and can produce unnatural behavior.The paper motivates using human experience to guide dexterous robot learning.
- Motivation: VR-based demonstrations are costly, difficult to scale, and limited to 25 demonstrations per task.Prior tasks were also sufficiently simple that RL often achieved similar performance with or without demonstrations.
- Platform: DexMV records human videos while providing aligned simulation tasks for a multi-finger Adroit robotic hand.The paired systems are intended to bridge 3D vision and robotic dexterous manipulation.
- Pipeline: Its three-stage pipeline estimates 3D hand-object poses, translates human trajectories into robot demonstrations, and applies imitation learning.Translation includes hand-motion retargeting and robot-action estimation, followed by benchmarking state-only and state-action demonstration methods.
- Results: The experiments cover relocate, pour, and place-inside tasks with diverse objects, and report large performance gains plus generalization to unseen instances.The paper presents these outcomes as evidence that human demonstrations benefit complex dexterous manipulation.
2 Related Work
Prior work studies dexterous manipulation through planning, reinforcement learning, imitation, trajectory following, and hand-object estimation. DexMV differs by translating hand-object video sequences into robot demonstrations for policy learning across goals and object configurations.
- Dexterous Manipulation: Dexterous manipulation with multi-finger hands is challenging, and RL-only training requires huge data samples for the 30-DoF Adroit hand.The related-work discussion contrasts this setting with regular 2-DoF grippers.
- Imitation Learning from Human Demonstrations: Imitation learning includes behavior cloning, inverse reinforcement learning, and RL augmented with demonstrations.Earlier approaches generally rely on expert policies or VR-collected demonstrations.
- Following Human Demonstrations: Trajectory-following methods reproduce expert motions, whereas DexMV emphasizes policies that generalize across goals and object configurations.This distinguishes policy learning from repeating a single expert trajectory.
- Hand-Object Interaction: Hand-object interaction research estimates poses, reasons about contact, and retargets hand poses for robot control.The cited teleoperation work generally does not involve objects, while DexMV converts hand-object pose sequences into robot demonstrations.
3 Overview
DexMV combines paired human-video and simulation systems with a pipeline that converts estimated human motion into robot demonstrations. These demonstrations are then used for imitation learning and reinforcement-learning-based policy training.
- Platform: DexMV provides paired computer-vision and simulation systems for the same dexterous manipulation tasks.The platform records human demonstrations and supplies corresponding robot-hand environments.
- Data Collection: Human demonstrations can be collected at approximately 100 demonstrations per hour.The platform uses real-world human demonstrations for three challenging simulator tasks.
- Demonstration Translation: 3D pose estimation and motion retargeting translate recorded human videos into robot-hand demonstrations.The translated demonstrations support policy learning rather than merely reproducing one trajectory.
- Policy Learning: The pipeline uses translated demonstrations to investigate multiple imitation-learning and reinforcement-learning algorithms.The overview describes the demonstrations as inputs to policy learning in the aligned simulation tasks.
4 DexMV Platform
The DexMV platform pairs a video-based human demonstration system with MuJoCo simulation for a 30-DoF Adroit hand. It evaluates three object-manipulation tasks across goals, object configurations, and unseen instances.
- Computer Vision System: The computer-vision system records human manipulation of diverse real objects using two RGB-D cameras mounted around a cubic frame.Videos capture front and side views during tasks such as relocating a sugar box.
- Simulation System: The simulation system uses MuJoCo and the Adroit Hand to provide manipulation environments aligned with the human demonstrations.Policies are trained with demonstrations augmenting reinforcement learning and tested under different goals and object configurations.
- Task Setup: The task state includes robot joint readings and object pose, with relocate additionally including the target position.The action is a 30-dimensional position-actuator command, one control command for each joint.
- Relocate: Relocate requires moving an object to a goal, including five complex objects and goal-conditioned target positions.Success depends on reaching the goal, without requiring a specific object orientation.
- Pour and Place Inside: Pour evaluates the percentage of particles transferred from a mug into a container, while place inside evaluates the percentage of object volume inside a container.Place inside also requires suitable object rotation and careful collision avoidance.
- Generalization: Generalization is evaluated on unseen object instances both within trained categories and outside them, including a ShapeNet camera.The tests extend beyond the objects used during training.
5 Pose Estimation
DexMV estimates 3D object and hand poses from video by combining RGB, point clouds, depth, and hand models. The method formulates hand-joint estimation as an optimization that aligns projected joints with image evidence and rendered depth.
- Object Pose Estimation: PVN3D detects objects and estimates their 6-DoF poses from RGB images and point clouds, using segmentation and dense keypoint voting.The pose includes translation T ∈R3 and rotation R ∈SO(3).
- Hand Pose Estimation: MANO represents each hand with 15 joint rotations, a global root pose, and shape parameters, from which 3D joints are computed kinematically.
- Hand Pose Estimation: Hand masks, 2D joints, and MANO parameters are estimated per frame, while the root pose is obtained from the depth image masked by the hand.
- Hand Pose Estimation: The optimization minimizes 2D reprojection error and masked depth-rendering error to refine the hand pose and root position.The objective optimizes θ_t, β_t, and r_t using the camera projection, hand kinematics, hand mask, and depth map.
- Hand Pose Estimation: The formulation extends to multiple calibrated cameras by minimizing objectives across views, and post-processing enforces temporal consistency between frames.
6 Demonstration Translation
DexMV translates human video poses into robot demonstrations by retargeting hand motion across different kinematics and recovering executable robot actions. The pipeline emphasizes task-space alignment, temporal smoothness, and physically feasible trajectories.
- Demonstration Translation: Human and robot hands have different degrees of freedom and geometry, so demonstration translation aligns hand motion before estimating robot motor torques.The two translation steps are hand motion retargeting and robot-action prediction.
- Hand Motion Retargeting: Retargeting computes a sequence of robot joint angles from a sequence of estimated human hand poses by solving an optimization problem at each time step.
- Hand Motion Retargeting: Compared with fingertip-only mapping, the proposed optimization objective is designed to avoid unexpected retargeting results while preserving task-space relationships.
- Hand Motion Retargeting: The proposed objective matches ten task-space vectors between human and robot hands while regularizing successive robot joint configurations.The vectors include fingertip-to-palm and fingertip-to-phalanx relationships; the regularizer promotes temporal consistency.
- Robot Action Estimation: The retargeted joint sequence is fitted with a minimum-jerk trajectory so inverse dynamics can recover torque from position, velocity, and acceleration.The torque is computed as τ(t) = finv(q(t), q′(t), q′′(t)); minimum jerk also supports natural motion and limits motor errors and wear.
- Time Alignment: Because videos run at around 30Hz and simulation at 120Hz, the continuous trajectory is sampled at simulation frequency before actions are computed.
7 Imitation Learning
DexMV incorporates translated demonstrations into reinforcement learning rather than using behavior cloning alone. It evaluates state-action and state-only imitation, including GAIL-based and DAPG-based approaches.
- Learning Setup: DexMV uses imitation-learning algorithms that incorporate demonstrations into reinforcement learning instead of relying on behavior cloning.
- Learning Setup: The learning problem is defined over an MDP with state, action, transition, reward, and discount factor, optimizing a policy with demonstrations and reward.
- Learning Settings: The evaluation covers state-action imitation and state-only imitation, with all methods using both demonstrations and task reward.
- State-Action Imitation: GAIL learns by matching the policy’s state-action occupancy measure to that of demonstrations through adversarial discriminator training.GAIL+ augments GAIL with a reward function using the cited reward-integration approach.
- State-Action Imitation: DAPG combines policy-gradient optimization with a demonstration term weighted by hyperparameters that determine the influence of demonstrated state-action pairs.
- State-Only Imitation: SOIL extends DAPG to state-only demonstrations by learning an inverse model that predicts missing actions while jointly optimizing the policy and inverse model.
8 Experiment
Experiments evaluate imitation learning on Relocate, Pour, and Place Inside, including algorithm comparisons, ablations, and generalization tests. Across tasks, demonstrations outperform pure RL, while DAPG or SOIL is strongest depending on task and setting.
- Relocate: Imitation learning methods outperform the RL baseline on all five Relocate objects, while pure RL fails to learn for mustard bottle and sugar box.SOIL performs best for sugar box; DAPG achieves comparable or better performance on mug, mustard bottle, and tomato soup can.
- Relocate: The proposed retargeting method improves SOIL over fingertip mapping, whereas removing L2 norms causes a large performance drop.The ablation attributes the latter effect to the importance of smooth robot motion in demonstrations.
- Relocate: More demonstrations improve SOIL’s sample efficiency and performance while reducing variance on Relocate with a tomato soup can.The study evaluates policies trained with different demonstration counts at 400, 600, and 800 iterations.
- Relocate: Using the same demonstrations, SOIL maintains consistent imitation-learning results across object scales from ×0.75 to ×1.125 and frictions from ×0.8 to ×1.2.These policies perform much better than pure RL under the varied environmental conditions.
- Pour and Place Inside: DAPG performs best on Pour and Place Inside, achieving 27.2% average particle transfer on Pour while RL hardly learns either task.DAPG benefits from analytically computed actions, whereas SOIL’s inverse-model learning is challenged by water particles and task complexity.
- Generalization: Policies generalize to ShapeNet mugs for Pour and ShapeNet cellphone objects for Place Inside, with performance close to the corresponding trained-object tasks.DAPG remains best across Pour generalization methods, while Place Inside shows robustness to small shape variations and single-object human demonstrations.
9 Conclusion
DexMV presents a platform and pipeline for learning dexterous manipulation from human videos. Its demonstration translation module bridges computer vision and simulation systems and supports benchmarking imitation-learning algorithms.
- Conclusion: DexMV provides paired computer-vision and simulation systems for learning dexterous manipulation from human videos.The platform records human manipulation videos and supplies interactive multi-finger robot environments aligned to the same tasks.
- Conclusion: The demonstration translation module bridges the two systems by converting human-video information into robot-learning demonstrations.The authors position this pipeline as a basis for benchmarking imitation-learning algorithms for dexterous manipulation.
B Video Data Collection
Human demonstrations are collected with Intel RealSense D435 cameras in short sessions designed to keep data collection scalable across objects and tasks.
- Video Data Collection: Each captured human demonstration lasts about 10 seconds, and only Pour requires particle resetting afterward, taking about 6 seconds.The authors report that the resulting procedure has low time cost and tends to scale across objects and tasks.
C Environments
DexMV defines three dexterous manipulation tasks in MuJoCo using YCB objects and the Adroit multi-finger hand model.
- Environments: The platform includes Relocate, Pour, and Place Inside tasks involving YCB objects in MuJoCo simulation.The environments use the open-source MuJoCo model of the Adroit Hand, with timestep 0.002 and frame skip 5.
C.1 Relocate
Relocate requires the robot to move an object from an initial position to a target position. The task specifies a 39-dimensional observation, distance-based rewards, and randomized object and target placement.
- The 39-dimensional observation combines robot-hand joint angles, hand-root position, object position, and target position.
- Reward increases as the hand approaches the object and target, and as the object approaches the target.The reward uses three distances: hand-object, hand-target, and object-target.
- Relocate moves an object from its initial position to a target position, using diverse objects rather than a single ball.
- Each episode randomizes the object and target xy positions within a (−0.3, 0.3) square, while target height is randomized between (0.15, 0.25).