Source-linked AI summary
VideoDex: Learning Dexterity from Internet Videos
Kenneth Shaw, Shikhar Bahl, Deepak Pathak
TL;DR
Collecting real-world robot experience is difficult because safety, time, and hardware constraints limit deployment. VideoDex addresses this by combining visual, action, and physical priors learned from internet human videos with a few in-domain demonstrations, and it outperforms many state-of-the-art methods across seven real-world manipulation tasks.
Problem
Safely collecting real-world interaction data for general robot learning is often infeasible because of safety, time, and hardware restrictions.
Method
VideoDex learns a single open-loop policy combining visual, action, and physical priors from passive internet human videos, then adapts it using a few in-domain examples.
Results
VideoDex outperforms many state-of-the-art robot learning methods on seven real-world manipulation tasks across high-DOF multi-fingered and 1-DOF gripper robot systems.
Takeaways & Limitations
Human videos can provide visual, action, and physical priors that support robot manipulation with limited in-domain robot experience.
Takeaways & Limitations
VideoDex relies on curated human video datasets and off-the-shelf hand detection modules that often produce erroneous 6D pose detections during object interaction.
Abstract
from arXiv · showhide
To build general robotic agents that can operate in many environments, it is often imperative for the robot to collect experience in the real world. However, this is often not feasible due to safety, time, and hardware restrictions. We thus propose leveraging the next best thing as real-world experience: internet videos of humans using their hands. Visual priors, such as visual features, are often learned from videos, but we believe that more information from videos can be utilized as a stronger prior. We build a learning algorithm, VideoDex, that leverages visual, action, and physical priors from human video datasets to guide robot behavior. These actions and physical priors in the neural network dictate the typical human behavior for a particular robot task. We test our approach on a robot arm and dexterous hand-based system and show strong results on various manipulation tasks, outperforming various state-of-the-art methods. Videos at https://video-dex.github.io
1 Introduction
VideoDex addresses the difficulty of safely collecting robot experience by using internet human videos beyond visual pretraining. It combines visual, action, and physical priors, then adapts with a few in-domain demonstrations to perform manipulation tasks.
- Robot learning faces a chicken-and-egg problem because safely collecting real-world interaction data requires inexperienced robots to be supervised.
- Prior work commonly uses human videos for visual representation pretraining, while VideoDex also extracts information about how humans move their limbs.
- VideoDex combines visual and action priors from human videos with a physical prior describing how the robot should move in the world.
- The method mixes experience from massive internet data with a few in-domain demonstrations to adapt its policy to real-world tasks.
- VideoDex outperforms many state-of-the-art robot learning methods across seven real-world manipulation tasks on both a high-DOF multi-fingered arm-hand system and a 1-DOF gripper arm system.
2 Related Work
Related work spans dexterous reinforcement learning, human-video datasets and pose estimation, and methods that transfer human observations into robot actions or representations.
- Learning for Dexterity: Dexterous reinforcement learning can achieve simulation results but requires substantial data, especially for high-DOF manipulation.
- Learning from Videos and Large-Scale Datasets: Curated internet-video datasets provide hand poses, hand-object interactions, semantically similar interactions, human poses, and action-driven manipulation data.
- Learning Action from Videos: Human-action learning research includes detecting people, estimating body-part poses, and modeling motion dynamics and interactions.
- Robot Learning by Watching Humans: Robot-learning methods use human datasets to learn cost functions, action correspondences, explicit actions, future hand or object locations, and representations.
- The experimentation uses separate collections of train objects and test objects.
3 Background
VideoDex uses Neural Dynamic Policies to generate smooth, safe open-loop robot trajectories and retargets human hand motion to the robot embodiment. These components convert human-video observations into robot-compatible action priors.
- Neural Dynamic Policies: Neural Dynamic Policies produce smooth and safe open-loop trajectories that can be rolled out for arbitrary lengths, accommodating varying-length human videos.
- Neural Dynamic Policies: The NDP equation combines a goal-directed dynamical system with a radial-basis forcing function over time.Here y is the robot coordinate frame, g the desired goal, fw the forcing function, x time, and α, β global constants.
- Learning from Watching Humans: VideoDex uses a single RGB camera teleoperation approach to collect demonstrations by retargeting human hand and body motion to a robot hand and arm.
- Learning from Watching Humans: Human hand retargeting manually defines corresponding palm-to-fingertip vectors and minimizes distances between human and robot hand poses through an energy function.
4 Learning Dexterity from Human Videos
VideoDex turns human internet videos into robot experience by retargeting human hand and wrist motion to robot embodiments, then pretraining policies with visual, action, and physical priors before real-robot adaptation.
- Action Priors from Human Activity Data: VideoDex uses human video information beyond visual features by pretraining an action prior that encodes typical task-specific robot actions.The action prior is a network initialization learned from human hand trajectories retargeted to the robot embodiment.
- Retargeting Human Motion: The method addresses the human–robot embodiment gap by detecting hands, mapping human hand poses to robot finger joints, and converting human wrist poses to robot arm poses.The resulting hand and wrist trajectories define human motion in the robot frame for action-prior pretraining.
- Retargeting Human Motion: Wrist retargeting compensates for moving egocentric cameras by estimating camera motion between the first frame and each subsequent frame with monocular SLAM.Perspective-n-point uses detected 2D hand keypoints and 3D hand-model keypoints to compute wrist transformations in camera coordinates.
- Learning with Human Videos: The policy combines an R3M-initialized visual encoder with physically inspired Neural Dynamic Policies that generate open-loop wrist and hand trajectories.The encoder conditions separate wrist and hand NDPs, whose integrators output trajectory shape and goal-conditioned trajectories.
- Learning with Human Videos: Human clips are retargeted and used to pretrain task-specific policies, which are then trained on a few teleoperated real-robot demonstrations.The reported human-video training uses 500–3000 clips per task category, with pick including close to 3000 clips.
5 Experimental Setup
Experiments evaluate VideoDex on seven real-world manipulation tasks using both a high-DoF multi-fingered hand-arm system and a two-finger gripper setup, with action priors pretrained from human videos.
- Evaluation Questions: The study compares VideoDex with methods lacking human-data action priors and examines the contributions of action, physical, visual, and training-design choices.The experiments specifically ask whether VideoDex supports general-purpose open-loop manipulation and how much its action and physical priors help.
- Task Setup: The experiments test pick, rotate, cover, uncover, push, place, and open manipulation tasks.Open uses three different drawers, while cover and uncover involve a soft cloth and a pan or plate.
- Task Setup: Each task uses about 120–175 real-robot demonstrations after action-prior pretraining on retargeted Epic Kitchens data.The demonstrations train the policy on the researchers’ physical setup.
- Robot Embodiments: VideoDex is evaluated on both a 16-DoF LEAP Hand with an arm and a two-finger gripper with an xArm.Separate action priors are trained for the two robot embodiments, with human open and closed hands mapped to gripper states.
6 Results
VideoDex is evaluated against action-prior-free baselines, alternative visual priors, physical-prior variants, grippers, and hand hardware. Across these experiments, action priors, physical priors, and the two-stream design generally improve generalization, sample efficiency, and manipulation performance.
- Effect of Action Priors: VideoDex generally matches or outperforms methods without human-data action priors, especially on held-out objects and instances.Action-prior initialization also produces trajectories closer to ground-truth robot trajectories than visual-prior-only initialization.
- Hand vs 2-Finger Gripper: In the 1-DOF gripper setting, VideoDex improves performance because human-video priors encode wrist trajectories and gripper-closing timing.
- Initial Pose Computation Comparison: VideoDex-Surface outperforms VideoDex-IMU, while fixed or randomized initial-pose variants remain comparable to or better than baselines without human action data.The authors suggest noisy sensor data may explain the lower IMU performance.
- Effect of Physical Priors and Architectural Choices: Physical priors generally improve over baselines without them, while separating wrist and hand actions improves learning relative to a single-stream policy.The two-stream design supports reusable grasps across locations and distinct interactions after object localization.
- Generalization with Less Data: With only 5 demonstrations per variant, VideoDex achieves a 30% success rate on unseen objects and retains an advantage with less robot experience.Policies generally reach the correct area but often fail to grasp objects properly.
- Effect of Visual Priors: Action priors are more impactful than visual priors, although visual priors remain useful and MVP outperforms VGG among the compared alternatives.
- Choice of Robotic Hand: LEAP Hand outperforms Allegro Hand by 7−12% on average across experiments, alongside fewer control inaccuracies and hardware failures.
7 Discussion and Limitations
VideoDex reports strong held-out-object results but remains bounded by curated data, noisy hand-pose detection, embodiment-specific retargeting, and open-loop behavior cloning.
- VideoDex focuses on curated human video datasets, though task-filtered internet videos could potentially broaden the data source.The authors use curated datasets for convenience and suggest action detectors as a future route to filtering internet videos by task.
- Off-the-shelf hand detectors often produce erroneous 6D pose estimates when hands interact with objects.
- Action-prior retargeting must be recomputed for each robot-parameter set and embodiment.
- The real-world behavior-cloning policy is open-loop and cannot react to environmental changes.The authors connect this design to the difficulty of keeping closed-loop behavior cloning and reinforcement learning safe in the real world.
B Additional Ablations
Ablations show that cross-task transfer and trajectory noise reduce performance, while replacing the visual prior also degrades results on test objects.
- Cross-task pretraining causes slight performance degradation on both uncover and place, especially on place.Policies pretrained on one task were fine-tuned on the other using robot data.
- Adding more Gaussian noise to demonstration trajectories hurts performance.The evaluated noise levels have standard deviations of 0.01 and 0.05.
- Replacing R3M features with ImageNet-initialized ResNet18 features causes performance to drop on test objects.This ablation indicates that the visual priors contribute to the method's performance.
C Retargeting Details
The retargeting pipeline converts passive human videos into robot-frame trajectories by estimating camera and hand geometry, reorienting and rescaling motion, and distilling hand retargeting for learning.
- Human video processing: Video clips from Epic-Kitchens depict humans completing tasks similar to the robot tasks and average 5–10 seconds.
- Camera and wrist estimation: COLMAP calibrates camera intrinsics, while FrankMocap and PnP estimate the wrist pose from human-hand observations.The pipeline uses 3D hand-model points and corresponding 2D image points for pose estimation.
- Camera motion: SLAM tracks camera motion through clips to compensate for moving cameras, using ORBSLAM3 for faster-than-real-time processing.The process starts two seconds before the action clip and recovers the camera trajectory relative to the first frame.
- Robot-frame transformation: The camera frame is reoriented using gravity-related information so trajectories become upright in the robot frame.Pitch and roll can be corrected from a ground-normal or accelerometer-derived vector, but yaw is not determined this way.
- Hand retargeting: Hand retargeting minimizes differences between human and robot keypoints, then distills the energy-minimizing process into a neural network trained on internet videos.Robot keypoints depend on the robot joint pose q, and the resulting retargeted trajectories provide actions for policy learning.
Task
VideoDex evaluates seven manipulation tasks spanning object handling, drawer opening, cloth placement, and finger-based pushing.
- The evaluation includes pick, rotate, open, cover, uncover, place, and push.
- Pick removes objects from a table or plate/pan, while rotate turns an object in place.
- Open requires opening a drawer, and cover or uncover places or removes cloth from a plate.
- Place transfers an object into a plate or pot/pan, whereas push pokes the object with the robot's fingers.
D Learning Pipeline Details
The learning pipeline uses pretrained visual features, pose information, multilayer perceptrons, and an NDP to produce trajectory predictions. The section also includes a parameter list.
- R3M ResNet18 features produce a 512-dimensional visual vector processed by a two-layer MLP with hidden dimension 512.
- Visual features are concatenated with the starting hand and wrist pose before separate MLP processing for hand and wrist trajectories.
- An NDP with a single hidden layer projects the processed input into the desired output size.
- The section includes a parameter list for the pipeline.
E Experimental Setup
Experiments use teleoperated dexterous-hand demonstrations collected with randomized tabletop manipulation setups. The hardware includes an LEAP hand, an XArm manipulator, cameras, and multiple external codebases.
- The teleoperation setup collects about 120-175 demonstrations in a uniform style from a trained operator.A manager places and randomizes objects, controls demonstrations, and manages the robot system.
- The main hardware combines an LEAP 16 DOF Hand mounted on an XArm 6 manipulator.The setup also uses Intel Realsense D415 cameras and four NVIDIA RTX 2080TIs for data collection, training, and teleoperation.
- The experiments additionally use the Allegro Hand to collect teleoperation data.
- The pipeline uses external codebases for human detection, NDP and behavior cloning, R3M, CQL, COLMAP, ORBSLAM3, and GoPro metadata extraction.