Source-linked AI summary
Phantom: Training Robots Without Robots Using Only Human Videos
Marion Lepert, Jiaying Fang, Jeannette Bohg
TL;DR
Robot learning is constrained by scarce, costly, and insufficiently diverse robot data, while human videos lack action labels and match robot appearance poorly. Phantom converts human videos into robot-compatible demonstrations through hand-pose estimation and visual editing, then deploys the resulting closed-loop policy zero-shot. It achieves high success across diverse tasks and novel scenes without robot data, while remaining constrained by hand-pose estimation, viewpoint assumptions, and strategy and grasp compatibility.
Problem
Robot data collection is slow, costly, and insufficiently diverse, while human videos lack action labels and differ visually from robots.
Method
Phantom estimates hand poses to derive robot actions, edits human images by replacing the arm with a rendered robot, and trains a closed-loop imitation policy.
Results
Phantom achieves high success rates across six tasks, including deformable-object manipulation and novel-environment generalization, without robot data.
Takeaways & Limitations
Human demonstrations alone can be converted into robot policies that support zero-shot deployment across diverse tasks and scenes.
Takeaways & Limitations
Performance depends on hand-pose estimators and requires compatible human and robot strategies, while demonstrations assume similar collection and deployment viewpoints.
Abstract
from arXiv · showhide
Training general-purpose robots requires learning from large and diverse data sources. Current approaches rely heavily on teleoperated demonstrations which are difficult to scale. We present a scalable framework for training manipulation policies directly from human video demonstrations, requiring no robot data. Our method converts human demonstrations into robot-compatible observation-action pairs using hand pose estimation and visual data editing. We inpaint the human arm and overlay a rendered robot to align the visual domains. This enables zero-shot deployment on real hardware without any fine-tuning. We demonstrate strong success rates-up to 92%-on a range of tasks including deformable object manipulation, multi-object sweeping, and insertion. Our approach generalizes to novel environments and supports closed-loop execution. By demonstrating that effective policies can be trained using only human videos, our method broadens the path to scalable robot learning.
1 Introduction
Phantom addresses the scarcity and embodiment gaps that limit robot learning by converting human videos into robot demonstrations without robot data. The framework uses hand pose estimation and data editing to train closed-loop policies for zero-shot deployment.
- Robot datasets remain much smaller than vision-language datasets because hardware-based collection is slow, costly, and difficult to diversify.
- Human videos are abundant and diverse but lack explicit action labels and differ substantially in appearance from robot embodiments.
- Phantom extracts actions from human hand poses, replaces the human arm with a rendered robot, and trains a closed-loop imitation policy using only human demonstrations.
- The method achieves high success rates across six tasks, including deformable-object manipulation and novel-environment generalization, without robot data.
- Data editing adapts cross-embodiment learning to human-to-robot transfer, producing robot observation-action pairs without robot data, manual annotations, or object models.
2 Related Works
Prior human-video methods commonly use robot data or intermediate representations to address the embodiment gap, limiting scalability or task coverage. Phantom instead applies data editing directly to human-to-robot transfer.
- In-the-wild video methods learn visual representations, rewards, world models, or object-motion priors, but generally rely on robot data and struggle with the embodiment gap.
- Curated human demonstrations simplify task learning but prior approaches often combine them with robot demonstrations or paired human-robot data.
- Object-centric methods estimate object poses or trajectories but are difficult to apply to deformables, granular materials, and multiple objects.
- Flow-based methods track embodiment or object points, yet several still require robot data for motion refinement or object detection.
- Robot-to-robot data-editing methods mitigate visual distribution shifts by removing one embodiment and overlaying another, inspiring Phantom’s human-to-robot adaptation.
3 Approach
Phantom converts third-person RGBD human demonstrations into robot observation-action pairs through hand-pose estimation, geometric refinement, and visual data editing. The resulting policy is trained for closed-loop zero-shot deployment with train-test visual consistency.
- Problem Setup: Each RGBD human demonstration is a sequence of third-person images showing a manipulation task performed with a pinch grasp.
- Problem Setup: The method converts each human frame into a robot observation-action pair, with edited observations matched to the target robot’s test-time image distribution.
- Problem Setup: Each robot action contains end-effector position, 6D orientation, and normalized gripper opening width.
- Problem Setup: The setup assumes known camera extrinsics and approximately matching collection and deployment viewpoints, although the scenes may differ.
- Action Labeling of Human Videos: HaMeR estimates hand keypoints and mesh vertices, while depth-based segmentation and ICP registration refine the hand pose and target action geometry.
- Action Labeling of Human Videos: Anatomical constraints limit occluded thumb and index joints to feasible motion, improving finger-pose estimation during grasping.
- Action Labeling of Human Videos: The target position is based on thumb-index fingertip geometry, while orientation uses a fitted plane and principal axis; gripper opening uses fingertip distance.
- Action Labeling of Human Videos: The bottom 20th percentile of predicted gripper distances per trajectory is assigned a fully closed gripper to mitigate grasp slippage.
4 Results
The method is evaluated across diverse manipulation tasks, robots, and unseen scenes using human-video demonstrations edited to resemble robot observations. Hand Inpaint performs strongly while avoiding the diffusion-based test-time cost of Hand Mask.
- Evaluation setup: Policies are evaluated on Franka and Kinova robots using Diffusion Policy with OSC and IK low-level controllers.Virtual robot renderings are generated using Mujoco.
- In-distribution scene: Hand Inpaint and Hand Mask achieve high success rates across all in-distribution tasks, whereas Red Line and Vanilla fail to complete any task.Table 1 reports 25 rollouts per evaluation and multiple completion levels for sweeping.
- In-distribution scene: The in-distribution evaluation covers book placement, cup stacking, sweeping, rope tying, and controlled box rotation.These tasks span rigid, deformable, and multi-object manipulation.
- Out-of-distribution scenes: Hand Inpaint achieves high success rates in three unseen environments, including an outdoor lawn, indoor lounge, and indoor lounge with an unseen surface.The sweeping evaluations include dynamic backgrounds such as moving cars and passersby.
- Method comparison: Hand Inpaint and Hand Mask perform comparably across scenes, but Hand Inpaint is on average 73% faster at rollout because it avoids test-time diffusion-based hand-mask generation.Hand Inpaint also produces training images that more closely resemble real-world robot data.
- Inpainting ablation: High-quality E2FGVI inpainting achieves an 84% success rate, while low-quality OpenCV inpainting achieves 76% on the unseen Indoor Lounge sweeping task.No-inpainting degrades performance, and the mask-only variant drops performance by 24 percentage points.
5 Conclusion
The paper demonstrates robot policy training from human videos alone and argues that simple data editing can support scalable cross-embodiment learning. The resulting robot observation-action pairs are positioned for integration into datasets for autoregressive generalist policies.
- Conclusion: The method trains robot policies without collecting robot data, using only human video demonstrations across rigid, deformable, and multiple-object manipulation tasks.It also demonstrates zero-shot deployment in novel scenes.
- Conclusion: Data editing converts human demonstrations into robot observation-action pairs that can be integrated into datasets for autoregressive generalist policies.The paper identifies this integration as a promising direction for future work.
6 Limitations
The approach has four stated limitations involving hand-pose estimation, strategy compatibility, grasp type, and task dynamics.
- Hand-pose estimation errors, especially under occlusion, limit the quality of target actions extracted from human videos.
- The robot must follow the human’s strategy, which can cause collisions or different object motions because robot grippers differ from human fingertips.
- Demonstrations are limited to pinch grasps because the evaluated robots use parallel jaw grippers.
- The evaluation covers only quasi-static tasks and does not address latency mismatch during real-hardware execution.
Appendix
The appendix explains that Phantom targets scalability without robot data and avoids limitations of paired-data, object-centric, simulation-based, and open-loop alternatives.
- Robot data bottleneck: Phantom targets true scalability by learning exclusively from human videos, avoiding the expense and bottleneck of teleoperated robot demonstrations.
- Object-centric bottleneck: Object-centric methods struggle with deformable, granular, liquid, and multiple-object scenarios because they depend on object tracking or rigid transformations.
- Simulation-based transfer: Unlike Im2Flow2Act, Phantom does not require manually built simulation environments or task-specific heuristic actions.
- Execution mode: Unlike R+X, Phantom supports closed-loop execution, which helps address unpredictable object dynamics in tasks such as sweeping.
- Paired-data methods: Unlike paired-data methods such as Vid2Robot and HOPMan, Phantom does not require paired human-robot demonstrations or substantial robot-only data.
- Robot-aligned collection: AR2-D2 requires iOS capture and live manual keypoint annotation, restricting demonstration speed, motion types, and shown tasks.
- Robot refinement: WHIRL relies on physical robot trials, whereas Phantom produces actions from human videos without prior robot interaction.
6.2 Phantom is robot agnostic
Phantom is robot agnostic: a human video can be converted into a robot demonstration for any robot capable of completing the task.
- A single human video can be converted into a robot demonstration for any robot capable of completing the task.
6.3 Evaluating the Benefits of Co-training with Diverse Human Data
The comparison examines whether scalable human-video data can approach teleoperated-data performance, while noting that the single-scene evaluation understates its cross-environment value.
- Experimental setup: The co-training experiment uses 100 teleoperated demonstrations collected in one Kinova scene for the sweeping task.
- Tradeoff: Human-video demonstrations trade some precision for lower-cost scaling across diverse environments because hand-pose estimation introduces uncertainty.
- Results: 52% success rate for 50 teleoperated demonstrations versus 44% for 50 human demonstrations was not statistically significant (p = 0.778).
- Results: 88% versus 64% success at 100 demonstrations was also not statistically significant (p = 0.095).
- Results: 84% success was reached with 300 human demonstrations, while a 300-demonstration robot-only condition was not trained because of its time requirement.
- Evaluation boundary: The comparison uses one scene and omits the cross-scene collection-time experiment because gathering equivalent robot data across many scenes was prohibitively slow.
6.5 Data collection Details
The data collection setup uses human video datasets, diverse evaluation surfaces, standardized diffusion-policy training, and virtual robot overlays.
- Human video datasets and their maximum demonstration horizons are documented in Table 7.
- Kinova sweep demonstrations use multiple collection surfaces and an unseen surface for policy evaluation.
- All Kinova experiments share diffusion-policy hyperparameters, including diverse-scene, inpainting-quality, and robot-versus-human-video comparisons.
- All tasks use the DDIM scheduler with 100 training steps and 10 inference steps.
- Training image augmentations include RandomCrop, RandomRotation, and ColorJitter, while the Cleat task uses only ColorJitter.
- Virtual robot overlays are generated using models from Mujoco Menagerie.
6.7 Detailed Task Descriptions
The evaluated tasks cover object placement variation, precision manipulation, deformable objects, and sweeping with single or multiple objects.
- Object placements vary across tasks, as visualized in Figure 10.
- Pick and Place Book samples the book within a 30 cm × 35 cm region with orientation varying by ±45 degrees.
- Rotate Box requires controlled 90-degree rotation onto a new face, with initial position sampled within a 40 cm x 35 cm region.
- Stack Cups requires placing the green cup inside the purple cup, whose diameters differ by only 1.5 cm.
- Tie Rope requires a precise cleat hitch using a highly deformable rope, with cleat position and orientation randomized.
- Franka Sweep Trash coordinates a sweeper with six loose objects whose unpredictable dynamics require continuous real-time adaptation.
6.8 Hand Mask Data Editing Method
The hand-mask method edits training and inference visuals to align human demonstrations with robot observations, while augmentation addresses arm-mask inconsistency.
- The method adapts Shadow’s data-editing strategy to human-to-robot transfer.
- During training, hand pixels are blacked out and a rendered target robot is overlaid at the estimated hand pose using camera extrinsics.
- During inference, a diffusion model predicts a human arm-and-hand segmentation mask from a 6-DOF robot pose to match training images.
- The hand-mask diffusion model generates 64×64 masks, upscales them with super-resolution, and uses temporal conditioning and attention injection.
- The robot mask remains unshifted so the policy can localize the embodiment without relying on the human hand mask.
- Randomly shifting an additional arm mask improves Stack Cups success from 40% to 52%, a 12-percentage-point increase.