Source-linked AI summary
Masquerade: Learning from In-the-wild Human Videos using Data-Editing
Marion Lepert, Jiaying Fang, Jeannette Bohg
TL;DR
Robot datasets remain scarce and human videos differ visually from robots, limiting direct transfer. Masquerade robotizes in-the-wild human videos through visual editing and co-training, producing substantially stronger transfer to unseen scenes than baselines.
Problem
Robot datasets are orders of magnitude smaller than language and vision datasets, while human videos lack precise action labels and have a visual embodiment gap with robots.
Method
Masquerade estimates hand poses, removes human arms, overlays a rendered bimanual robot, and co-trains on 675K robotized frames alongside 50 real demonstrations per task.
Results
12% →74% average success across three out-of-distribution scenes, with Masquerade strongly outperforming all baselines in every tested scene.
Takeaways & Limitations
Explicitly closing the visual embodiment gap unlocks in-the-wild human video data for robot policy learning and supports zero-shot transfer to unseen scenes.
Takeaways & Limitations
The method discards frames with fast motions, heavy occlusions, or egocentric camera motion because current pose and camera handling are limited.
Abstract
from arXiv · showhide
Robot manipulation research still suffers from significant data scarcity: even the largest robot datasets are orders of magnitude smaller and less diverse than those that fueled recent breakthroughs in language and vision. We introduce Masquerade, a method that edits in-the-wild egocentric human videos to bridge the visual embodiment gap between humans and robots and then learns a robot policy with these edited videos. Our pipeline turns each human video into robotized demonstrations by (i) estimating 3-D hand poses, (ii) inpainting the human arms, and (iii) overlaying a rendered bimanual robot that tracks the recovered end-effector trajectories. Pre-training a visual encoder to predict future 2-D robot keypoints on 675K frames of these edited clips, and continuing that auxiliary loss while fine-tuning a diffusion policy head on only 50 robot demonstrations per task, yields policies that generalize significantly better than prior work. On three long-horizon, bimanual kitchen tasks evaluated in three unseen scenes each, Masquerade outperforms baselines by 5-6x. Ablations show that both the robot overlay and co-training are indispensable, and performance scales logarithmically with the amount of edited human video. These results demonstrate that explicitly closing the visual embodiment gap unlocks a vast, readily available source of data from human videos that can be used to improve robot policies.
I. INTRODUCTION
Robot learning is constrained by scarce, costly data and a visual embodiment gap between human videos and robot observations. Masquerade addresses both challenges by robotizing in-the-wild videos, pretraining on them, and retaining the auxiliary objective during policy fine-tuning.
- Robot datasets are orders of magnitude smaller than NLP and CV datasets because real-world robot data is slow and expensive to collect.
- In-the-wild human videos offer massive, diverse manipulation data but lack precise action labels and differ visually from robots.
- Masquerade estimates hand poses, removes human arms, and overlays a rendered robot to convert in-the-wild videos into robotized clips.
- The method pretrains a vision encoder on edited videos to predict future 2D robot poses, then co-trains it with a policy head using limited real robot demonstrations.
- Across three bimanual tasks and three novel environments per task, Masquerade generalizes far beyond baselines.
A. Learning from In-the-wild Human Videos
Prior approaches use human videos for representation learning, auxiliary supervision, or motion priors, but generally leave the human–robot visual embodiment gap implicit. Masquerade combines explicit visual alignment with co-training to exploit larger-scale in-the-wild video for challenging bimanual robot learning.
- Prior work pretrains visual encoders, infers rewards, learns world models, or extracts hand trajectories from human videos for robot learning.
- Existing methods generally do not explicitly address the visual embodiment gap between human hands and robot grippers.
- Masquerade directly closes this gap through 2D inpainting and robot overlays, then combines alignment with co-training on edited human and robot data.
- The resulting pipeline targets robust performance on challenging, long-horizon bimanual tasks rather than only simple tasks.
- Unlike approaches limited to curated demonstrations, the method extends these techniques to large-scale in-the-wild videos with greater scale and diversity.
III. METHOD
Masquerade uses large-scale egocentric human videos together with a small robot demonstration set. Its three-stage pipeline edits human videos, pretrains a visual encoder, and co-trains an imitation policy using both data sources.
- The human dataset consists of naturally occurring bimanual kitchen videos recorded in diverse real-world scenes with egocentric cameras.
- The robot dataset contains a small set of bimanual demonstrations captured from the robot’s egocentric camera with known camera parameters.
- Masquerade first edits human videos to reduce the embodiment gap, then pretrains a vision encoder, and finally co-trains an imitation policy on robot and edited human data.
- The method converts each human clip into a synthetic robot demonstration and extracts 2D hand keypoints as action labels.
1) Visual editing of in-the-wild videos:
The visual-editing stage estimates hand motion, removes visible human arms, and composites a virtual bimanual robot into each egocentric frame. Because monocular videos lack reliable absolute depth, projected 2D keypoints supervise the vision model while camera motion is compensated by homography warping.
- 1) Visual editing of in-the-wild videos:: Masquerade estimates 21 anatomical keypoints per hand and maps them to smoothed 3D robot end-effector poses.
- 1) Visual editing of in-the-wild videos:: Human arms are segmented and inpainted away, after which a virtual bimanual robot following the recovered poses is rendered and composited into the scene.
- 1) Visual editing of in-the-wild videos:: Monocular hand-pose estimation cannot provide accurate absolute 3D pose, so projected 2D keypoints are used as auxiliary vision-model supervision rather than direct policy inputs.
- 1) Visual editing of in-the-wild videos:: Future keypoints are warped into the current frame using homographies to compensate for egocentric camera motion.
3) Data Filtering:
Masquerade filters unstable clips and trains a language-conditioned vision encoder with 2-D keypoint supervision alongside a robot policy loss. The co-training objective aligns edited human videos with real robot demonstrations.
- 3) Data Filtering:: Frames with excessive camera motion or invalid actions from keypoint errors and kinematic limits are filtered before policy learning.The filtering retains clips with stable, reliably labeled actions.
- 1) Vision encoder pretraining:: The vision encoder f(x,z) is pretrained on processed human videos to predict future 2-D robot keypoints from visual features and clip-level language embeddings.An MLP maps encoder features to the 2-D keypoint targets, while FiLM conditions features on the language embedding.
- 2) Policy learning using cotraining:: During policy training, the pretraining loss is retained while a diffusion-based action head learns from task-specific robot demonstrations.The policy loss predicts robot Cartesian end-effector actions, and both losses are optimized simultaneously.
- 2) Policy learning using cotraining:: A rendered robot is overlaid on the edited human videos so the model sees a robot embodiment during both representation learning and policy training.The overlay is used to reduce the visual gap between edited human data and real robot data.
IV. RESULTS
Masquerade is evaluated on three long-horizon bimanual kitchen tasks using large-scale edited human video data and 50 robot demonstrations per task. Evaluation uses partial-progress scoring across unseen scenes.
- IV. RESULTS: 10K clips totaling 675K frames from Epic Kitchens and 50 task-specific robot demonstrations per task train the policy.The vision encoder uses a ViT-Base initialized with ImageNet weights, and the action head follows Diffusion Policy.
- IV. RESULTS: Three long-horizon bimanual tasks are evaluated in out-of-distribution scenes, with each rollout scored by assigning one third of the total score to each subtask.The tasks are Stack Pots, Scrape Potato, and Sweep Chilis.
B. Baselines
Masquerade is compared with human-video representation learning and general-purpose robotics backbones on three bimanual tasks. The evaluation uses out-of-distribution scenes and reports substantially stronger performance for Masquerade.
- B. Baselines: Baselines include HRP, ImageNet, and DINOv2, covering human-video affordance pretraining and general-purpose visual representations.All models use the ViT-base architecture.
- B. Baselines: 62 percentage points (12% →74%) is the average improvement over all baselines across every tested out-of-distribution scene.The comparison covers three tasks, with 50 robot demonstrations collected in one scene for each task.
- B. Baselines: Three out-of-distribution scenes are evaluated for each task, alongside task-specific scene layouts shown for Stack Pots, Scrape Potato, and Sweep Chilis.The figure distinguishes in-distribution scenes from out-of-distribution scenes by task row.
D. Do robot overlays improve performance?
Robot overlays and co-training both materially improve policy performance in out-of-distribution scenes, while increasing edited-video data steadily raises success.
- Removing robot overlays causes a steep performance drop across Stack Pots, Scrape Potato, and Sweep Chilis in OOD Scene 1.
- Removing co-training also causes a dramatic performance drop because the encoder forgets representations learned from human videos.
- Both robot overlays and co-training are essential for robust success rates in OOD settings, based on 25 rollouts per bar.
- 0% → 2%, 10% → 26%, 50% → 47%, and 100% → 68% success on Stack Pots as the edited-video fraction increases.Each condition used 25 rollouts with identical training epochs.
- Success rises monotonically with more edited human videos in the Stack Pots data-scaling experiment.Results were measured in OOD Scene 1 with error bars showing ± SEM over 25 rollouts.
G. In-distribution vs Out-of-distribution performance
Masquerade generalizes from its training scene to unseen scenes with a smaller performance drop than baselines, while remaining bounded by limitations in pose, depth, camera, and grasp retargeting.
- Masquerade maintains similar in-distribution and OOD performance on Sweep Chilis, unlike baselines that suffer large drops.
- Fast motion and heavy occlusion degrade monocular hand-pose estimation, forcing problematic frames to be discarded.Improved hand-pose estimators could improve the overlays.
- Missing depth causes incorrect robot-pixel ordering around scene objects, and egocentric camera motion requires filtering many frames.
- The data-editing pipeline could also support reward learning, motion-prior extraction, and video generation beyond policy-learning representations.
- Across three long-horizon bimanual tasks, Masquerade outperforms baselines by over 5× and exhibits minimal in-distribution-to-OOD degradation.The method uses 675K robotized frames and 50 real demonstrations per task.
- Ablations identify robot overlays and co-training as indispensable, while scaling the human-video corpus steadily improves success rates.
APPENDIX
The appendix specifies the training data, robot hardware, policy-training configurations, and camera-based bimanual setup used by Masquerade.
- Training configuration: Co-training experiments test λ values from 0.5 to 40, with λ = 10 performing best.The diffusion policy uses a DDPM noise scheduler with 100 training and inference steps.
- Training data: 675,713 Epic Kitchens frames train the vision encoder, while each task contributes 50 bimanual robot demonstrations.The human videos are edited before encoder training; robot demonstrations are collected with an Oculus headset.
- Video preprocessing: Camera-motion filtering removes frames exceeding 5 cm translation or 0.5 rad rotation per timestep.Occluded-hand actions are carried forward from the last visible frame, while fully invisible hands receive a fixed out-of-frame label.
- Robot setup: The bimanual platform uses two Kinova Gen3 7-dof arms with Robotiq 2F-85 grippers and a rigidly mounted egocentric ZED Mini camera.Cartesian control uses inverse kinematics and a low-level joint-position controller running at 1000 Hz.
E. Camera extrinsics detailed explanation
Masquerade fixes the robot–camera transform within each sequence while evaluating generalization across randomized object placements and unseen scenes.
- Camera extrinsics: The camera pose is frozen to each sequence’s first frame, and later keypoints are interpreted in that view via homographies.This preserves a fixed robot–camera transformation within a sequence despite camera movement between sequences.
- Evaluation: The evaluation averages success over three bimanual tasks and three out-of-distribution scenes, using 10 rollouts per scene.The tasks are Stack Pots, Scrape Potato, and Sweep Chilis; error bars show ± SEM.
- Evaluation: Detailed per-task, per-scene evaluation results are reported in Figure 8.The figure provides the task- and scene-level breakdown underlying the aggregate success-rate comparison.
- Object placement: Object initialization is randomized within task-specific colored regions for pots, plates, spatulas, bowls, sponges, and chilis.Figure 11 specifies distinct yellow, blue, and red placement regions for each task.