PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

Seungyong Lee, Suhyeok Jang, Minyoung Kim, Jinwoo Shin

arXiv:2611.05163v1cs.ROcs.AI

TL;DR

Robot learning for new tasks still depends on robot demonstrations, while human videos leave visual and action gaps for dexterous embodiments. RoboTryOn translates human videos into robot demonstrations with action labels. It raises average success on novel out-of-domain tasks from 2.1% to 60.0% and matches teleoperation-trained performance at about one-third of the collection cost.

  • Problem

    Human videos are a scalable source of demonstrations, but visual and action gaps limit their use for dexterous robot embodiments.

    Source: §1 · p. 1

  • Method

    RoboTryOn translates human videos into robot videos using robot keyframes and hand-skeleton-guided synthesis, then infers action labels with an inverse dynamics model.

    Source: §1 · p. 2

  • Results

    2.1% to 60.0%: average success on out-of-domain tasks with unseen goals, skills, and objects across two embodiments; human-video training also matches teleoperation-only performance at about one-third the collection cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Synthesized demonstrations support learning seven tasks, including bimanual tasks, without robot demonstrations of those tasks.

    Source: §1 · p. 2, §5 · p. 9

    Each new embodiment requires a few hours of teleoperated robot video, and pseudo-labels do not capture forces or accurately label force-dependent or fine-precision tasks such as tight insertion.

    Source: A. Limitations and Future Work · p. 16

Abstract

from arXiv · show

Robot foundation models have shown remarkable performance in robot control, yet adapting them to a new task still relies heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, but overlaying rendered robots on human videos or retargeting human poses into robot actions leaves significant visual or action gaps for high-DoF robot embodiments. To address this limitation, we propose RoboTryOn, which treats human-to-robot translation as virtual try-on: try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Specifically, RoboTryOn converts a human video into a video of the target robot specified by a reference image, while preserving the original task, objects, and background. To maintain visual consistency, we translate selected keyframes autoregressively and use them to guide video synthesis with motion conditioned on the human hand skeleton. Additionally, we use an inverse dynamics model (IDM) that incorporates depth features to infer action labels from synthesized videos for policy learning. Our experiments across multiple humanoid embodiments show that policies trained with our synthetic data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.

1. Introduction

RoboTryOn addresses the visual and action gaps that make human videos difficult to use as dexterous robot demonstrations. It translates human videos into robot demonstrations with action labels, improving success on unseen tasks and matching teleoperation-trained performance at lower collection cost.

1. Introduction

  • The approach targets the cost and effort of teleoperation, which is especially burdensome for dexterous hands requiring coordination of many degrees of freedom and fine-grained contacts.Teleoperation also requires robot hardware and a teleoperation system.Source: §1 · p. 1
  • Human videos are abundant and capture object manipulation, including coordinated bimanual interaction, without requiring a robot during recording.Egocentric videos range from in-the-wild collections to annotated corpora recorded for robot learning.Source: §1 · p. 1
  • Human videos leave visual and action gaps: robot deployment observes a different embodiment, while reconstructed hand poses cannot specify exact joint commands or grip force.Prior visual edits must also be applied to the robot camera stream at training and deployment.Source: §1 · p. 1
  • RoboTryOn translates selected human frames into robot keyframes, synthesizes motion guided by hand-skeleton flow, and infers joint-space actions from the generated video.The generative models learn from robot videos through a shared arm-mask and hand-skeleton interface, so inference does not require paired human and robot demonstrations.Source: §1 · p. 2
  • 2.1% to 60.0%: RoboTryOn raises average success on out-of-domain tasks with unseen goals, skills, and objects across two embodiments.It reaches 45.9% success on bimanual tasks despite teleoperated training demonstrations containing only single-arm manipulation, and matches teleoperation-only performance at about one-third of the collection cost.Source: §1 · p. 2
  • The contributions include eliminating test-time visual editing and producing better action labels than retargeted actions.The system also uses generative models trained on robot data alone to turn egocentric videos into robot demonstrations.Source: §1 · p. 2

2. Related Work

Prior work uses human videos for transferable representations, retargeted or rendered robot data, and generative translation. RoboTryOn combines embodiment translation with action labeling, using keyframes and skeleton-guided synthesis for temporally consistent robot videos.

2. Related Work

  • Human videos support robot learning through visual representations, foundation-model training, motion retargeting, rendered observations, and simulation-based digital twins.Some simulation approaches also refine action trajectories.Source: §2 · p. 2
  • Generative translation ranges from paired human–robot training in Mitty (Song et al., 2025) to H2R-Grounder (Ci et al., 2025), which reconstructs robot videos from inpainted backgrounds and gripper-pose cues.RoboTryOn supports multiple dexterous embodiments with shared generative models and produces action-labeled demonstrations for real-world policy learning.Source: §2 · p. 2
  • Video-synthesis work addresses temporal consistency, while inverse dynamics models infer actions from generated videos; RoboTryOn adopts keyframes to guide synthesis.DreamVVT generates keyframes to guide video synthesis, and RoboTryOn applies this decomposition to embodiment translation.Source: §2 · p. 2
  • RoboTryOn’s pipeline translates human keyframes, synthesizes intervening robot motion under hand-skeleton guidance, then labels actions with an inverse dynamics model.The figure also depicts masked-frame inpainting using a previous robot keyframe as reference.Source: Figure 1 · p. 3

3. Method

RoboTryOn turns egocentric human demonstrations into robot demonstrations by translating keyframes, synthesizing a robot video, and inferring its actions and states. The resulting synthetic demonstrations can train a policy alone or alongside teleoperated data.

3.1. Problem Setup and Overview

  • RoboTryOn uses a human video and target-robot reference image to generate a robot demonstration, translating keyframes, synthesizing video, inferring actions, and deriving states.The generated demonstrations can train the policy alone or together with teleoperated demonstrations.Source: §3.1 · p. 3, §3.5 · p. 5

3.2. Keyframe Translation

  • Keyframes are selected at low-speed wrist moments with sufficient displacement, then translated in temporal order using a target-robot reference image, an arm-and-hand mask, and a hand skeleton.Skeleton perturbation and dropout during training make the model robust to imperfect retargeting; the skeleton offers coarse motion guidance while the learned prior supplies hand configuration.Source: §3.2 · p. 4

3.3. Video Synthesis

  • The video model synthesizes the full robot video between translated keyframes, guided by hand-skeleton motion.When a video exceeds the model's maximum length, consecutive clips share their boundary keyframe.Source: §3.3 · p. 4

3.4. Pseudo Action Labeling

  • An inverse dynamics model predicts joint-space action chunks from pairs of synthesized-video observations, and the states are derived from those action labels.The model is trained with teleoperated actions as targets; inverse kinematics corrects wrist positions while retaining the predicted hand joints.Source: §3.4 · pp. 4–5

3.5. Policy Training

  • The policy is trained on synthetic demonstrations alone or combined with teleoperated data, and generates action chunks conditioned on visual observations, proprioceptive state, and task instruction.At inference, action chunks are sampled from Gaussian noise using Euler ODE sampling.Source: §3.1 · p. 3, §3.5 · p. 5

4. Experiments

Experiments on two dexterous bimanual robots cover four tasks with teleoperated demonstrations and seven human-video-only tasks. RoboTryOn is evaluated against human-video conversion baselines, through analyses of visual and action gaps, collection cost, execution speed, and pipeline ablations.

4.2. Main Results

  • In-domain success rises from 47.3% to 75.9% on OpenArm and from 69.6% to 79.5% on RB-Y1.The authors attribute the gains to added variation in object placements and arm and hand motions.Source: §4.2 · p. 6
  • Out-of-domain success rises from 3.1% to 73.0% on OpenArm and from 1.0% to 46.9% on RB-Y1; both baselines remain below 10%.The seven tasks introduce novel objects, goals, skills, or bimanual manipulation, with no robot demonstrations.Source: §4.2 · p. 6

4.3. Analysis

  • RoboTryOn reduces Fréchet SigLIP distance to real robot videos from 125.5 to 32.7, while overlay reduces it by about half and masking leaves it largely unchanged.RoboTryOn overlaps with real robot videos in embedding space while the policy can use raw camera observations at test time.Source: §4.3 · p. 7
  • Open-loop replay succeeds in 69.7% of episodes with IDM labels versus 34.8% with retargeted labels, with the difference arising mainly during grasping.Retargeted motions approach objects, but imprecise hand reconstruction and different hand geometry prevent secure grasps.Source: §4.3 · p. 7
  • Human-video-only training matches 50 teleoperated demonstrations per task at about one third of the collection cost.Adding an equal number of human videos raises success from 47.3% to 71.4% with less than one additional recording hour.Source: §4.3 · p. 8
  • Matching teleoperated speed raises in-domain success from 75.9% to 85.7% while keeping out-of-domain success similar; 2× training yields the fastest policy but lower success on both groups.The policy's task completion time follows the speed of its synthesized training videos.Source: §4.3 · p. 8

4.4. Ablations

  • Removing any of four pipeline components reduces success; removing hand-skeleton flow causes the largest drop, while wrist-correction removal hurts out-of-domain tasks more.The component ablations respectively disrupt scene consistency, object and scene preservation, motion following, or action-label correction.Source: §4.4 · p. 9
  • RoboTryOn also improves RLDX-1-Img success from 53.6% to 60.7% in-domain and from 0.0% to 40.3% out-of-domain.This tests the synthesized demonstrations with a VLA architecture different from π0.5.Source: §4.4 · p. 9

5. Conclusion

RoboTryOn translates human video into robot demonstrations while preserving the demonstrated motion. Synthesized demonstrations teach seven tasks, including bimanual ones, without task-specific robot demonstrations and match teleoperated performance at about a third of the collection cost.

5. Conclusion

  • Human video translated by RoboTryOn matches teleoperated-demo success rates at about a third of their collection cost.The approach keeps the demonstrated motion while swapping the human embodiment for a robot.Source: §5 · p. 9
  • Synthesized demonstrations teach policies seven tasks without robot demonstrations, including bimanual tasks.The evaluation uses two real robots with dexterous hands.Source: §5 · p. 9

Appendix Contents

  • The appendices cover implementation, experimental and analysis details, flow-matching, and additional results.Source: Appendix Contents · p. 15

A. Limitations and Future Work

RoboTryOn’s scope is bounded by its training data, pseudo-label precision, camera viewpoint, robot capabilities, and evaluation setting. Future work targets larger human-video corpora and more accurate inverse-dynamics labels.

A. Limitations and Future Work

  • Tasks requiring forces or fine precision, such as tight insertion, are not yet labeled accurately enough by the pseudo-labels.The IDM reads joints from pixels, and label precision is bounded by stereo video and hand reconstruction.Source: A. Limitations and Future Work · p. 16
  • Moving-camera egocentric videos fall outside the video model and IDM training distribution; the IDM predicts fixed neck and base actions.Source: A. Limitations and Future Work · p. 16
  • RoboTryOn was evaluated on tabletop manipulation on two robots; policy evaluation on the translated GR-1 videos was unavailable.Source: A. Limitations and Future Work · p. 16
  • The demonstrated motion must be executable: retiming cannot fix targets beyond arm reach or grasps the robot hand cannot form.Source: A. Limitations and Future Work · p. 16
  • Adding an embodiment requires a few hours of teleoperated video for model training, a cost paid once per robot rather than per task.Synthesis also trades collection time for compute, favoring tasks expensive to teleoperate.Source: A. Limitations and Future Work · p. 16
  • Future work proposes translating large egocentric corpora and improving IDM accuracy, including by conditioning it on reconstructed wrist position.Executed policy actions could also become IDM training data.Source: A. Limitations and Future Work · p. 16

B.1. Policy Training Details

The pipeline selects human-video keyframes, translates them into robot frames, synthesizes intervening video, and derives actions with an inverse dynamics model. Skeleton guidance, retiming, and wrist correction shape the resulting demonstrations.

B.1. Policy Training Details

  • π0.5 and RLDX-1-Img are fully fine-tuned for 30k steps, with model-specific action-head designs and action chunks.π0.5 uses 50-step chunks; RLDX-1-Img uses 16-step chunks.Source: §B.1 · p. 16

B.2.1. Keyframe Selection

  • Keyframes come from slow-wrist moments after sufficient displacement, with confidence-aware fallbacks and a 145-frame maximum interval.First and last frames are always included; pick-and-place videos typically yield about four keyframes, and longer tasks six to seven.Source: §B.2.1 · pp. 16–17

B.2.2. Hand Skeleton

  • Hand reconstruction combines stereo views, while a shared 17-keypoint skeleton maps human and robot hands and provides only coarse guidance.Training perturbs and drops keypoints so the learned robot-hand prior supplies detailed hand configuration.Source: §B.2.2 · p. 17

B.2.3. Keyframe Translation

  • Shape-agnostic masks obscure arm silhouettes while preserving manipulated objects; the first robot keyframe uses a robot reference to select the embodiment.Later keyframes are translated autoregressively, and reference frames are sampled two to five seconds before the target.Source: §B.2.3 · p. 17

B.2.4. Video Synthesis

  • The video model synthesizes 10–25-second demonstrations from clips of at most 145 frames, guided by rendered skeleton flow.Clips share boundary keyframes and are concatenated in time.Source: §B.2.4 · p. 18

B.2.5. Retiming

  • Retiming resamples skeleton flow within task phases to match teleoperated mean action speeds, especially for human grasping and lifting.Source: §B.2.5 · p. 18

B.2.6. Inverse Dynamics Model

  • The IDM predicts 20-step joint-action chunks from frames one second apart, using a depth-feature backbone shared across embodiments.It smooths overlapping sliding-window predictions and derives synthesized states from action labels with a four-step lag.Source: §B.2.6 · p. 18

B.2.7. End-Effector Correction

  • End-effector correction replaces wrist translation using reconstructed human wrists, then solves arm joints by inverse kinematics while preserving orientation and hand joints.It applies to both arms in bimanual demonstrations.Source: §B.2.7 · p. 18

B.2.8. Training Setup

The image model, video model, and IDM share a corpus of teleoperated robot episodes, with distinct model configurations. The appendix also specifies baseline implementations and inference settings.

B.2.8. Training Setup

  • All three models train on teleoperated episodes from OpenArm, RB-Y1, and GR-1, totaling 8.82 hours across the stated collections.The OpenArm and RB-Y1 recordings are stereo; GR-1 recordings are monocular.Source: §B.2.8 · p. 18
  • Training uses a fine-tuned FLUX.2 image model, an in-context LTX-2.5 video LoRA, and a Depth Anything V2 IDM with a DreamGen action head.Source: §B.2.8 · p. 19

B.3. Baselines

  • Both baselines use RoboTryOn’s backbone, teleoperated episodes, and human videos; their implementations use retargeted pseudo-actions and differ in visual editing.EgoMimic masks arms, while Masquerade inpaints the human arm and composites a robot rendering.Source: §B.3 · pp. 19–20

B.4. Inference Details

  • Inference generates keyframes sequentially, synthesizes video in two stages, and applies the IDM in sliding windows; the pipeline timing is reported for a 15-second demonstration.The timing scope is one embodiment on a single H100.Source: §B.4 · p. 20

C.1. Real-World Environment

The real-world evaluation uses two humanoid embodiments across in-domain pick-and-place and out-of-domain manipulation tasks. A matched OpenArm simulation evaluates selected pick-and-place tasks using recorded human-video scenes.

C.1. Real-World Environment

  • The real-world setup uses OpenArm and RB-Y1 humanoids with two 7-DoF arms, dexterous hands, neck-mounted stereo cameras, and egocentric visual inputs.OpenArm has 6-DoF hands; RB-Y1 has 20-DoF hands.Source: §C.1 · p. 20
  • Seven out-of-domain tasks have 50 human videos each and no robot demonstrations, while four in-domain pick-and-place tasks have 50 teleoperated demonstrations per robot and 50 human videos each.The out-of-domain tasks include pouring, brushing, sequential two-hand box conveyance, air-fryer use, and ball-in-cup manipulation.Source: §C.1 · p. 21
  • Real-world evaluation uses 28 trials per task, randomizes object starting positions, and scores bimanual tasks by the successful completion of each hand’s required action.Task-specific success criteria include reaching a marked region for brushing and tilting a bottle over the target cup for pouring.Source: §C.1 · pp. 21–22

C.2. Simulation Environment

  • The Isaac Lab simulation matches the real OpenArm embodiment and camera viewpoint, and evaluates three pick-and-place tasks using object positions recorded in the corresponding human-video scenes.PnP Doll is excluded because its simulation asset is unavailable; each of 66 episodes corresponds to one human video and succeeds when the object is picked up and placed in the container.Source: §C.2 · p. 22

D. Analysis Details

The analysis compares visual similarity and open-loop action replay, then describes the flow-matching sampling procedure and the collection-cost conditions. Replay results favor RoboTryOn over retargeting on the reported Bottle, Box, and Ball counts.

D. Analysis Details

  • Frames from the seven tasks without robot data lie closer to real robot frames than edited frames from the four tasks with robot data, so the visual overlap is not just reproduction of training episodes.The analysis compares pooled SigLIP features and Fréchet distances across human, edited, RoboTryOn, and teleoperated corpora.Source: D. Analysis Details · pp. 22–23
  • RoboTryOn succeeds on 17 Bottle, 14 Box, and 15 Ball episodes; retargeting succeeds on 14, 7, and 2, respectively, in 22-episode open-loop replay.RoboTryOn without wrist correction succeeds on 14, 16, and 12 episodes, respectively.Source: D. Analysis Details · p. 23, Table 8 · p. 23

E. Flow-Matching Objective and Sampling

  • Conditional flow matching interpolates noisy input from Gaussian noise to clean data, trains a velocity network to predict the target velocity, and samples by integrating that velocity from noise to data.The Euler sampler performs repeated updates with step size Δτ; the paper’s policy and RoboTryOn models serve as velocity functions.Source: E. Flow-Matching Objective and Sampling · pp. 23–24
  • The cost analysis counts about 38 teleoperated episodes versus 214 human videos per person-hour, while comparing policies trained on the four pick-and-place tasks alone.Teleoperation uses two people and yields about 14 minutes of demonstration per person-hour, compared with about 45 minutes of human video recorded by one person.Source: Table 10 · p. 24, Table 9 · p. 24

F.1. Data Collection Cost

The cost analysis compares teleoperation with human-video collection and plots policy success against collection cost. Per-task result tables provide the breakdowns for the main experiments and policy backbones.

F.1. Data Collection Cost

  • One person-hour yields about 14 minutes of teleoperated demonstration or 45 minutes of egocentric human video, corresponding to 38 versus 214 episodes per person-hour.Teleoperation occupies two people, and people complete each task faster than a teleoperated robot.Source: §F.1 · p. 24
  • The cost comparison includes teleoperation-only policies and policies trained with equal splits of teleoperated and human-video demonstrations, evaluated on four pick-and-place tasks.The ten policies in this analysis are not the Table 1a policies, which also train on seven tasks without robot demonstrations.Source: §F.1 · p. 24

F.2. Per-Task Results

  • Per-task results are provided for the main experiments, the π0.5 and RLDX-1-Img backbones, and the component variants.The corresponding tables report success rates, with robot demonstrations available only for pick-and-place tasks in the main-experiment breakdown.Source: §F.2 · p. 25, Table 11 · p. 25

F.3. Ablated Variants

The ablations show distinct failure patterns when keyframe consistency or skeleton-guided motion is removed. The paper also reports translations across three robot embodiments using a single model set.

F.3. Ablated Variants

  • Independent keyframe translation produces inconsistent coffee-pot grasp configurations and visible hand jitter, while the full model maintains a consistent grasp.The fingers wrap below the handle at 5.0 seconds and reach behind it at 6.0 seconds in the ablated variant.Source: §F.3 · p. 25
  • Using only the first keyframe allows the coffee pot to change into a box, while removing skeleton flow makes motion deviate from the demonstrator’s trajectory and disrupts the handle grasp.The object change occurs inside the container at 11.0 seconds; the grasp configuration is lost at 3.5 seconds without skeleton flow.Source: §F.3 · p. 25
  • A single set of models translates human videos into OpenArm, RB-Y1, and GR-1 videos, changing only the embodiment.The per-task ablation table reports success rates for the component variants, while the backbone table reports OpenArm results for π0.5.Source: Figure 10 · p. 26, Table 13 · p. 26, Table 12 · p. 26

F.4. Qualitative Results

The qualitative results show RoboTryOn translations across three robot embodiments and illustrate policy rollouts, long-horizon consistency, and synthesis and action-label failures.

F.4. Qualitative Results

  • One model set translates human videos into the OpenArm, RB-Y1, and GR-1.The paper presents these translations as qualitative results.Source: §F.4 · p. 26

F.4.2. Rollouts on RB-Y1

  • Air Fryer requires inserting fingers between the handle and body before closing, a fine-grained grasp beyond the teleoperated data and IDM-label resolution.The longer, multi-joint WujiHand2 fingers also tend to push the fryer away when closing in front of the handle, unlike the shorter 1-DoF Inspire RH56F1 fingers.Source: §F.4.2 · pp. 26–27

F.4.1. Rollouts on OpenArm

  • OpenArm rollouts cover all eleven tasks: four with robot demonstrations and seven with human video only.The policy was trained on teleoperated and synthesized episodes.Source: §F.4.1 · p. 26, Figure 12 · p. 27

F.4.2. Rollouts on RB-Y1

  • On RB-Y1, rollouts show four demonstration tasks and six human-video-only tasks; Air Fryer is omitted because no policy completes it (0% success).The rollout policy was trained on teleoperated and synthesized episodes.Source: §F.4.2 · p. 26, Figure 13 · p. 28

F.4.3. Long-Horizon Synthesis

  • Long-horizon synthesis divides a 25-second human video into clips at selected keyframes, with adjacent clips sharing boundary keyframes to preserve robot and object consistency.The figure shows the human video and selected keyframes above the generated robot video and clip boundaries below.Source: §F.4.3 · pp. 27–28, Figure 14 · p. 28
  • Failure examples include disappearing objects and fingers passing through a handle during generation, plus imprecise IDM labels that place grasping hands below target handles or bottles.The label errors lead to failed coffee-pot and bottle-to-shelf rollouts.Source: Figure 15 · p. 29
Loading 2611.05163v1…