PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scale

Pei Yang, Hai Ci, Yiren Song, Mike Zheng Shou

arXiv:2512.04537v1cs.CV

TL;DR

Large-scale robot data is scarce, and human videos are difficult to use directly for robot training, especially in complex third-person scenes. X-Humanoid adapts Wan 2.2 for human-to-humanoid video editing and trains it on synthetic paired videos. In user studies, participants favored its motion consistency and overall video quality, and the work releases robotized Ego-Exo4D videos.

  • Problem

    Large-scale robot data is scarce, while human videos cannot be used directly because human and robot appearances differ and third-person scenes involve complex full-body motion and occlusion.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    X-Humanoid adapts Wan 2.2 into a video-to-video human-to-humanoid editor, finetuned using paired synthetic videos created from community assets in Unreal Engine.

    Source: §1 · p. 2

  • Results

    69.0% of users rated the method best for motion consistency, and 75.0% preferred it for overall video quality against video-editing baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The released synthetic and robotized video datasets provide resources intended to help alleviate data scarcity for humanoid VLA and world-model research.

    Source: §5 · p. 9

    The method mainly considers single-person videos, leaving its behavior undefined in multi-person scenes.

    Source: §5 · p. 9

Abstract

from arXiv · show

The advancement of embodied AI has unlocked significant potential for intelligent humanoid robots. However, progress in both Vision-Language-Action (VLA) models and world models is severely hampered by the scarcity of large-scale, diverse training data. A promising solution is to "robotize" web-scale human videos, which has been proven effective for policy training. However, these solutions mainly "overlay" robot arms to egocentric videos, which cannot handle complex full-body motions and scene occlusions in third-person videos, making them unsuitable for robotizing humans. To bridge this gap, we introduce X-Humanoid, a generative video editing approach that adapts the powerful Wan 2.2 model into a video-to-video structure and finetunes it for the human-to-humanoid translation task. This finetuning requires paired human-humanoid videos, so we designed a scalable data creation pipeline, turning community assets into 17+ hours of paired synthetic videos using Unreal Engine. We then apply our trained model to 60 hours of the Ego-Exo4D videos, generating and releasing a new large-scale dataset of over 3.6 million "robotized" humanoid video frames. Quantitative analysis and user studies confirm our method's superiority over existing baselines: 69% of users rated it best for motion consistency, and 62.1% for embodiment correctness.

1. Introduction

Large-scale robot data is scarce, while direct use of human videos is blocked by the visual embodiment gap and third-person complexity. X-Humanoid adapts Wan 2.2 for human-to-humanoid video editing, trains it on synthetic pairs, and releases robotized Ego-Exo4D videos; user studies favor its motion consistency and video quality.

1. Introduction

  • VLA models and world models face a scarcity of large-scale robot data, and human videos cannot be used directly because human and robot appearances differ.Source: §1 · p. 2
  • Robotizing third-person videos is harder than arm-overlay editing because it involves full-body motion, dynamic backgrounds, and severe occlusions.Source: §1 · p. 2
  • X-Humanoid adapts Wan 2.2 into a video-to-video model that replaces a person with a fixed humanoid while preserving the person’s action and other video content.Source: §1 · p. 2
  • The Unreal Engine pipeline creates over 17 hours of paired 1080p, 30 fps human-humanoid videos, and the trained model generates a 60-hour, 3.6-million-frame robotized Ego-Exo4D dataset.The model was LoRA-finetuned using 6.4% of the synthetic dataset.Source: §1 · p. 2
  • 69.0% of users rated X-Humanoid best for motion consistency, and 75.0% preferred it for overall video quality against video-editing baselines.The authors also report faithful preservation of effects such as motion blur in challenging in-the-wild videos.Source: §1 · p. 2
  • The paper releases synthetic paired videos and robotized Ego-Exo4D videos as resources intended to help alleviate robotics data scarcity.Source: §1 · p. 2

2. Related Work

Related work frames large-scale robot data as a need for VLA and world models, while noting the challenges of using human footage directly. Existing approaches include pose- or reward-based methods, egocentric arm overlays, and video-editing models; third-person robotization remains difficult, motivating generative editing.

2. Related Work

  • Robot data scarcity constrains VLA policy generalization and world-model future prediction accuracy, while existing human-activity videos offer an opportunity to address it.Source: §2 · p. 2
  • Pose or trajectory guidance and reward-function pretraining are prior ways to address the visual appearance gap between humans and robots.Source: §2 · p. 2
  • The synthetic-pair pipeline aligns asset skeletons, transfers the same animation to human and humanoid characters, then records them in shared scenes with identical camera setups; examples vary scenes, motions, camera parameters, and framing.The examples include obstacle occlusions and partial-body or off-center views.Source: Figure 2 · p. 3, Figure 3 · p. 3
  • Manual robot-video collection is costly and limited in scale and scene diversity, while egocentric arm overlays can produce occlusion artifacts; third-person robotization remains unexplored and more complex.The cited arm-overlay data was reported to improve VLA performance.Source: §2 · p. 3
  • Although models such as Kling, Aleph, VACE, and MoCha support image-conditioned video editing, the authors found them prone to poor results on this task and instead adapted Wan 2.2.Source: §2 · pp. 3–4

3. Methodology

X-Humanoid formulates third-person human-to-humanoid editing as preserving the scene while matching the person’s action and frame alignment. Its methodology synthesizes paired training videos, adapts Wan 2.2 for video-to-video generation, and finetunes it with flow matching.

3.1. Problem Formulation

  • The model takes a third-person human activity video and outputs the same scene with a humanoid performing the same action, with motion strictly aligned frame by frame.Source: §3.1 · p. 4

3.2. Creating Human-Humanoid Paired Videos

  • Manual IK retargeting aligns character skeletons, shared animations are baked onto human and humanoid characters, and paired scenes are recorded across varied camera paths; synthesis produced 11,172 video pairs across 14 scenes.The videos contain over 2.8 million frames; capture varied exposure, color grading, focal lengths, apertures, occlusions, and character positions.Source: §3.2 · p. 4

3.3. Model Architecture

  • The adapted Wan 2.2 architecture concatenates input condition tokens before generation tokens, uses identical positional embeddings and a unidirectional attention mask, and denoises only generation tokens.Source: §3.3 · p. 4

3.4. Finetuning

  • Finetuning uses flow matching to predict ground-truth velocity along a linear path from pure noise x0 ∼ N(0, I) to clean latent tokens x1, with mean-squared error loss; inference integrates predicted velocity from t = 0 to 1.Source: §3.4 · p. 4

4. Experiments

Experiments compare X-Humanoid with video-editing baselines using synthetic ground-truth metrics and real-video user preferences, then test architecture, finetuning, prompts, and in-the-wild videos. Reported results favor the 5B model and show strong user preference for its motion and embodiment consistency.

4.1. Experimental Setup

  • Evaluation used 50 clips each from synthetic validation and Ego-Exo4D; 29 participants compared real-video results, while synthetic videos were scored with PSNR, SSIM, and MSE.The synthetic pairs came from 14 virtual scenes, with 12 for training and two reserved for validation.Source: §4.1 · pp. 4–5

4.2. Baseline Comparison

  • 69.0% of users rated X-Humanoid best or joint-best for motion consistency, 62.1% for Tesla Optimus appearance, and around three quarters preferred its background consistency and overall quality.Table 1 reports a two-part evaluation: synthetic-video similarity metrics and user preferences on real Ego-Exo4D videos.Source: Table 1 · p. 5, §4.2 · pp. 5, 7
  • MoCha generally maintained motion but missed body details; Kling had embodiment errors in motion continuity, while Aleph failed on both embodiment and motion.The qualitative comparison samples frames 15, 30, 45, 60, and 75; Kling also changes scene actions, such as dropping rather than returning a bag.Source: Figure 5 · p. 6, §4.2 · p. 7

4.3. Ablation Study

  • The 5B model performed best on the editing task: VACE models failed to produce correct robot embodiment, while the 14B model had lower video quality and over 10-times-greater training and inference times.The computational comparison is normalized per frame at batch size one.Source: Figure 6 · p. 7, Table 3 · p. 7, §4.3 · p. 8
  • 500 finetuning steps were optimal for real videos: 200 left overlay and occlusion errors, while 700 overfit synthetic data and produced implausible real-video content.The reported failure at 700 steps includes a bicycle frame fusing with the humanoid’s leg.Source: Table 4 · p. 7, §4.3 · p. 8
  • The original “Humanoid video” prompt performed best; other prompts weakened text adherence, and removing “Humanoid” yielded incorrect robot embodiments.The prompt ablation indicates the model associated “Humanoid” with the robot’s appearance.Source: Figure 7 · p. 7, §4.3 · p. 8

4.4. Robotizing In-the-wild Internet Videos

  • Internet videos retained motion, fine detail, camera cuts, aspect ratio, video quality, and motion blur; the authors attribute this capability to Wan 2.2’s generative capability.Figure 8 also shows examples with pillarboxes.Source: §4.4 · p. 8, Figure 8 · p. 8

5. Conclusion

The conclusion presents video-to-video editing finetuned on self-created synthetic pairs and releases synthetic and robotized video datasets as resources for humanoid VLA and world-model research. The approach mainly considers single-person videos and requires new LoRAs for new humanoid embodiments.

5. Conclusion

  • The work releases a 17-hour synthetic dataset and a 60-hour robotized Ego-Exo4D version as resources for humanoid VLA and world models.The robotized dataset features the Tesla Optimus humanoid.Source: §5 · p. 9
  • The editing approach translates human videos into humanoid videos with a video-to-video model finetuned on self-created synthetic video pairs.Source: §5 · p. 9
  • The method mainly considers single-person videos, leaving behavior in multi-person scenes undefined.The authors suggest explicit controls as future work.Source: §5 · p. 9
  • Using a new humanoid embodiment requires training a new LoRA; one-shot, image-conditioned methods are proposed as future work.Source: §5 · p. 9

A. Implementation Details of VACE Models

The VACE baselines used source human videos as extra input and were trained on the same dataset partitions as the main method. Their prompt strategy was changed after a fixed humanoid-video prompt produced numerous failures.

A. Implementation Details of VACE Models

B. Qualitative Ablation on Finetuning Steps

The finetuning-step ablation identifies different problems at 200 and 700 steps: incorrect occlusions at the lower count and overfitting to synthetic data at the higher count. Prior-preservation loss is suggested for exploring overfitting mitigation.

B. Qualitative Ablation on Finetuning Steps

C. More on Baseline Comparison

This section adds a baseline-comparison sample and analyzes a model failure case. It also provides randomly sampled test-set generation videos, with screenshots in Fig. 12.

C. More on Baseline Comparison

D. Network Inference Cost

A 480p, 90-frame video takes approximately 7.5 minutes to edit on an H200 GPU. Qualitative comparisons show strong motion and embodiment performance, with remaining failures on small details and under-table poses.

D. Network Inference Cost

  • 7.5 minutes and approximately 56.2GB GPU memory are required to edit a 480p, 90-frame video on an H200 GPU.Source: D. Network Inference Cost · p. 13
  • Our method produces the most consistent motions with a correct Tesla Optimus embodiment, while MoCha, Kling, and Aleph show motion or embodiment trade-offs.Source: Figure 10 · p. 14
  • Small nearby details can disappear, and under-table leg poses may have knee-occlusion errors; only Kling generated those poses reasonably, through an undesired hallucination.Source: Figure 11 · p. 15
  • The test-set visualization uses a randomly sampled subset for broader inspection of editing performance and output-video usability.Source: Figure 12 · p. 16
Loading 2512.04537v1…